Streamline difference event processing
Reported by Ben Hood | August 8th, 2011 @ 01:32 PM
This ticket is an in-memory variant of #294.
The following process should be used to handle difference events in a domain:
- Scan proceeds as normal. The current match buffering code is removed.
- At the completion of a scan, a sync operation (terminology TBD)
is initiated. This operation will:
- Generate a sync GUID or increasing ID
- Iterate over all unmatched records in the Version Correlation Store, and generate a difference event with the data from the store along with the sync ID
- The differences store will create new differences where the data is unique, and update existing differences with the sync GUID where it isn't
- At the completion of the correlation store iteration, a message is sent to remove all differences that aren't marked with the current sync id. This will remove (that is, mark as matched) any difference that isn't currently in the correlation store (and therefore has reached a matched state)
Comments and changes to this ticket
-
Ben Hood August 8th, 2011 @ 01:33 PM
- no changes were found...
-
Ben Hood August 10th, 2011 @ 09:58 AM
- Tag changed from “persistent session events” to “persistent session events, ignores”
-
Ben Hood August 10th, 2011 @ 10:04 AM
(from [74b1e8e931c8a7f1410330d390b8af92ae6cc807]) [#332] Initial shuffling ready to introduce a difference writer. https://github.com/lshift/diffa/commit/74b1e8e931c8a7f1410330d390b8...
-
Ben Hood August 10th, 2011 @ 11:25 AM
(from [9232e5c0c496696111a3b9b1f8e0a2e6c7093179]) [#332] Mark the last time we saw a difference, and add ability to remove old ones. https://github.com/lshift/diffa/commit/9232e5c0c496696111a3b9b1f8e0...
-
Paul Jones August 10th, 2011 @ 11:29 AM
- Tag changed from “persistent session events, ignores” to “persistent session events”
- Milestone cleared.
- State changed from “new” to “resolved”
- Assigned user set to “Ben Hood”
- Milestone order changed from “106” to “0”
Ready for QA.
Note that the implementation I've written is actually substantially simpler than this, since it makes use of a lastSeen timestamp on differences. When a difference is re-seen during a scan, it has it's lastSeen updated. At the end of a scan, we remove all differences that existed before the lastSeen time.
-
Ben Hood August 10th, 2011 @ 11:35 AM
(from [825c8063bed8c3c57aecc0e756e9c3cbf7509bf6]) [#332] Improve test coverage of cutoff mechanism. https://github.com/lshift/diffa/commit/825c8063bed8c3c57aecc0e756e9...
-
Ben Hood August 10th, 2011 @ 12:44 PM
(from [95e4217a0d334f3f40739550edfc3a7f25fff462]) [#332] Remove unused methods and dependencies. https://github.com/lshift/diffa/commit/95e4217a0d334f3f40739550edfc...
-
Ben Hood August 10th, 2011 @ 12:44 PM
(from [3c842fa01915bc93ba36d88ef058c33ab5321763]) [#332] Introduce helper methods in test. https://github.com/lshift/diffa/commit/3c842fa01915bc93ba36d88ef058...
-
Please Sign in or create a free account to add a new ticket.
With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.
Create new ticket
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป
A Real Time Differencing Tool.
Ben Hood