Diffa Diffa

Currently, when the version policy processes a change, it emits an event that gets processed by the DifferencesManager and results in an entry being created/updated in the Differences Store. The Version Store was also informed of this change, but may not flush the changes until some later point.

If the agent were to crash between this process happening and the flush occurring, then the change could end up being lost, and the difference store holding data that the correlation store doesn't have. Since changes are sent incrementally to the difference store, there isn't any way of cleaning this up or even detecting it.

Given the incremental nature of the transfer, changes should never be seen outside of the version store until they've been flushed to disk. This will probably introduce a delay in realtime event processing - but if the realtime flush timeout is suitably low, then it shouldn't be a large issue.

It might be worth tackling #388 first.

As an operational procedure, it is recommended that in the event of the agent crashing, the each pair should be removed and re-added to ensure that no data was lost.