#2 ✓closed
Ben Hood

Generalize entity partition criteria and granularity function

Reported by Ben Hood | September 27th, 2010 @ 10:28 AM | in 0.9.3

Motivation

ATM the business date is hard baked into the protocol as a partitioning scheme. Whilst this works for a number of applications (e.g. trading), it is not general enough for applications that don't really have this notion (e.g. consider synchronizing ref data, which realistically doesn't have a business date). To do this, we are going to have to change the protocol in these ways:

  • Replace the business date attribute with a generalized partition attribute
  • Generalize the Range Granularity concept to be able to communicate any aggregation function on the network - these function names can be categorized into:
    • Well known function names (e.g. DAILY, MONTHLY, YEARLY for timestamp based partitions, nth character of a GUID based partitions )
    • User defined function names that are configured into the agent for a particular participant

Given the fact the detection date is going to get persisted (see later comments on ticket #1), the Hibernate-mapped Correlation entity will get a last update timestamp, and as part of this issue, the date field of the Correlation entity should be migrated to be a category string.

Use Cases

  • Specifying which attribute of an entity to use as a partition criterion should be configurable
  • Support range queries on:
    • Dates
    • Sequence numbers
  • Support list queries of a specified set of attributes
  • Allow multiple constraints to be queried simultaneously

Proposed Design

The solution is composed of of the following concepts:

  • Category - this is name of the attribute an entity that is configured as a partitioning variable
  • Constraint - a query for digests contains 0 or more constraints. A constraint has the following structure:
    • Category Name
    • A well known function name to apply in the participant
    • EITHER a list of values OR a range of values TO apply
  • Digest - this is similar to the current structure, but is now changed to carry categories. A digest has the following structure:
    • Id (as before)
    • Version (as before)
    • [{k,v}] - a list of key-value pairs that represent category-name-value pairs

The intended approach is to maintain a Lucene based index so that categories can be added and indexed dynamically without having to manipulate the DDL.

Comments and changes to this ticket

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

A Real Time Differencing Tool.

Shared Ticket Bins

People watching this ticket

Attachments

Referenced by

Pages