Handling large numbers of entries
Reported by Paul Jones | January 24th, 2011 @ 01:02 PM | in 0.9.3
Currently, the HibernateVersionCorrelationStore retrieves ids from the indexer, and then sends them into the database via a series of constraints (one per id). There is a high likelihood that this is going to hit database limits, resulting in queries being rejected when there are lots of entities in the system.
To correct this, we're considering making the Lucene index the primary store for correlations. Correlations would no longer live in the database, and would be directly stored (and retrieved) from the index.
To progress this, the plan is:
- Create a performance benchmark test loading very large numbers
of entities to confirm the failure scenario;
- Eliminate the HibernateVersionCorrelationStore, and make the LuceneIndexer implement the VersionCorrelationStore functionality directly;
Comments and changes to this ticket
-

Ben Hood January 25th, 2011 @ 01:04 PM
- Assigned user set to Paul Jones
- State changed from new to open
-

Ben Hood January 26th, 2011 @ 08:18 AM
(from [bfd7265f26d86aa9b0d9ee15d309bc0e8332118c]) [#162] Add initial benchmark performance test. https://github.com/lshift/diffa/commit/bfd7265f26d86aa9b0d9ee15d309...
-

Ben Hood January 26th, 2011 @ 08:18 AM
(from [6e130b8f9da5732e231d8d2f5541add6fa115f83]) [#162] First cut at VersionCorrelationStore running in Lucene. https://github.com/lshift/diffa/commit/6e130b8f9da5732e231d8d2f5541...
-

-

Ben Hood January 26th, 2011 @ 09:01 AM
(from [3e659f1e93d4c73559936b31862d008b768f7268]) [#162] Simplify constraint types and definitions. https://github.com/lshift/diffa/commit/3e659f1e93d4c73559936b31862d...
-

Ben Hood January 26th, 2011 @ 09:28 AM
(from [ab4326ce9cb13bc77eef801468d787e2213a0374]) [#162] Introduce theories for type related tests in LuceneVersionCorrelationStoreTest https://github.com/lshift/diffa/commit/ab4326ce9cb13bc77eef801468d7...
-

Ben Hood January 26th, 2011 @ 11:36 AM
(from [bf2d919472d5c218800c570d14a779fbc06fbc39]) [#162] Strong type attributes and constraints to ensure range constraints can be successfully executed around integers. https://github.com/lshift/diffa/commit/bf2d919472d5c218800c570d14a7...
-

Ben Hood January 26th, 2011 @ 11:49 AM
(from [1574e250dfeb92e624b3ff34aa3a1c08df8123c3]) [#162] Ensure correlations returned via query mechanism are in ID order. https://github.com/lshift/diffa/commit/1574e250dfeb92e624b3ff34aa3a...
-

-

Paul Jones January 26th, 2011 @ 11:57 AM
- State changed from open to resolved
- Assigned user changed from Paul Jones to Ben Hood
Completed. I've also handle to bundle in typing of QueryConstraints with this, since it became necessary to make integer range queries work with Lucene.
-

Ben Hood January 26th, 2011 @ 02:01 PM
- State changed from resolved to open
- Assigned user changed from Ben Hood to Paul Jones
QA notes:
-
It would be good to be able to parameterize the version count when running the tests, so that we can create a TC configuration that runs once per day during the night that performs a very large test.
-
VersionCorrelationStorePerfTest:
- L36 contains dead code
-
LuceneVersionCorrelationStoreTest:
- Dead code in flushStore function
- unmatchPairFromUpstreamShouldBeIndicatedInReturnValue - is this spelt correctly?
- I need the use of the excludedStrAttrs in the AttributeSystem explained to me
- The (TODO) theorize tests should be marked with a ticket id
- queryUpstreamRangeReturnsInIDOrder - why can't this be theorized?
-
{DateRangeConstraint,IntegerRangeConstraint}:
- This contains little documentation
-
EndpointTest:
- Dead code in L54?
-
QueryConstraint:
- The docs could be a bit clearer - maybe the bullet point list of one item can go
-
LuceneVersionCorrelationStore:
- The class has no class level documentation
- What is the significance of the comment on L120 and L130?
- L157 case statement - can any other type of *RangeConstraint materialize in this clause?
- L168: Are IndexSearcher instances throw away items?
- When creating a new document, what is the pairWithId field for?
- val maxHits = 10000 - is this still being used?
- docToCorrelation - I think 1 line per field assignment is more readable than multiple assignments per line
-

-

Paul Jones January 26th, 2011 @ 02:57 PM
QA notes:
-
It would be good to be able to parameterize the version count when running the tests, so that we can create a TC configuration that runs once per day during the night that performs a very large test.- system property added -
VersionCorrelationStorePerfTest:
L36 contains dead code- put back into use
-
LuceneVersionCorrelationStoreTest:
Dead code in flushStore function- removedunmatchPairFromUpstreamShouldBeIndicatedInReturnValue - is this spelt correctly?- I need the use of the excludedStrAttrs in the AttributeSystem explained to me - the correlation currently stores Map[String, String], whereas the AttributeSystem has strongly typed attributes. That method just yanks out the string versions.
The (TODO) theorize tests should be marked with a ticket id- annotated with #164- queryUpstreamRangeReturnsInIDOrder - why can't this be theorized? - the actual underlying data types are irrelevant, so theorising them wouldn't provide any extra testing value
-
{DateRangeConstraint,IntegerRangeConstraint}:
This contains little documentation- documented
-
EndpointTest:
- Dead code in L54? - the comments are actually documenting the mapping from field names to values (eg, abc is abc_attribute). Previously, the values were self explanatory. Now, not so much - so I documented them.
-
QueryConstraint:
The docs could be a bit clearer - maybe the bullet point list of one item can go- Javadoc added
-
LuceneVersionCorrelationStore:
The class has no class level documentation- javadoc added- What is the significance of the comment on L120 and L130? - I've reworded the comment to make it clearer.
- L157 case statement - can any other type of RangeConstraint materialize in this clause?* - at this point, these are the only two constraint types we're supporting (and testing). Having it fail here at least makes the failure clearly defined.
- L168: Are IndexSearcher instances throw away items? - since we're passing in the reader, the actual close method is a no-op, so we're not doing any harm not closing them.
- When creating a new document, what is the pairWithId field for? - the updateDocument call only accepts a single term. Since our IDs are composite (pair and id), when we create a document, we put a composite key on it - just for the purposes of finding it to update.
val maxHits = 10000 - is this still being used?- no. gone.- docToCorrelation - I think 1 line per field assignment is more readable than multiple assignments per line - the intent of the current layout was to group similar fields together. If you like, I can break them up - I just though this conserved a bit of space by keeping related things together.
-
-

-

-

Ben Hood February 4th, 2011 @ 02:48 PM
- Milestone set to 0.9.3
- Milestone order changed from 11 to 0
Please Sign in or create a free account to add a new ticket.
With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป
A Real Time Differencing Tool.
People watching this ticket
Referenced by
-
162
Handling large numbers of entries
(from [bfd7265f26d86aa9b0d9ee15d309bc0e8332118c])
[#162] ...
-
162
Handling large numbers of entries
(from [6e130b8f9da5732e231d8d2f5541add6fa115f83])
[#162] ...
-
162
Handling large numbers of entries
(from [f694614c6a26b45673a410aceff9bacf725cac5f])
[#162] ...
-
162
Handling large numbers of entries
(from [3e659f1e93d4c73559936b31862d008b768f7268])
[#162] ...
-
162
Handling large numbers of entries
(from [ab4326ce9cb13bc77eef801468d787e2213a0374])
[#162] ...
-
162
Handling large numbers of entries
(from [bf2d919472d5c218800c570d14a779fbc06fbc39])
[#162] ...
-
162
Handling large numbers of entries
(from [1574e250dfeb92e624b3ff34aa3a1c08df8123c3])
[#162] ...
-
162
Handling large numbers of entries
(from [2f5684eb42eee9506d29e7912092a190ea9c013c])
[#162] ...
-
162
Handling large numbers of entries
(from [38d8da543fb6328bfb4bbbcc919be6afffbdc854])
[#162] ...
-
162
Handling large numbers of entries
(from [bfa809f2eee1f85e8f8cb56679e41a58c5eec102])
Merged ...