Segment scans
Reported by Ben Hood | October 20th, 2011 @ 04:47 PM | in 1.0 (closed)
To avoid the cost of a running a complete scan, it should be possible to segment a scan for configured bounds into smaller sub-scans. It should then be possible to initiate a scan that will only request participant queries which lie within the confines of the constrained view. This means that the configuration for a pair can have very wide bounds (or even be boundless in some dimensions). Pairings could have a number of views configured into them which can be triggered separately and either manually, as a scheduled job or both.
This means that real time events can be fed into into a pair with no regard of views, the partitioning consideration would be the incoming event fits into the configured bounds of the pair, which now could be quite wide or even unbounded.
It should also be possible to declare the pair to be unscannable - this would provide protection against somebody accidentally triggering the widest possible scan.
Potential extensions to this work could include:
- Auto-segmenting the scan of a wide partition into n sequential subscans, configured by some kind of portioning methodology (e.g. scan a pair in monthly/weekly/daily segments)
- Specifying views on the fly - these need not necessarily be persistent, they could just be one-shot views;
- Specifying a view date range as a period offset to the current system time, e.g. a view for the last 24 hours, for the last week, etc
- Being able to trigger a view in the UI;
- Triggering a one-shot view from the UI;
Comments and changes to this ticket
-
Ben Hood October 20th, 2011 @ 06:20 PM
- Tag set to “optimizations, scanning”
-
Ben Hood October 20th, 2011 @ 06:25 PM
- Milestone cleared.
- Assigned user set to “Paul Jones”
- Milestone order changed from “139” to “0”
-
Ben Hood October 21st, 2011 @ 06:21 AM
(from [41552d3c2b7e77a43035cf4ad6f6481dc6be5d79]) [#395] Add basic data model for storing views. https://github.com/lshift/diffa/commit/41552d3c2b7e77a43035cf4ad6f6...
-
Ben Hood October 21st, 2011 @ 06:55 AM
(from [f672cf041e53bd38596832b5ff858bb2527ef7af]) [#395] Support reading/writing views in XML. https://github.com/lshift/diffa/commit/f672cf041e53bd38596832b5ff85...
-
Ben Hood October 24th, 2011 @ 07:31 AM
(from [f4e8482aa6a1ed0491ee7749c490b7506453432f]) [#395] Add database migration for views, plumb through their use and selection. https://github.com/lshift/diffa/commit/f4e8482aa6a1ed0491ee7749c490...
-
Ben Hood October 24th, 2011 @ 09:17 PM
(from [d569a5b6afee177c850e538649b142e93ce87f1d]) [#395] Allow scans to be initiated with a specific view. https://github.com/lshift/diffa/commit/d569a5b6afee177c850e538649b1...
-
Ben Hood October 25th, 2011 @ 02:45 AM
(from [be16154ff4870c8b35e769298727883e4e875ed0]) [#395] Add end-to-end test of view scan.
- Test persistence of view configuration, actually make content persist. https://github.com/lshift/diffa/commit/be16154ff4870c8b35e769298727...
-
Ben Hood October 25th, 2011 @ 11:51 AM
(from [8741c08b41af953267cefc8c2ff2e36cfedb63e1]) [#395] Add support for scanning views to settings page.
- Add some dummy views to the web participants.
- Change encapsulation for view name to be form param instead of query param. https://github.com/lshift/diffa/commit/8741c08b41af953267cefc8c2ff2...
-
Ben Hood October 25th, 2011 @ 11:37 PM
(from [42f2c9d2d27bb52f5c87536e008b9a03608d01ac]) [#395] QA note: s/Categorised/Categorized/gc https://github.com/lshift/diffa/commit/42f2c9d2d27bb52f5c87536e008b...
-
Ben Hood October 30th, 2011 @ 11:05 PM
(from [660e0d5d3d03cbe07a9c567b7f2155b082d9feec]) [#395] QA note: Validate that view categories are within bounds of endpoint categories. https://github.com/lshift/diffa/commit/660e0d5d3d03cbe07a9c567b7f21...
-
Ben Hood October 30th, 2011 @ 11:43 PM
(from [2d8e744d2c6aa90138d949189a8eda5a8d9b154a]) [#395] QA note: Add validation that pair views are defined on endpoints too. https://github.com/lshift/diffa/commit/2d8e744d2c6aa90138d949189a8e...
-
Ben Hood October 31st, 2011 @ 01:46 AM
(from [0dc18e1c552ad35f7a5ddf66c57f2fcc1cc1ad7b]) [#395] Support scheduling of cron specs for views. https://github.com/lshift/diffa/commit/0dc18e1c552ad35f7a5ddf66c57f...
-
Paul Jones October 31st, 2011 @ 01:48 AM
- Assigned user changed from “Paul Jones” to “Ben Hood”
- State changed from “new” to “resolved”
Ready for QA.
Includes:
- Configuration of views on endpoints & pairs;
- Validation that the configuration provided for views is workable;
- Execution of scans on views (both in the REST API and via the UI);
- Scheduling of view scans;
-
-
Ben Hood October 31st, 2011 @ 07:39 AM
- State changed from “resolved” to “closed”
-
Ben Hood November 5th, 2011 @ 11:35 AM
- Milestone set to “1.0”
- Milestone order changed from “33” to “0”
Please Sign in or create a free account to add a new ticket.
With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.
Create new ticket
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป
A Real Time Differencing Tool.
Ben Hood