Support caching of zoomed data
Reported by Paul Jones | August 15th, 2011 @ 04:57 PM | in 0.9-series (closed)
Zoomed data is currently re-calculated on every request, regardless of the changes that have occurred. This can be very expensive, and isn't easily cached when users can zoom or pan.
The proposed design for resolving this is:
- Consider the heatmap to be made up of a series of tiles;
- When the heatmap is being rendered, generate the zoom response by assembling the visible tiles (where a tile is a cached time slice of aggregates at a given zoom level). Tiles are created on demand, and stored in a LRU cache;
- Zooming currently consists of doubling or halving the current view, allowing for many different zooms. To make the caching work more effectively, these zoom levels should be fixed to a given set of values;
- When slicing a heatmap into tiles, the tiles should be time aligned to ensure that as time passes the tiles don't need to be rebuilt;
- Cached tiles can be purged whenever a write is done against the store;
Follow up tickets:
- #354 for more intelligent zoom requesting
Comments and changes to this ticket
-

Ben Hood August 17th, 2011 @ 04:07 PM
- State changed from new to open
- Assigned user set to Ben Hood
-

Ben Hood August 17th, 2011 @ 06:34 PM
(from [a02ecb3052985d0b092c1d3838fd839c94f09f0b]) [#341] Commenced work on zooming tiles https://github.com/lshift/diffa/commit/a02ecb3052985d0b092c1d3838fd...
-

-

-

-

-

-

-

Ben Hood August 19th, 2011 @ 08:18 PM
(from [c3c746e5a5fec52370119c678df25315cf9134d5]) [#341] Added test for previous chronological event https://github.com/lshift/diffa/commit/c3c746e5a5fec52370119c678df2...
-

-

Ben Hood August 23rd, 2011 @ 07:02 PM
(from [a4138301a33baa98c0732336fd15f98643e60ce2]) [#341] Merged persistent-diffs into 341 https://github.com/lshift/diffa/commit/a4138301a33baa98c0732336fd15...
-

Ben Hood August 23rd, 2011 @ 08:21 PM
(from [472b31b4143024c5a87080c15e607fd47d23b759]) [#341] Put observation point in time into tile interface https://github.com/lshift/diffa/commit/472b31b4143024c5a87080c15e60...
-

Ben Hood August 23rd, 2011 @ 08:25 PM
(from [5443936060dc2a6a8da2703e3e7f0e7194f8880e]) [#341] Put commented out tests back in https://github.com/lshift/diffa/commit/5443936060dc2a6a8da2703e3e7f...
-

-

Ben Hood August 24th, 2011 @ 03:27 AM
(from [e57577814a011c7303fd008577f951345ceb2d21]) [#341] Added more slots to level 0 data point https://github.com/lshift/diffa/commit/e57577814a011c7303fd008577f9...
-

Ben Hood August 24th, 2011 @ 03:34 AM
(from [46f3bbd4b5a767439cf6443ea80b781aed3da2a5]) [#341] Added datapoint for different pair https://github.com/lshift/diffa/commit/46f3bbd4b5a767439cf6443ea80b...
-

Ben Hood August 24th, 2011 @ 06:40 PM
(from [8946dcc73e1d46e90ad4979695f1e18887420b37]) [#341] Added more data points to test https://github.com/lshift/diffa/commit/8946dcc73e1d46e90ad4979695f1...
-

-

Ben Hood August 25th, 2011 @ 09:06 AM
(from [dd407b830ec83708be75f03772885b4e74d5de97]) [#341] Keyed on previous detecion time https://github.com/lshift/diffa/commit/dd407b830ec83708be75f0377288...
-

Ben Hood August 25th, 2011 @ 10:26 AM
(from [046efc4e5e8597511d66c9c5cdae24335de31d10]) [#341] Play through events at each zoom level https://github.com/lshift/diffa/commit/046efc4e5e8597511d66c9c5cdae...
-

-

Ben Hood August 25th, 2011 @ 01:59 PM
(from [c18826afabc8f583a17eef8766dfdbfe87d76c0a]) [#341] Clear caches as part of clearing differences https://github.com/lshift/diffa/commit/c18826afabc8f583a17eef8766df...
-

Ben Hood August 25th, 2011 @ 04:15 PM
(from [76dbc5b61656704960f37518e1ad482b00117279]) [#341] Cleanup up zoom caches as part of pair/domain removal https://github.com/lshift/diffa/commit/76dbc5b61656704960f37518e1ad...
-

-

Ben Hood August 25th, 2011 @ 05:09 PM
(from [0a5a13fd8e0b574cce880cdffb6d4fedce71d77c]) [#341] Documented the zoom cache https://github.com/lshift/diffa/commit/0a5a13fd8e0b574cce880cdffb6d...
-

Ben Hood August 25th, 2011 @ 05:17 PM
(from [a33ea698929d94c16b4867074ae69b2c91ea74b1]) [#341] Defer the dirty flag creation https://github.com/lshift/diffa/commit/a33ea698929d94c16b4867074ae6...
-

Ben Hood August 26th, 2011 @ 08:44 AM
(from [64a6574fd10bc0a6b02e337c2e518539f8756569]) [#341] QA remark: moved cache wrapper into separate util package https://github.com/lshift/diffa/commit/64a6574fd10bc0a6b02e337c2e51...
-

Ben Hood August 26th, 2011 @ 10:07 AM
(from [9263017b1e57295e8c14b2be4b52b4352994bdc4]) [#341] QA remarks: Pass in explicit timestamp for onStoreUpdate, renamed start tile function https://github.com/lshift/diffa/commit/9263017b1e57295e8c14b2be4b52...
-

Ben Hood August 27th, 2011 @ 02:44 AM
(from [6daa8ecd91244e0c31b3c12ebe390b6393cb5017]) [#341] Started to investigate whether absolute times could be used instead of indexes https://github.com/lshift/diffa/commit/6daa8ecd91244e0c31b3c12ebe39...
-

Ben Hood August 27th, 2011 @ 12:43 PM
(from [e7f6544a44c7871a21ed3500ee105154c6e89755]) [#341] Marking dirty tiles is based on the start of the interval a timestamp falls into https://github.com/lshift/diffa/commit/e7f6544a44c7871a21ed3500ee10...
-

Ben Hood August 27th, 2011 @ 07:04 PM
(from [0f6fc90d61b58b6659e15f50a28f5a3739d77db6]) [#341] Added test for containing interval https://github.com/lshift/diffa/commit/0f6fc90d61b58b6659e15f50a28f...
-

Ben Hood August 27th, 2011 @ 07:09 PM
(from [701083c4fc8cd64a51f49dc9f30ce21695f700ad]) [#341] Added test for starting interval https://github.com/lshift/diffa/commit/701083c4fc8cd64a51f49dc9f30c...
-

-

Ben Hood August 27th, 2011 @ 07:18 PM
(from [1d4c11e6b14d6966f2575369d859d1f681947d95]) [#341] Trimmed down ZoomCache test https://github.com/lshift/diffa/commit/1d4c11e6b14d6966f2575369d859...
-

-

-

-

Ben Hood August 27th, 2011 @ 10:31 PM
(from [87ea358678c360a8873410b5a7a7752e959ea521]) [#341] Get rid of more dead code https://github.com/lshift/diffa/commit/87ea358678c360a8873410b5a7a7...
-

Ben Hood August 29th, 2011 @ 11:30 AM
(from [c20fcb42079181b51027a7e02117551fa7b3419e]) [#341] Tile requests are time bound https://github.com/lshift/diffa/commit/c20fcb42079181b51027a7e02117...
-

Ben Hood August 29th, 2011 @ 12:09 PM
(from [2e33d92d43b89da39649d09d384088ce947b98f4]) [#341] Expected tile range constraints test to all zoom levels https://github.com/lshift/diffa/commit/2e33d92d43b89da39649d09d3840...
-

Ben Hood August 29th, 2011 @ 12:17 PM
(from [f2902e606971561948c3e48129654736d41c60e9]) [#341] Wire tiled events through the differences manager https://github.com/lshift/diffa/commit/f2902e606971561948c3e4812965...
-

-

Ben Hood August 29th, 2011 @ 12:29 PM
- Tag changed from persistent session events to persistent session events, zooming
-

Ben Hood August 29th, 2011 @ 12:47 PM
(from [d9b816e4d2ca95afd07cd6b13012d5075a1e198e]) [#341] Addressed QA remark about not limiting singleQueryOpts globally https://github.com/lshift/diffa/commit/d9b816e4d2ca95afd07cd6b13012...
-

Ben Hood August 29th, 2011 @ 04:50 PM
(from [a1ab2d93ed4c32238a6507853875d82418208e9e]) [#341] Wire through tiling to itest https://github.com/lshift/diffa/commit/a1ab2d93ed4c32238a6507853875...
-

Ben Hood August 29th, 2011 @ 06:07 PM
(from [efd957c537962dfc9f6f346fedfc164bd570a5b3]) [#341] Changed cache construction to use a DB cursor and build the aggregate itself https://github.com/lshift/diffa/commit/efd957c537962dfc9f6f346fedfc...
-

-

Ben Hood August 30th, 2011 @ 09:09 PM
(from [0a75932b221d5c9bf120da0ca122b6dce2d5168c]) [#341] Added test to calculate containing aligned grouped tiles https://github.com/lshift/diffa/commit/0a75932b221d5c9bf120da0ca122...
-

Ben Hood August 30th, 2011 @ 09:48 PM
(from [552042931f6e267cd51d9bbdef23f7556cab154b]) [#341] Added test to slice time https://github.com/lshift/diffa/commit/552042931f6e267cd51d9bbdef23...
-

-

-

Ben Hood August 31st, 2011 @ 06:17 PM
(from [18175879869634804bae4dd6976defbabff2c48c]) [#341] Made diff manager test theoretical https://github.com/lshift/diffa/commit/18175879869634804bae4dd6976d...
-

Ben Hood August 31st, 2011 @ 06:58 PM
(from [ab989e0d2807116af33c50d1df4c83a5c9e10142]) [#341] Fixed bug in containing interval alignment https://github.com/lshift/diffa/commit/ab989e0d2807116af33c50d1df4c...
-

Ben Hood August 31st, 2011 @ 07:07 PM
(from [76e9dd06ddd342eeb211a6f9b0850ebe279e1183]) [#341] Added a half hourly datapoint for tiles https://github.com/lshift/diffa/commit/76e9dd06ddd342eeb211a6f9b085...
-

Ben Hood August 31st, 2011 @ 08:33 PM
(from [bd0d9da38c1eb1e5cb9d1652f89d90eccdac2b43]) [#341] Added test case for intraday sub hourly queries https://github.com/lshift/diffa/commit/bd0d9da38c1eb1e5cb9d1652f89d...
-

Ben Hood August 31st, 2011 @ 08:44 PM
(from [855fbccae4491d2029c5bf1e2ebb650215dda50b]) [#341] Introduce ability to have multiple events in a scenario https://github.com/lshift/diffa/commit/855fbccae4491d2029c5bf1e2ebb...
-

Ben Hood August 31st, 2011 @ 08:55 PM
(from [73bc2489ab1e0f1ef909f5e176930090bddd0d37]) [#341] Added data point for a sub day spanned query https://github.com/lshift/diffa/commit/73bc2489ab1e0f1ef909f5e17693...
-

Ben Hood August 31st, 2011 @ 09:22 PM
(from [0acb6f317d8f49a071013dff0a79393cc9aff1b6]) [#341] Made aligned interval containment more tolerant https://github.com/lshift/diffa/commit/0acb6f317d8f49a071013dff0a79...
-

Ben Hood August 31st, 2011 @ 09:49 PM
(from [972adcb367cc30b06145924a41fcde858e9b692e]) [#341] Added datapoint for daily granularity https://github.com/lshift/diffa/commit/972adcb367cc30b06145924a41fc...
-

-

-

Ben Hood September 1st, 2011 @ 10:09 AM
(from [c6f6ef161a543aa819196ac063c788e1bd946881]) [#341] Added test for containing group interval https://github.com/lshift/diffa/commit/c6f6ef161a543aa819196ac063c7...
-

Ben Hood September 1st, 2011 @ 10:13 AM
(from [7ef2eb0b99a3d607567334c78cd382fb2b493a0d]) [#341] Added some more data points https://github.com/lshift/diffa/commit/7ef2eb0b99a3d607567334c78cd3...
-

Ben Hood September 1st, 2011 @ 10:34 AM
(from [535710bbe5d74837d31bbc629fdf28d61551206b]) [#341] Fixed array deserializiation bug https://github.com/lshift/diffa/commit/535710bbe5d74837d31bbc629fdf...
-

Ben Hood September 1st, 2011 @ 10:41 AM
(from [34aef6d639d80e56fce9f55667cb8c30722fc589]) [#341] Added half hourly data point https://github.com/lshift/diffa/commit/34aef6d639d80e56fce9f55667cb...
-

Ben Hood September 1st, 2011 @ 11:07 AM
(from [1a99813fcb1c68722ec32e3d7aef00d4e24bd694]) [#341] Fixed in group tile interval for daily granularity https://github.com/lshift/diffa/commit/1a99813fcb1c68722ec32e3d7aef...
-

-

Ben Hood September 1st, 2011 @ 12:32 PM
(from [64fdd40700efcb7fa028b7fea86194b1ce1a2e2d]) [#341] Qualify the dirty flags by the tile group timestamp https://github.com/lshift/diffa/commit/64fdd40700efcb7fa028b7fea861...
-

Ben Hood September 1st, 2011 @ 12:56 PM
(from [49ca6f7fbade4236a087c3f0fdc7326f3036c427]) [#341] Put eventsShouldUpdateZoomCache back in https://github.com/lshift/diffa/commit/49ca6f7fbade4236a087c3f0fdc7...
-

Ben Hood September 1st, 2011 @ 02:22 PM
(from [a81e64f039354b42f8e922ccf9118c065cdf8a8d]) [#341] Reactivate posterity test https://github.com/lshift/diffa/commit/a81e64f039354b42f8e922ccf911...
-

Ben Hood September 1st, 2011 @ 03:17 PM
(from [39d87c24d4f946f0d40ff63c6b41ab3c22675971]) [#341] Fixed bug where events from different tile ranges would set dirty flags for other timestamps https://github.com/lshift/diffa/commit/39d87c24d4f946f0d40ff63c6b41...
-

Ben Hood September 1st, 2011 @ 05:14 PM
(from [fe11df05d174b4d60816533ca53ec2882a7a0d0c]) [#341] Bound zoom levels into the UI https://github.com/lshift/diffa/commit/fe11df05d174b4d60816533ca53e...
-

Ben Hood September 1st, 2011 @ 05:20 PM
(from [74e10828ff1fcd9eac23f314f4fcffbb599a24b2]) [#341] Addressed QA remark about InvalidZoomLevelException mapping https://github.com/lshift/diffa/commit/74e10828ff1fcd9eac23f314f4fc...
-

-

Ben Hood September 1st, 2011 @ 05:35 PM
(from [45d4b4a2de8bbc929e1b9f540805c947e9526eef]) [#341] Last commit was applied to wrong mapper https://github.com/lshift/diffa/commit/45d4b4a2de8bbc929e1b9f540805...
-

Ben Hood September 1st, 2011 @ 05:45 PM
(from [065a3db2e93e2ed11a5fcb0794df93d28823385c]) [#341] Jersey doesn't appear to like mapping int values https://github.com/lshift/diffa/commit/065a3db2e93e2ed11a5fcb0794df...
-

Ben Hood September 2nd, 2011 @ 09:52 AM
(from [0e87e8c2341f87268c822427b604023d6198522b]) [#341] Force the time zone handling to UTC, put the zoom level into the etag https://github.com/lshift/diffa/commit/0e87e8c2341f87268c822427b604...
-

Ben Hood September 2nd, 2011 @ 02:21 PM
(from [125421e5b808f74fe75bf636c58ec8cc2e89b96d]) [#341] Fixed time zone bug and adjusted tests accordingly https://github.com/lshift/diffa/commit/125421e5b808f74fe75bf636c58e...
-

Ben Hood September 2nd, 2011 @ 03:52 PM
(from [08b63e20f50d987e664eb16ecb5cdea0f9fcda27]) [#341] Cross-day spans should be aligned to the default grouping level for quarter hourly and half hourly granularity https://github.com/lshift/diffa/commit/08b63e20f50d987e664eb16ecb5c...
-

-

Ben Hood September 2nd, 2011 @ 05:32 PM
(from [5d61fed7100059190c758ae375237032893602fa]) [#341] Get 96 columns into the UI, not 97 https://github.com/lshift/diffa/commit/5d61fed7100059190c758ae37523...
-

Ben Hood September 2nd, 2011 @ 05:49 PM
(from [010d16d08617848f2d6ec8e621fa72b04b1e01c2]) [#341] Close the DB cursor explicitly https://github.com/lshift/diffa/commit/010d16d08617848f2d6ec8e621fa...
-

-

Ben Hood September 2nd, 2011 @ 06:08 PM
- State changed from open to resolved
- Assigned user changed from Ben Hood to Paul Jones
Resolving for QA
-

Ben Hood September 3rd, 2011 @ 04:33 AM
(from [77043af9ff89065fb8010067c22f60e31bbe0175]) [#341] Bind zoom control updating to the right event so it actually triggers. https://github.com/lshift/diffa/commit/77043af9ff89065fb8010067c22f...
-

Ben Hood September 3rd, 2011 @ 04:33 AM
(from [69490be0dc232de0bf74b4dd6097122a4b2d34d5]) [#341] Don't count matched events. https://github.com/lshift/diffa/commit/69490be0dc232de0bf74b4dd6097...
-

Ben Hood September 3rd, 2011 @ 05:04 AM
(from [c8801b957fb76c63f6be77aa65159ba8e640783c]) Merge [#341] into persistent-diffs https://github.com/lshift/diffa/commit/c8801b957fb76c63f6be77aa6515...
-

Paul Jones September 3rd, 2011 @ 05:06 AM
- State changed from resolved to closed
I added a few patches to correct a JS-based UI zooming bug (that I think might have existed for a while), and also to correct the fact that the count method was returning matched events as well as unmatched events.
=> QA+
-

Ben Hood November 14th, 2011 @ 05:35 PM
- Milestone set to 0.9-series
- Milestone order changed from 111 to 0
Please Sign in or create a free account to add a new ticket.
With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป
A Real Time Differencing Tool.