Client-side Caching

[gordonwatts]

• Caching code needs testing coverage
• Code must be robust against crashes and network loss mid-transform
• Robust against switching between download files and signed url

[kyungeonchoi]

Github issues on caching

Bug

• Changing Sample's name caues a cache miss iris-hep/idap-200gbps-atlas#135
• Cache Collisions when you ignore cache for a re-run #375
• Cached data do not use the sample name from the new spec #418
• If USER isn't defined, then servicex crashes #213
• Caching behavior of permutation of XRootDFiles #396
• Add locks to protect cache DB from concurrent writes #361
• Default cache dir #285
• No network access required for fully cached query #295

Feature

• Allow per-request override of cache area #383
• Frontend crashes while working transform is in progress are not saved #468
• Client-side Caching #280 (This one)

Working branches

• Don't used cached results if there were any failures #318
  • Solve Add locks to protect cache DB from concurrent writes #361

Wish list for 3.0

• Code must be robust against crashes and network loss mid-transform (Gordon) Frontend crashes while working transform is in progress are not saved #468
• Robust against switching between download files and signed url (Gordon)
• Caching code needs testing coverage (Gordon)
• Download to non-temporary space (KyungEon)

[gordonwatts]

After 3.0.1 triage, turned @kyungeonchoi 's list into:

Github issues on caching

Bug

• Cache Collisions when you ignore cache for a re-run #375
• Cached data do not use the sample name from the new spec #418
• If USER isn't defined, then servicex crashes #213
• Caching behavior of permutation of XRootDFiles #396
• Add locks to protect cache DB from concurrent writes #361
• Default cache dir #285
• No network access required for fully cached query #295

In progress updates to pacakge

• Frontend crashes while working transform is in progress are not saved #468
• Client-side Caching #280 (This one)

So this can be closed as soon as the last issue (#468) is closed.

[ponyisi]

All the 3.0.1 fixes are in, so closing.