Re-upload artifacts that encountered read-errors

Review Request #1361 - Created Nov. 19, 2014 and submitted

Information
David Taylor
pants
794
3bb3644...
Reviewers
pants-reviews
benjyw, patricklaw, zundel

Before https://rbcommons.com/s/twitter/r/1272/, any artifact that wasn't successfully read either due to a normal cache-miss or corruption, would be re-compressed and uploaded.

While this generated a large amount of pointless work for clients and cache servers in the common case of normal cache-misses (particularly with thundering herds after changes that invalidated large numbers of cache keys), it did have the beneficial behavior of automatically causing corrupted artifacts to be replaced, if they were corrupted in such a way as to cause decompression errors.

Preserving the gains from that change (eg not re-uploading after a cache-miss if the artifact is in the cache by the time the client has it), we can restore that behvaior by keeping track of which cache keys were simply not in the cache vs those that were but could not be read, and when uploading artifacts, only skip existing ones that were simple missses.

We can do this without substantially changing the interface of use_cached_files by introducing a False-y return value that additionally indicates a read-error.

https://travis-ci.org/pantsbuild/pants/builds/41499518

Issues

  • 0
  • 1
  • 0
  • 1
Description From Last Updated
Patrick Lawson
David Taylor
David Taylor
Review request changed

Status: Closed (submitted)

Change Summary:

46c9fc3ddcc9ed6adb2ce3e204fd5e6c37d1ecac
Loading...