Re-upload artifacts that encountered read-errors
Review Request #1361 - Created Nov. 19, 2014 and submitted
|benjyw, patricklaw, zundel|
Before https://rbcommons.com/s/twitter/r/1272/, any artifact that wasn't successfully read either due to a normal cache-miss or corruption, would be re-compressed and uploaded.
While this generated a large amount of pointless work for clients and cache servers in the common case of normal cache-misses (particularly with thundering herds after changes that invalidated large numbers of cache keys), it did have the beneficial behavior of automatically causing corrupted artifacts to be replaced, if they were corrupted in such a way as to cause decompression errors.
Preserving the gains from that change (eg not re-uploading after a cache-miss if the artifact is in the cache by the time the client has it), we can restore that behvaior by keeping track of which cache keys were simply not in the cache vs those that were but could not be read, and when uploading artifacts, only skip existing ones that were simple missses.
We can do this without substantially changing the interface of use_cached_files by introducing a False-y return value that additionally indicates a read-error.
python 2/3 comment
Revision 2 (+48 -8)