RESTfulCache delegates tarfile handling to LocalArtifactCache

Review Request #1233 - Created Oct. 27, 2014 and submitted

Information
David Taylor
pants
737
e6550d7...
Reviewers
pants-reviews
benjyw, johanoskarsson, patricklaw, zundel

The motivation for this change is to allow artifacts downloaded by RESTfulCache to be directly added to the LocalArtifactCache without needing to decompress and recompress them as we do now.

At a high level, this is done by refactoring RESTfulCache such that it no longer handles creating or extracting tarfiles directly and instead delegates those operations to a LocalArtifactCache instance passed in its constructor, which may store them in the process.

This is a revision of a previous proposal (https://rbcommons.com/s/twitter/r/985/) along similar lines, though this iteration has RESTful artifact cache always passed a LocalArtifactCache at construction, to which it can always delegate all local file interaction (tarfile creation and extraction)

This simplifies RESTful cache, reducing it to only the HTTP-related functionality. This also eliminates duplication of tarfile creation and extraction logic, which now justs lives in the LocalArtifactCache.

TempLocalArtifactCache provides an alternate implementation of LocalArtifactCache, backed by tmpfiles instead of a cache_root. This can be passes to RESTfulCache to handle artifacts when running in a remote-only cache configuration.

Notes:

This change removes all usage of CombinedCache -- the only case where it was used,
composing an HTTP and local cache, is covered by the above changes.

It also removes the "backfill" behavior from CombinedCache, which only made sense in the above setup: two remote caches should likely use rsyncd or whatever to share files rather than relying on clients building to shuttle artifacts between them and local caches can use symlinks or mounts.

Caches also now use stdlib logging instead of context.log since the latter
has references to the runtracker making it almost impossible to pickle.

Caches now use string.format instead of % for composing log messages.

Add more Sphinx style docstrings.

https://travis-ci.org/pantsbuild/pants/builds/39519073 passed

Issues

  • 0
  • 15
  • 0
  • 15
Description From Last Updated
Patrick Lawson
Eric Ayers
David Taylor
David Taylor
Eric Ayers
Eric Ayers
Benjy Weinberger
David Taylor
Eric Ayers
David Taylor
David Taylor
Review request changed

Status: Closed (submitted)

Change Summary:

edcd459b9cf96c3e5ffc8d6965df5f2e731b3b84
Benjy Weinberger

   

Now the naming is weird. Maybe we can get rid of this and just throw ArtifactCacheError directly instead?

  1. David Turner beat me to it: https://rbcommons.com/s/twitter/r/1261/

Loading...