Refactor http handling in pex

Review Request #778 - Created July 28, 2014 and submitted

Brian Wickman (Twitter)
jsirois, kwlzn, patricklaw
This reworks http handling in pex to be more performant and to allow for alternate implementations and connection disciplines.  It also fixes the general flakiness around untranslatable packages.

The pex.http submodule is gone and each of its packages are moved into pex directly:

Crawler is out of the business of caching -- instead this is handed off to the http layer.

Link is out of the business of fetching -- it is now only a wrapper around a URL.

Web/CachedWeb is killed in favor of a new class pex.http.Context.  Subclasses need only implement 'open(link)' and return a file-like object.  There are three concrete implementations:
  - UrllibContext (python standard library http context)
  - RequestsContext (requests-based http context)
  - CachingRequestsContext (a requests-based http context with CacheControl if available)

The Requests-based contexts also support https cert validation and hash fragment verification (via StreamFilelike) bringing it up to security parity with pip.

The rest of the API is modified as minimally as possible to accommodate the above.  Users consuming the 'pex' binary and those who just use 'resolve' with default implementations will be unaffected.

Changes that will break pants:
  Obtainer now takes a context instead of a crawler (don't dwell on this too much -- Obtainer will be deleted altogether in the next review.)
  Translators no longer take conn_timeout since they no longer do any fetching -- this responsibility is delegated to the Context implementations.

Increments to 0.8.0-rc0.
pex.{crawler,link,http} have improved coverage over their predecessors.  The only thing I can think that might be worse is that UrllibContext does nothing to try to recover from errors -- it's mostly assumed that people will use the RequestsContext.
Kris Wilson
Brian Wickman (Twitter)
Brian Wickman (Twitter)
Kris Wilson
Brian Wickman (Twitter)
Review request changed

Status: Closed (submitted)