Refactor http handling in pex

Review Request #778 — Created July 29, 2014 and submitted

jsirois, kwlzn, patricklaw
This reworks http handling in pex to be more performant and to allow for alternate implementations and connection disciplines.  It also fixes the general flakiness around untranslatable packages.

The pex.http submodule is gone and each of its packages are moved into pex directly:

Crawler is out of the business of caching -- instead this is handed off to the http layer.

Link is out of the business of fetching -- it is now only a wrapper around a URL.

Web/CachedWeb is killed in favor of a new class pex.http.Context.  Subclasses need only implement 'open(link)' and return a file-like object.  There are three concrete implementations:
  - UrllibContext (python standard library http context)
  - RequestsContext (requests-based http context)
  - CachingRequestsContext (a requests-based http context with CacheControl if available)

The Requests-based contexts also support https cert validation and hash fragment verification (via StreamFilelike) bringing it up to security parity with pip.

The rest of the API is modified as minimally as possible to accommodate the above.  Users consuming the 'pex' binary and those who just use 'resolve' with default implementations will be unaffected.

Changes that will break pants:
  Obtainer now takes a context instead of a crawler (don't dwell on this too much -- Obtainer will be deleted altogether in the next review.)
  Translators no longer take conn_timeout since they no longer do any fetching -- this responsibility is delegated to the Context implementations.

Increments to 0.8.0-rc0.
pex.{crawler,link,http} have improved coverage over their predecessors.  The only thing I can think that might be worse is that UrllibContext does nothing to try to recover from errors -- it's mostly assumed that people will use the RequestsContext.
  1. seems legit.
  2. pex/ (Diff revision 1)
    curious - any reason to not contain these crawl* funcs as methods of the Crawler class?
    1. No particularly great reason.  I can consolidate.
  3. pex/ (Diff revision 1)
    no pass necessary since you provide a docstring (and above on line 46).
  1. Ship It!
Review request changed

Status: Closed (submitted)