Fix CachedWeb handling of file urls - they have no http response code.

Review Request #40 — Created March 3, 2014 and submitted

jsirois
commons
jsirois/commons-python/fix-local-repo-caching
230
patricklaw, wickman
commit 5ac9104e9a130d06efe96654a022f22aebf383ed
Author: John Sirois <jsirois@twitter.com>
Date:   Mon Mar 3 13:16:46 2014 -0700

    Fix CachedWeb handling of file urls - they have no http response code.

 src/python/twitter/common/python/http/http.py        |  3 ++-
 tests/python/twitter/common/python/http/test_http.py | 16 ++++++++++++++++
 2 files changed, 18 insertions(+), 1 deletion(-)
New test before the fix:
$ ./pants tests/python/twitter/common/python/:all -v
...
============================================================================================================================== test session starts ===============================================================================================================================
platform linux3 -- Python 2.6.8 -- pytest-2.3.5 -- /home/jsirois/.pyenv/versions/2.6.8/bin/python
collected 8 items 

tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:33: test_http PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:39: test_resolve_timeout PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:54: test_http PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:65: test_http PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:70: test_local_cached_open FAILED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:85: test_maybe_local PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:116: test_connect_timeout_using_open PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:132: test_http PASSED

==================================================================================================================================== FAILURES ====================================================================================================================================
_____________________________________________________________________________________________________________________________ test_local_cached_open _____________________________________________________________________________________________________________________________

    def test_local_cached_open():
      cache = safe_mkdtemp()
      web = CachedWeb(cache=cache)
    
      source_dir = safe_mkdtemp()
      source = os.path.join(source_dir, 'filename')
      with open(source, 'w') as fp:
        fp.write('data')
    
>     with contextlib.closing(web.open(source)) as cached_fp1:
        assert 'data' == cached_fp1.read()

tests/python/twitter/common/python/http/test_http.py:79: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <twitter.common.python.http.http.CachedWeb object at 0x2aa9810>, url = '/tmp/tmpimypg_/filename', ttl = None, conn_timeout = None

    def open(self, url, ttl=None, conn_timeout=None):
      """Return a file-like object with the content of the url."""
      expired = self.expired(url, ttl=ttl)
      with TRACER.timed('Opening %s' % ('(cached)' if not expired else ''), V=1):
        if expired:
          try:
            self.cache(url, conn_timeout=conn_timeout)
          except (urllib_error.URLError, HTTPException) as exc:
            if not self._failsoft or url not in self:
>             raise FetchError(exc)
E             FetchError: <urlopen error Non-200 response code from /tmp/tmpimypg_/filename>

/tmp/tmpPm02n9/twitter/common/python/http/http.py:223: FetchError
======================================================================================================================= 1 failed, 7 passed in 0.19 seconds =======================================================================================================================
tests.python.twitter.common.python.http.test_crawler                            .....   SUCCESS
tests.python.twitter.common.python.http.test_http                               .....   FAILURE


And after the fix:
$ ./pants tests/python/twitter/common/python/:all -v
...
============================================================================================================================== test session starts ===============================================================================================================================
platform linux3 -- Python 2.6.8 -- pytest-2.3.5 -- /home/jsirois/.pyenv/versions/2.6.8/bin/python
collected 8 items 

tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:33: test_http PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:39: test_resolve_timeout PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:54: test_http PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:65: test_http PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:70: test_local_cached_open PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:85: test_maybe_local PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:116: test_connect_timeout_using_open PASSED
tests/python/twitter/common/python/http/test_http.py <- /home/jsirois/dev/3rdparty/commons/tests/python/twitter/common/python/http/test_http.py:132: test_http PASSED

============================================================================================================================ 8 passed in 0.17 seconds
...
WI
  1. Ship It!
  2. 
      
JS
JS
Review request changed

Status: Closed (submitted)

TR
  1. Ship It!
  2. 
      
Loading...