don't always patch the current interpreter into the compatible interpreters set.

Review Request #82 — Created March 8, 2014 and submitted

wickman
commons
pants-reviews
travis
don't always patch the current interpreter into the compatible interpreters set.


TR
  1. I patched this in and am getting the same issue. I'll try to take a look at this or come and grab you if I get stuck.
    1. Taking a look at this as I'd like to get CI green again. Here's a full log of what I see with this patched in locally.
      
      https://gist.github.com/traviscrawford/8ae00acfb0feb4c95d1e
      
      Error:
      
      ==================================================================================== test session starts ====================================================================================
      platform darwin -- Python 2.6.5 -- py-1.4.20 -- pytest-2.5.2
      plugins: cov
      collected 19 items 
      
      tests/python/twitter/common/log/test_tracer.py ..
      tests/python/twitter/common/log/test_reader.py ...
      tests/python/twitter/common/log/test_scribe_handler.py ssssssss.sss.s
      
      =========================================================================== 7 passed, 12 skipped in 0.46 seconds ============================================================================
      Failed to install stdout:
      
      Failed to install stderr:
      Traceback (most recent call last):
        File "<stdin>", line 4, in <module>
        File "setup.py", line 2, in <module>
          from setuptools import setup
      ImportError: No module named setuptools
      
      Failed to run test!
      Traceback (most recent call last):
        File "/Users/travis/Python/CPython-2.6.8/lib/python2.6/site-packages/twitter/pants/python/test_builder.py", line 204, in _run_python_test
          builder = chroot.dump()
        File "/Users/travis/Python/CPython-2.6.8/lib/python2.6/site-packages/twitter/pants/python/python_chroot.py", line 169, in dump
          self.MEMOIZED_THRIFTS[thr] = self._generate_thrift_requirement(thr)
        File "/Users/travis/Python/CPython-2.6.8/lib/python2.6/site-packages/twitter/pants/python/python_chroot.py", line 140, in _generate_thrift_requirement
          return self._generate_requirement(library, PythonThriftBuilder)
        File "/Users/travis/Python/CPython-2.6.8/lib/python2.6/site-packages/twitter/pants/python/python_chroot.py", line 131, in _generate_requirement
          sdist = builder.build(interpreter=self._interpreter)
        File "/Users/travis/Python/CPython-2.6.8/lib/python2.6/site-packages/twitter/pants/python/code_generator.py", line 103, in build
          return SdistBuilder.build(self.sdist_root, self.target, interpreter=interpreter)
        File "/Users/travis/Python/CPython-2.6.8/lib/python2.6/site-packages/twitter/pants/python/sdist_builder.py", line 41, in build
          raise cls.SetupError(str(e))
      SetupError: Failed to install /Users/travis/src/commons/dist/py-scribe.A6WnPf/codegen.d6zX8F
    2. First issue with the above stack trace was somehow I installed pants into my site-packages. pip uninstalled that stuff.
      
      This reproduces the issue:
      ./build-support/python/clean.sh && ./pants.bootstrap tests/python/twitter/common/log:test_log_with_scribe -k TestHandler -v
      
      I added these debug lines and we see setuptools is importable just before the call that fails, so I need to look at what's different about the context in which that call executes.
      
      [tw-mbp13-travis commons]$ git diff
      diff --git a/src/python/twitter/pants/python/sdist_builder.py b/src/python/twitter/pants/python/sdist_builder.py
      index f35f693..0f165d7 100644
      --- a/src/python/twitter/pants/python/sdist_builder.py
      +++ b/src/python/twitter/pants/python/sdist_builder.py
      @@ -36,6 +36,8 @@ class SdistBuilder(object):
           packager = Packager(setup_root, interpreter=interpreter,
               install_dir=os.path.join(setup_root, 'dist'))
           try:
      +      import setuptools
      +      print(dir(setuptools))
             return packager.sdist()
           except Packager.InstallFailure as e:
             raise cls.SetupError(str(e))
      [tw-mbp13-travis commons]$ 
      
      ERROR
      
      Wrote /Users/travis/src/commons/dist/pants.pex
      Build operating on targets: OrderedSet([PythonTests(tests/python/twitter/common/log/BUILD:test_log_with_scribe)])
      ['Command', 'Distribution', 'Extension', 'Feature', 'Require', '_Command', '__all__', '__builtins__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__version__', '_dont_write_bytecode', '_get_unpatched', 'bootstrap_install_from', 'compat', 'convert_path', 'depends', 'dist', 'distutils', 'extension', 'find_packages', 'findall', 'lib2to3_fixer_packages', 'os', 'run_2to3_on_doctests', 'setup', 'setuptools', 'sys', 'version']
      Failed to install stdout:
      
      Failed to install stderr:
      Traceback (most recent call last):
        File "<stdin>", line 4, in <module>
        File "setup.py", line 2, in <module>
          from setuptools import setup
      ImportError: No module named setuptools
      
      Failed to run test!
      Traceback (most recent call last):
        File "/Users/travis/src/commons/pants.pex/twitter/pants/python/test_builder.py", line 204, in _run_python_test
          builder = chroot.dump()
        File "/Users/travis/src/commons/pants.pex/twitter/pants/python/python_chroot.py", line 169, in dump
          self.MEMOIZED_THRIFTS[thr] = self._generate_thrift_requirement(thr)
        File "/Users/travis/src/commons/pants.pex/twitter/pants/python/python_chroot.py", line 140, in _generate_thrift_requirement
          return self._generate_requirement(library, PythonThriftBuilder)
        File "/Users/travis/src/commons/pants.pex/twitter/pants/python/python_chroot.py", line 131, in _generate_requirement
          sdist = builder.build(interpreter=self._interpreter)
        File "/Users/travis/src/commons/pants.pex/twitter/pants/python/code_generator.py", line 103, in build
          return SdistBuilder.build(self.sdist_root, self.target, interpreter=interpreter)
        File "/Users/travis/src/commons/pants.pex/twitter/pants/python/sdist_builder.py", line 43, in build
          raise cls.SetupError(str(e))
      SetupError: Failed to install /Users/travis/src/commons/dist/py-scribe.bZ31WL/codegen.jGCneA
      tests.python.twitter.common.log.test_log_with_scribe                            ..... EXCEPTION
      pants.pex version e38cde6aa5912199f99d319780de24b3d8311437
      [tw-mbp13-travis commons]$ 
      
    3. Can you "print(interpreter)" there?  It should indicate if it has setuptools and which one it's using.
    4. Oof. Looking to see where this comes from.
      
      =============> tcrawford debug Interpreter: PythonInterpreter('/opt/twitter/bin/python', PythonIdentity('CPython', 2, 6, 5), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/wheel-0.22.0-py2.6.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/setuptools-2.2-py2.6.egg'})
      
      I get that error when doing:
      ./build-support/python/clean.sh && ./pants.bootstrap tests/python/twitter/common/log:test_log_with_scribe -k TestHandler -v
      
      
      
      Strangely, as my next command this works, and uses a different interpreter.
      
      
      
      [tw-mbp13-travis commons]$ PANTS_DEV=1 ./pants.bootstrap tests/python/twitter/common/log:test_log_with_scribe -k TestHandler -v
      *** running pants in dev mode from ./src/python/twitter/pants/bin/pants_exe.py ***
      Build operating on targets: OrderedSet([PythonTests(tests/python/twitter/common/log/BUILD:test_log_with_scribe)])
      =============> tcrawford debug Interpreter: PythonInterpreter('/Users/travis/src/commons/.python/bootstrap/bin/python', PythonIdentity('CPython', 2, 6, 8), {('psutil', '1.1.2'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('setuptools', '2.1.2'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('python', '2.6.8'): '/Users/travis/Python/CPython-2.6.8/lib/python2.6/lib-dynload', ('requests', '2.0.0'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('lockfile', '0.9.1'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('python-daemon', '1.5.5'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('wsgiref', '0.1.2'): '/Users/travis/Python/CPython-2.6.8/lib/python2.6', ('pystache', '0.5.3'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('pip', '1.4.1'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages'})
      ['Command', 'Distribution', 'Extension', 'Feature', 'Require', '_Command', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_dont_write_bytecode', '_get_unpatched', 'bootstrap_install_from', 'compat', 'convert_path', 'depends', 'dist', 'distutils', 'extension', 'find_packages', 'findall', 'lib2to3_fixer_packages', 'os', 'run_2to3_on_doctests', 'setup', 'setuptools', 'sys', 'version']
      ============================================================================================= test session starts ==============================================================================================
      platform darwin -- Python 2.6.8 -- py-1.4.20 -- pytest-2.5.2 -- /Users/travis/src/commons/.python/bootstrap/bin/python
      plugins: cov
      collected 19 items 
      
      tests/python/twitter/common/log/test_scribe_handler.py:136: TestHandler.test_buffer_client_fail PASSED
      tests/python/twitter/common/log/test_scribe_handler.py:151: TestHandler.test_buffer_client_fail_close_client_fail PASSED
      tests/python/twitter/common/log/test_scribe_handler.py:166: TestHandler.test_buffer_client_fail_close_connect_fail PASSED
      tests/python/twitter/common/log/test_scribe_handler.py:220: TestHandler.test_buffer_client_recover PASSED
      tests/python/twitter/common/log/test_scribe_handler.py:180: TestHandler.test_buffer_connect_fail PASSED
      tests/python/twitter/common/log/test_scribe_handler.py:194: TestHandler.test_buffer_connect_fail_close_client_fail PASSED
      tests/python/twitter/common/log/test_scribe_handler.py:208: TestHandler.test_buffer_connect_fail_close_connect_fail PASSED
      tests/python/twitter/common/log/test_scribe_handler.py:237: TestHandler.test_buffer_connect_recover PASSED
      tests/python/twitter/common/log/test_scribe_handler.py <- ../../../../private/var/folders/h1/193t7nxx5sz9pd6pfbjmlvr40000gp/T/tmp5m6ZeM/.deps/unittest2-0.5.1-py2.6.egg/unittest2/case.py:58: TestHandler.test_buffer_no_scribe SKIPPED
      tests/python/twitter/common/log/test_scribe_handler.py:125: TestHandler.test_buffer_ok PASSED
      tests/python/twitter/common/log/test_scribe_handler.py:103: TestHandler.test_drop_client_fail PASSED
      tests/python/twitter/common/log/test_scribe_handler.py:115: TestHandler.test_drop_connect_fail PASSED
      tests/python/twitter/common/log/test_scribe_handler.py <- ../../../../private/var/folders/h1/193t7nxx5sz9pd6pfbjmlvr40000gp/T/tmp5m6ZeM/.deps/unittest2-0.5.1-py2.6.egg/unittest2/case.py:58: TestHandler.test_drop_no_scribe SKIPPED
      tests/python/twitter/common/log/test_scribe_handler.py:91: TestHandler.test_drop_ok PASSED
      
      ==================================================================================== 5 tests deselected by '-kTestHandler' =====================================================================================
      ============================================================================== 12 passed, 2 skipped, 5 deselected in 0.18 seconds ==============================================================================
      tests.python.twitter.common.log.test_log_with_scribe                            .....   SUCCESS
      [tw-mbp13-travis commons]$ 
      
    5. Its coming from here:
      https://github.com/twitter/commons/blob/master/src/python/twitter/pants/commands/build.py#L70
      
      Adding this log line we see:
      print('============> %s %s' % (self.__class__, interpreters))
      
      OUTPUT:
      
      ============> <class 'twitter.pants.commands.build.Build'> [PythonInterpreter('/opt/twitter/bin/python', PythonIdentity('CPython', 2, 6, 5), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/wheel-0.22.0-py2.6.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/setuptools-2.2-py2.6.egg'})]
      
    6. does self.error not terminate the command?  if not, then there should be returns after all of them.
    7. Looking at my path, I expect this python:
      
      [tw-mbp13-travis commons]$ which python
      /Users/travis/Python/CPython-2.6.8/bin/python
    8. no, it's going to choose the minimum interpreter that meets the requirement, hence choosing 2.6.5 over 2.6.8
    9. Huh, depending on the invocation I get a different interpreter.
      
      *************************
      CLEAN BUILD CHOOSES 2.6.5
      *************************
      
      [tw-mbp13-travis commons]$ ./build-support/python/clean.sh && ./pants.bootstrap tests/python/twitter/common/log:test_log_with_scribe -k TestHandler -v
      
      ===========> <class 'twitter.pants.python.interpreter_cache.PythonInterpreterCache'> [PythonInterpreter('/Users/travis/src/commons/.python/bootstrap/bin/python', PythonIdentity('CPython', 2, 6, 8), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.8/wheel-0.22.0-py2.6.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.8/setuptools-2.2-py2.6.egg'}), PythonInterpreter('/usr/bin/python', PythonIdentity('CPython', 2, 7, 5), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.7.5/wheel-0.22.0-py2.7.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.7.5/setuptools-2.2-py2.7.egg'}), PythonInterpreter('/Users/travis/Python/CPython-2.7.3/bin/python', PythonIdentity('CPython', 2, 7, 3), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.7.3/wheel-0.22.0-py2.7.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.7.3/setuptools-2.2-py2.7.egg'}), PythonInterpreter('/opt/twitter/bin/python', PythonIdentity('CPython', 2, 6, 5), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/wheel-0.22.0-py2.6.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/setuptools-2.2-py2.6.egg'}), PythonInterpreter('/Users/travis/Python/CPython-3.2.3/bin/python3.2', PythonIdentity('CPython', 3, 2, 3), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-3.2.3/wheel-0.22.0-py3.2.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-3.2.3/setuptools-2.2-py3.2.egg'}), PythonInterpreter('/Users/travis/Python/PyPy-1.9/bin/pypy', PythonIdentity('PyPy', 2, 7, 2), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/PyPy-2.7.2/wheel-0.22.0-py2.7.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/PyPy-2.7.2/setuptools-2.2-py2.7.egg'})]
      
      ============> <class 'twitter.pants.python.test_builder.PythonTestBuilder'> PythonInterpreter('/opt/twitter/bin/python', PythonIdentity('CPython', 2, 6, 5), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/wheel-0.22.0-py2.6.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/setuptools-2.2-py2.6.egg'})
      
      
      **************************
      NEXT COMMAND CHOOSES 2.6.8
      **************************
      
      ./pants.bootstrap tests/python/twitter/common/log:test_log_with_scribe -k TestHandler -v
      
      ===========> <class 'twitter.pants.python.interpreter_cache.PythonInterpreterCache'> [PythonInterpreter('/usr/bin/python', PythonIdentity('CPython', 2, 7, 5), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.7.5/wheel-0.22.0-py2.7.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.7.5/setuptools-2.2-py2.7.egg'}), PythonInterpreter('/Users/travis/Python/CPython-2.7.3/bin/python', PythonIdentity('CPython', 2, 7, 3), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.7.3/wheel-0.22.0-py2.7.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.7.3/setuptools-2.2-py2.7.egg'}), PythonInterpreter('/opt/twitter/bin/python', PythonIdentity('CPython', 2, 6, 5), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/wheel-0.22.0-py2.6.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/setuptools-2.2-py2.6.egg'}), PythonInterpreter('/Users/travis/Python/CPython-3.2.3/bin/python3.2', PythonIdentity('CPython', 3, 2, 3), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-3.2.3/wheel-0.22.0-py3.2.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-3.2.3/setuptools-2.2-py3.2.egg'}), PythonInterpreter('/Users/travis/src/commons/.python/bootstrap/bin/python', PythonIdentity('CPython', 2, 6, 8), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.8/wheel-0.22.0-py2.6.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.8/setuptools-2.2-py2.6.egg'}), PythonInterpreter('/Users/travis/Python/PyPy-1.9/bin/pypy', PythonIdentity('PyPy', 2, 7, 2), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/PyPy-2.7.2/wheel-0.22.0-py2.7.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/PyPy-2.7.2/setuptools-2.2-py2.7.egg'})]
      
      ============> <class 'twitter.pants.python.test_builder.PythonTestBuilder'> PythonInterpreter('/Users/travis/src/commons/.python/bootstrap/bin/python', PythonIdentity('CPython', 2, 6, 8), {('psutil', '1.1.2'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('setuptools', '2.1.2'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('python', '2.6.8'): '/Users/travis/Python/CPython-2.6.8/lib/python2.6/lib-dynload', ('requests', '2.0.0'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('lockfile', '0.9.1'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('python-daemon', '1.5.5'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('wsgiref', '0.1.2'): '/Users/travis/Python/CPython-2.6.8/lib/python2.6', ('pystache', '0.5.3'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('pip', '1.4.1'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages'})
      
    10. What seems to be happening is the interpreter cache has /Users/travis/src/commons/.python/bootstrap/bin/python, but the cached version has different dependencies than the interpreter currently being used. We fall back to the min acceptable interpreter.
      
      Running ./pants.bootstrap tests/python/twitter/common/log:test_log_with_scribe -k TestHandler -v again - after a clean build - causes the existing interpreter cache to have been updated and its used.
      
      
      ===========> <class 'twitter.pants.python.interpreter_cache.PythonInterpreterCache'> [PythonInterpreter('/Users/travis/src/commons/.python/bootstrap/bin/python', PythonIdentity('CPython', 2, 6, 8), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.8/wheel-0.22.0-py2.6.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.8/setuptools-2.2-py2.6.egg'}), PythonInterpreter('/Users/travis/Python/CPython-2.7.3/bin/python', PythonIdentity('CPython', 2, 7, 3), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.7.3/wheel-0.22.0-py2.7.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.7.3/setuptools-2.2-py2.7.egg'}), PythonInterpreter('/Users/travis/Python/PyPy-1.9/bin/pypy', PythonIdentity('PyPy', 2, 7, 2), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/PyPy-2.7.2/wheel-0.22.0-py2.7.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/PyPy-2.7.2/setuptools-2.2-py2.7.egg'}), PythonInterpreter('/opt/twitter/bin/python', PythonIdentity('CPython', 2, 6, 5), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/wheel-0.22.0-py2.6.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/setuptools-2.2-py2.6.egg'}), PythonInterpreter('/usr/bin/python', PythonIdentity('CPython', 2, 7, 5), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.7.5/wheel-0.22.0-py2.7.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.7.5/setuptools-2.2-py2.7.egg'}), PythonInterpreter('/Users/travis/Python/CPython-3.2.3/bin/python3.2', PythonIdentity('CPython', 3, 2, 3), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-3.2.3/wheel-0.22.0-py3.2.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-3.2.3/setuptools-2.2-py3.2.egg'})]
      
      ===========> me is <class 'twitter.pants.python.interpreter_cache.PythonInterpreterCache'> PythonInterpreter('/Users/travis/src/commons/.python/bootstrap/bin/python', PythonIdentity('CPython', 2, 6, 8), {('psutil', '1.1.2'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('setuptools', '2.1.2'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('python', '2.6.8'): '/Users/travis/Python/CPython-2.6.8/lib/python2.6/lib-dynload', ('requests', '2.0.0'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('lockfile', '0.9.1'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('python-daemon', '1.5.5'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('wsgiref', '0.1.2'): '/Users/travis/Python/CPython-2.6.8/lib/python2.6', ('pystache', '0.5.3'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('pip', '1.4.1'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages'})
      
      ===========> using min interpreter <class 'twitter.pants.python.interpreter_cache.PythonInterpreterCache'> [PythonInterpreter('/opt/twitter/bin/python', PythonIdentity('CPython', 2, 6, 5), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/wheel-0.22.0-py2.6.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.5/setuptools-2.2-py2.6.egg'})]
      
    11. Looking at new interpreter creation, here's the first time its created for /Users/travis/src/commons/.python/bootstrap/bin/python
      
      
      ------------> new python interpreter <class 'twitter.common.python.interpreter.PythonInterpreter'> PythonInterpreter('/Users/travis/src/commons/.python/bootstrap/bin/python', PythonIdentity('CPython', 2, 6, 8), {('psutil', '1.1.2'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('setuptools', '2.1.2'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('python', '2.6.8'): '/Users/travis/Python/CPython-2.6.8/lib/python2.6/lib-dynload', ('requests', '2.0.0'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('lockfile', '0.9.1'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('python-daemon', '1.5.5'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('wsgiref', '0.1.2'): '/Users/travis/Python/CPython-2.6.8/lib/python2.6', ('pystache', '0.5.3'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('pip', '1.4.1'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages'})
        File "./src/python/twitter/pants/bin/pants_exe.py", line 30, in <module>
          from twitter.pants.commands import Command
        File "/Users/travis/src/commons/src/python/twitter/pants/commands/__init__.py", line 120, in <module>
          Command._register_modules()
        File "/Users/travis/src/commons/src/python/twitter/pants/commands/__init__.py", line 47, in _register_modules
          __import__(fq_module)
        File "/Users/travis/src/commons/src/python/twitter/pants/commands/build.py", line 28, in <module>
          from twitter.pants.python.interpreter_cache import PythonInterpreterCache
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 31, in <module>
          from twitter.common.python.obtainer import Obtainer
        File "/Users/travis/src/commons/src/python/twitter/common/python/obtainer.py", line 5, in <module>
          from .translator import ChainedTranslator
        File "/Users/travis/src/commons/src/python/twitter/common/python/translator.py", line 54, in <module>
          class SourceTranslator(TranslatorBase):
        File "/Users/travis/src/commons/src/python/twitter/common/python/translator.py", line 73, in SourceTranslator
          interpreter=PythonInterpreter.get(),
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 206, in get
          return cls.from_binary(sys.executable)
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 282, in from_binary
          cls.CACHE[binary] = cls._from_binary_internal(path_extras)
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 232, in _from_binary_internal
          return cls(sys.executable, PythonIdentity.get(), dict(iter_extras()))
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 375, in __init__
          traceback.print_stack()
      
      
      
      
      
      
      Then some stuff happens in the interpreter cache that makes a number of instances – for the same binary – that all have different dependencies.
      
      This doesn't seem like the right behavior. I need to learn more about this, but it seems like we should do something like:
      
      * Error out if trying to make a PythonInterpreter for a binary we already have.
      * Ignore duplicates of a PythonInterpreter we already found.
      
      At read time, it seems like the latest entry for a binary wins, since the last one defined here is what's being used. Need to look a bit more, but in this doesn't seem same the more I look at it.
      
      
      
      ------------> new python interpreter <class 'twitter.common.python.interpreter.PythonInterpreter'> PythonInterpreter('/Users/travis/src/commons/.python/bootstrap/bin/python', PythonIdentity('CPython', 2, 6, 8), {('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.8/setuptools-2.2-py2.6.egg', ('lockfile', '0.9.1'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('setuptools', '2.1.2'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('python', '2.6.8'): '/Users/travis/Python/CPython-2.6.8/lib/python2.6/lib-dynload', ('pip', '1.4.1'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('wsgiref', '0.1.2'): '/Users/travis/Python/CPython-2.6.8/lib/python2.6', ('python-daemon', '1.5.5'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('psutil', '1.1.2'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('pystache', '0.5.3'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('requests', '2.0.0'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages'})
        File "./src/python/twitter/pants/bin/pants_exe.py", line 198, in <module>
          main()
        File "./src/python/twitter/pants/bin/pants_exe.py", line 192, in main
          _run()
        File "./src/python/twitter/pants/bin/pants_exe.py", line 164, in _run
          command = command_class(run_tracker, root_dir, parser, command_args)
        File "/Users/travis/src/commons/src/python/twitter/pants/commands/build.py", line 62, in __init__
          self.interpreter_cache.setup()
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 186, in setup
          self.setup_paths(setup_paths)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 165, in setup_paths
          self.setup_interpreter(interpreter)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 149, in setup_interpreter
          return resolve(self._config, interpreter, logger=self._logger)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 114, in resolve
          interpreter = resolve_interpreter(config, interpreter, setuptools_requirement, logger=logger)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 65, in resolve_interpreter
          return interpreter.with_extra(egg.name, egg.raw_version, egg.url)
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 380, in with_extra
          return self.__class__(self._binary, self._identity, extras)
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 375, in __init__
          traceback.print_stack()
      ------------> new python interpreter <class 'twitter.common.python.interpreter.PythonInterpreter'> PythonInterpreter('/Users/travis/src/commons/.python/bootstrap/bin/python', PythonIdentity('CPython', 2, 6, 8), {('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.8/setuptools-2.2-py2.6.egg', ('lockfile', '0.9.1'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('setuptools', '2.1.2'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.8/wheel-0.22.0-py2.6.egg', ('python', '2.6.8'): '/Users/travis/Python/CPython-2.6.8/lib/python2.6/lib-dynload', ('requests', '2.0.0'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('python-daemon', '1.5.5'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('wsgiref', '0.1.2'): '/Users/travis/Python/CPython-2.6.8/lib/python2.6', ('pystache', '0.5.3'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('pip', '1.4.1'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages', ('psutil', '1.1.2'): '/Users/travis/src/commons/.python/bootstrap/lib/python2.6/site-packages'})
        File "./src/python/twitter/pants/bin/pants_exe.py", line 198, in <module>
          main()
        File "./src/python/twitter/pants/bin/pants_exe.py", line 192, in main
          _run()
        File "./src/python/twitter/pants/bin/pants_exe.py", line 164, in _run
          command = command_class(run_tracker, root_dir, parser, command_args)
        File "/Users/travis/src/commons/src/python/twitter/pants/commands/build.py", line 62, in __init__
          self.interpreter_cache.setup()
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 186, in setup
          self.setup_paths(setup_paths)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 165, in setup_paths
          self.setup_interpreter(interpreter)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 149, in setup_interpreter
          return resolve(self._config, interpreter, logger=self._logger)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 116, in resolve
          return resolve_interpreter(config, interpreter, wheel_requirement, logger=logger)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 65, in resolve_interpreter
          return interpreter.with_extra(egg.name, egg.raw_version, egg.url)
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 380, in with_extra
          return self.__class__(self._binary, self._identity, extras)
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 375, in __init__
          traceback.print_stack()
      ------------> new python interpreter <class 'twitter.common.python.interpreter.PythonInterpreter'> PythonInterpreter('/Users/travis/src/commons/.python/bootstrap/bin/python', PythonIdentity('CPython', 2, 6, 8), {})
        File "./src/python/twitter/pants/bin/pants_exe.py", line 198, in <module>
          main()
        File "./src/python/twitter/pants/bin/pants_exe.py", line 192, in main
          _run()
        File "./src/python/twitter/pants/bin/pants_exe.py", line 164, in _run
          command = command_class(run_tracker, root_dir, parser, command_args)
        File "/Users/travis/src/commons/src/python/twitter/pants/commands/build.py", line 62, in __init__
          self.interpreter_cache.setup()
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 186, in setup
          self.setup_paths(setup_paths)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 166, in setup_paths
          pi = self.interpreter_from_path(path)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 142, in interpreter_from_path
          interpreter = PythonInterpreter(executable, identity)
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 375, in __init__
          traceback.print_stack()
      ------------> new python interpreter <class 'twitter.common.python.interpreter.PythonInterpreter'> PythonInterpreter('/Users/travis/src/commons/.python/bootstrap/bin/python', PythonIdentity('CPython', 2, 6, 8), {('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.8/setuptools-2.2-py2.6.egg'})
        File "./src/python/twitter/pants/bin/pants_exe.py", line 198, in <module>
          main()
        File "./src/python/twitter/pants/bin/pants_exe.py", line 192, in main
          _run()
        File "./src/python/twitter/pants/bin/pants_exe.py", line 164, in _run
          command = command_class(run_tracker, root_dir, parser, command_args)
        File "/Users/travis/src/commons/src/python/twitter/pants/commands/build.py", line 62, in __init__
          self.interpreter_cache.setup()
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 186, in setup
          self.setup_paths(setup_paths)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 166, in setup_paths
          pi = self.interpreter_from_path(path)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 143, in interpreter_from_path
          return resolve(self._config, interpreter, logger=self._logger)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 114, in resolve
          interpreter = resolve_interpreter(config, interpreter, setuptools_requirement, logger=logger)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 65, in resolve_interpreter
          return interpreter.with_extra(egg.name, egg.raw_version, egg.url)
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 380, in with_extra
          return self.__class__(self._binary, self._identity, extras)
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 375, in __init__
          traceback.print_stack()
      ------------> new python interpreter <class 'twitter.common.python.interpreter.PythonInterpreter'> PythonInterpreter('/Users/travis/src/commons/.python/bootstrap/bin/python', PythonIdentity('CPython', 2, 6, 8), {('wheel', '0.22.0'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.8/wheel-0.22.0-py2.6.egg', ('setuptools', '2.2'): '/Users/travis/src/commons/.pants.d/python/interpreters/CPython-2.6.8/setuptools-2.2-py2.6.egg'})
        File "./src/python/twitter/pants/bin/pants_exe.py", line 198, in <module>
          main()
        File "./src/python/twitter/pants/bin/pants_exe.py", line 192, in main
          _run()
        File "./src/python/twitter/pants/bin/pants_exe.py", line 164, in _run
          command = command_class(run_tracker, root_dir, parser, command_args)
        File "/Users/travis/src/commons/src/python/twitter/pants/commands/build.py", line 62, in __init__
          self.interpreter_cache.setup()
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 186, in setup
          self.setup_paths(setup_paths)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 166, in setup_paths
          pi = self.interpreter_from_path(path)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 143, in interpreter_from_path
          return resolve(self._config, interpreter, logger=self._logger)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 116, in resolve
          return resolve_interpreter(config, interpreter, wheel_requirement, logger=logger)
        File "/Users/travis/src/commons/src/python/twitter/pants/python/interpreter_cache.py", line 65, in resolve_interpreter
          return interpreter.with_extra(egg.name, egg.raw_version, egg.url)
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 380, in with_extra
          return self.__class__(self._binary, self._identity, extras)
        File "/Users/travis/src/commons/src/python/twitter/common/python/interpreter.py", line 375, in __init__
          traceback.print_stack()
      
    12. I'm not able to reproduce this on my personal machine. While this should work, I'm going to clean up the pythons on my work machine path and see if that coaxes pants back into building. I don't believe anyone else is seeing this at present, so if cleaning up my path works I'm going to table this.
  2. 
      
WI
TR
  1. Ship It!
  2. 
      
WI
Review request changed

Status: Closed (submitted)

Loading...