initial draft of publishing a pants test infra package to facilitate external pants plugin testing

Review Request #997 — Created Sept. 5, 2014 and submitted

jinfeng
pants
github_539
539
f6da6e6...
pants-reviews
areitz, ity, jsirois, zundel
  • initial draft of publishing a pants test infra package to facilitate external pants plugin testing.

this is a draft, and trying to figure out how to test this publishing, and seeing failure below. Have read http://pantsbuild.github.io/publish.html but it's mostly about jvm/jar publishing, not so much about python egg stuff.

$ PANTS_DEV=1 ./pants goal publish tests/python/pants_test:test_infra --publish-dryrun --no-publish-commit --publish-force
Running pants in dev mode from /Users/jinfeng/workspace/github-pants/src/python/pants/bin/pants_exe.py
/usr/local/lib/python2.7/site-packages/setuptools-3.4.1-py2.7.egg/pkg_resources.py:1031: UserWarning: /Users/jinfeng/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).

10:13:52 00:00 [main]
See a report at: http://localhost:52017/run/pants_run_2014_09_05_10_13_52_141
10:13:52 00:00 [bootstrap]
10:13:52 00:00 [setup]
10:13:52 00:00 [parse]
FAILURE: This repo is not configured to publish externally! Please configure per
http://pantsbuild.github.io/publish.html#authenticating-to-the-artifact-repository,
or re-run with the '--publish-local' flag.

           FAILURE
JS
  1. Python publishing is not directly addressed with a doc topic. However, it is indirectly addressed by the pants release docs: http://pantsbuild.github.io/release.html

    In particular the script that points to has all the details: https://github.com/pantsbuild/pants/blob/master/build-support/bin/release.sh
    Most critically, the actual publish function:

    function publish() {
      run_local_pants setup_py --run="sdist upload" --recursive //src/python/pants:pants-packaged
    }
    

    You can test publishing works save for the upload the same way the script does, see dry_run_install: https://github.com/pantsbuild/pants/blob/master/build-support/bin/release.sh#L50
    In short it:
    runs ./pants setup_py --recursive [target(s)] - this plops sdists in dist/
    creates a fresh temporary virtualenv
    attempts to install the package - currently just pantsbuild.pants - in the venv
    do some rudimentary check the venv has the goods - just something like this would probably be a good start:
    bash (venv) $ python -c "import pants_test" || echo "import failed!"

    1. I've run the steps outlined in the release.sh, failed at the pip install part:

      $ pip install --find-links=file://Users/jinfeng/workspace/github-pants/dist pantsbuild.pants.testinfra==0.0.23
      Downloading/unpacking pantsbuild.pants.testinfra==0.0.23
        Running setup.py (path:/Users/jinfeng/workspace/github-pants/dist/pantsbuild.pants.testinfra/build/pantsbuild.pants.testinfra/setup.py) egg_info for package pantsbuild.pants.testinfra
      
      Downloading/unpacking Markdown==2.1.1 (from pantsbuild.pants.testinfra==0.0.23)
        Downloading Markdown-2.1.1.tar.gz (242kB): 242kB downloaded
        Running setup.py (path:/Users/jinfeng/workspace/github-pants/dist/pantsbuild.pants.testinfra/build/Markdown/setup.py) egg_info for package Markdown
      
      Downloading/unpacking Pygments==1.4 (from pantsbuild.pants.testinfra==0.0.23)
        Downloading Pygments-1.4.tar.gz (3.5MB): 3.5MB downloaded
        Running setup.py (path:/Users/jinfeng/workspace/github-pants/dist/pantsbuild.pants.testinfra/build/Pygments/setup.py) egg_info for package Pygments
      
      Downloading/unpacking ansicolors==1.0.2 (from pantsbuild.pants.testinfra==0.0.23)
        Downloading ansicolors-1.0.2.tar.gz
        Running setup.py (path:/Users/jinfeng/workspace/github-pants/dist/pantsbuild.pants.testinfra/build/ansicolors/setup.py) egg_info for package ansicolors
      
      Downloading/unpacking antlr-python-runtime==3.1.3 (from pantsbuild.pants.testinfra==0.0.23)
        Could not find any downloads that satisfy the requirement antlr-python-runtime==3.1.3 (from pantsbuild.pants.testinfra==0.0.23)
        Some externally hosted files were ignored (use --allow-external antlr-python-runtime to allow).
      Cleaning up...
      No distributions at all found for antlr-python-runtime==3.1.3 (from pantsbuild.pants.testinfra==0.0.23)
      Storing debug log for failure in /Users/jinfeng/.pip/pip.log
      

      Where is the dependency on antlr-python-runtime 3.1.3 specified? I assume pip is trying to download that from pypi and strangely antlr-python-runtime 3.1.3 does exist on pypi:

      https://pypi.python.org/pypi/antlr_python_runtime/3.1.3

    2. Does reading ~/.pip/pip.log help?

      I suspect its this issue (1st try, partly fixed 2nd try, fully fixed 3rd try):
      1.) $ venv_setup antlr antlr-python-runtime==3.1.3
      Creating virtualenv antlr @ /home/jsirois/.pythons/venvs/antlr
      New python executable in /home/jsirois/.pythons/venvs/antlr/bin/python
      Installing setuptools, pip...done.
      Downloading/unpacking antlr-python-runtime==3.1.3
      Could not find any downloads that satisfy the requirement antlr-python-runtime==3.1.3
      Some externally hosted files were ignored (use --allow-external antlr-python-runtime to allow).
      Cleaning up...
      No distributions at all found for antlr-python-runtime==3.1.3
      Storing debug log for failure in /home/jsirois/.pip/pip.log
      Failed to Downloading/unpacking antlr-python-runtime==3.1.3
      Could not find any downloads that satisfy the requirement antlr-python-runtime==3.1.3
      Some externally hosted files were ignored (use --allow-external antlr-python-runtime to allow).
      Cleaning up...
      No distributions at all found for antlr-python-runtime==3.1.3
      Storing debug log for failure in /home/jsirois/.pip/pip.log
      Failed to setup /home/jsirois/.pythons/venvs/antlr

      2.) jsirois@gill ~ $ venv_setup antlr antlr-python-runtime==3.1.3 --allow-external antlr-python-runtime
      Recreating virtualenv antlr @ /home/jsirois/.pythons/venvs/antlr
      New python executable in /home/jsirois/.pythons/venvs/antlr/bin/python
      Installing setuptools, pip...done.
      Downloading/unpacking antlr-python-runtime==3.1.3
      Could not find any downloads that satisfy the requirement antlr-python-runtime==3.1.3
      Some insecure and unverifiable files were ignored (use --allow-unverified antlr-python-runtime to allow).
      Cleaning up...
      No distributions at all found for antlr-python-runtime==3.1.3
      Storing debug log for failure in /home/jsirois/.pip/pip.log
      Failed to Downloading/unpacking antlr-python-runtime==3.1.3
      Could not find any downloads that satisfy the requirement antlr-python-runtime==3.1.3
      Some insecure and unverifiable files were ignored (use --allow-unverified antlr-python-runtime to allow).
      Cleaning up...
      No distributions at all found for antlr-python-runtime==3.1.3
      Storing debug log for failure in /home/jsirois/.pip/pip.log
      Failed to setup /home/jsirois/.pythons/venvs/antlr

      3.) jsirois@gill ~ $ venv_setup antlr antlr-python-runtime==3.1.3 --allow-external antlr-python-runtime --allow-unverified antlr-python-runtime
      Recreating virtualenv antlr @ /home/jsirois/.pythons/venvs/antlr
      New python executable in /home/jsirois/.pythons/venvs/antlr/bin/python
      Installing setuptools, pip...done.
      Downloading/unpacking antlr-python-runtime==3.1.3
      antlr-python-runtime an externally hosted file and may be unreliable
      antlr-python-runtime is potentially insecure and unverifiable.
      Downloading antlr_python_runtime-3.1.3.tar.gz (76kB): 76kB downloaded
      Running setup.py (path:/home/jsirois/.pythons/venvs/antlr/build/antlr-python-runtime/setup.py) egg_info for package antlr-python-runtime

      Installing collected packages: antlr-python-runtime
      Running setup.py install for antlr-python-runtime

      Successfully installed antlr-python-runtime
      Cleaning up...

      This issue and its fix is also hinted at here:
      jsirois@gill ~/dev/3rdparty/jsirois-pants2 (master) $ head 3rdparty/python/requirements.txt
      ansicolors==1.0.2

      antlr_python_runtime==3.1.3
      --allow-external antlr-python-runtime
      --allow-unverified antlr-python-runtime

      argparse==1.2.1
      Markdown==2.1.1
      mock==1.0.1
      mox==0.5.3

    3. Nope, curious what part pip.log should I look for?

      I only saw some lines mentioning antlr:

      Downloading/unpacking antlr-python-runtime==3.1.3 (from pantsbuild.pants.testinfra==0.0.23)
        Getting page https://pypi.python.org/simple/antlr-python-runtime/
        URLs to search for versions for antlr-python-runtime==3.1.3 (from pantsbuild.pants.testinfra==0.0.23):
        * https://pypi.python.org/simple/antlr-python-runtime/3.1.3
        * https://pypi.python.org/simple/antlr-python-runtime/
        Getting page https://pypi.python.org/simple/antlr-python-runtime/3.1.3
        Could not fetch URL https://pypi.python.org/simple/antlr-python-runtime/3.1.3: 404 Client Error: Not Found
        Will skip URL https://pypi.python.org/simple/antlr-python-runtime/3.1.3 when looking for download links for antlr-python-runtime==3.1.3 (from pantsbuild.pants.testinfra==0.0.23)
        Analyzing links from page https://pypi.python.org/simple/antlr-python-runtime/
          Skipping http://antlr3.org/download/Python/antlr_python_runtime-3.1.3.tar.gz (from https://pypi.python.org/simple/antlr-python-runtime/) because it is externally hosted.
          Skipping link http://www.antlr3.org/ (from https://pypi.python.org/simple/antlr-python-runtime/); not a file
        Could not parse version from link: file:///Users/jinfeng/workspace/github-pants/dist/examplea.zip
        Skipping link file:///Users/jinfeng/workspace/github-pants/dist/examplea.zip; wrong project name (not antlr-python-runtime)
        Could not parse version from link: file:///Users/jinfeng/workspace/github-pants/dist/exampleb.zip
        Skipping link file:///Users/jinfeng/workspace/github-pants/dist/exampleb.zip; wrong project name (not antlr-python-runtime)
        Could not parse version from link: file:///Users/jinfeng/workspace/github-pants/dist/examplec.zip
        Skipping link file:///Users/jinfeng/workspace/github-pants/dist/examplec.zip; wrong project name (not antlr-python-runtime)
        Skipping link file:///Users/jinfeng/workspace/github-pants/dist/pantsbuild.pants.testinfra-0.0.23.tar.gz; wrong project name (not antlr-python-runtime)
        Skipping link file:///Users/jinfeng/workspace/github-pants/dist/unicode-testproject.zip; wrong project name (not antlr-python-runtime)
        Could not find any downloads that satisfy the requirement antlr-python-runtime==3.1.3 (from pantsbuild.pants.testinfra==0.0.23)
        Some externally hosted files were ignored (use --allow-external antlr-python-runtime to allow).
      Cleaning up...
      

      Somehow, the link has "simple" in it: https://pypi.python.org/simple/antlr-python-runtime/3.1.3, but the actual page is "https://pypi.python.org/pypi/antlr_python_runtime/3.1.3" with "pypi"

      Yup, now I've changed my command line and it worked (with the added '-r' option):

      $ pip install --find-links=file://Users/jinfeng/workspace/github-pants/dist pantsbuild.pants.testinfra==0.0.23 -r /Users/jinfeng/workspace/github-pants/3rdparty/python/requirements.txt
      

      So my question is, if my patch goes in like this, some external repo (like twitter/common or pants_internal) they need to have this requirements.txt with --allow-external antlr-python-runtime, --allow-unverified antlr-python-runtime listed as well and specified in the -r option in their install script?

      Another question, you said we should consider putting these two pantsbuild.pants and pantsbuild.pants.testinfra together (two separate pypi packages) but the same release. So I should modify the https://github.com/pantsbuild/pants/blob/master/build-support/bin/release.sh, right? (in a separate RB of course).

    4. The key lines in your pip log:

        Could not find any downloads that satisfy the requirement antlr-python-runtime==3.1.3 (from pantsbuild.pants.testinfra==0.0.23)
        Some externally hosted files were ignored (use --allow-external antlr-python-runtime to allow).
      

      They suggest adding the command line flag: --allow-external antlr-python-runtime. My example 1/2/3 output above was meant to show this same process. You'll need a second flag after you fix this as well --allow-unverified antlr-python-runtime

      So to be really explicit this time, you need exactly:
      pip install --find-links=file://Users/jinfeng/workspace/github-pants/dist pantsbuild.pants.testinfra==0.0.23 --allow-external antlr-python-runtime --allow-unverified antlr-python-runtime

      My pointer to 3rdparty/python/requirements.txt above was just to show this is all safe and already done for pants internally via its requirements.txt file.

      Finally - correct on release.sh in a seperate RB.

  2. tests/python/pants_test/BUILD (Diff revision 1)
     
     

    Eventually it would be good to get a CHANGELOG going here but a TODO will suffice. It could be the same changelog already in-use for pantsbuild.pants assuming we'll want 1 release operation (still), with the only difference being 2 sdists produced.

    1. will "TODO" now.

  3. 
      
JI
JS
  1. Ship It!

  2. 
      
JI
  1. thanks. CI has been kicked off at: https://travis-ci.org/jinfeng/jinfeng-pants-fork/builds/34542742

    1. passed. can committers help me to patch it to master? thanks!

  2. 
      
JS
  1. In master @ https://github.com/pantsbuild/pants/commit/a9068406693f6621cb87d005deb76e30c03cb58b
    Please mark this review as submitted.

    Going forward - please edit the summary and description such that they form the commit message you want to go in (summary is 1st line, description is the body separated by a single empty line).
    I just edited this one down to the summary since the relevant to the commit.

    1. You can run rbt patch -c XXX in a new branch to preview what happens.

  2. 
      
JI
Review request changed

Status: Closed (submitted)

Loading...