Added 'http_debug' config option to turn on debugging for HTTP headers in ivy

Review Request #1874 — Created March 6, 2015 and discarded

zundel
pants
zundel/extra-ivy-logging
1213
pants-reviews
benjyw, jsirois

Added 'http_debug' config option to turn on debugging for HTTP headers in ivy.

See online discussion https://groups.google.com/forum/#!topic/pants-devel/xfrYj0_UB-c

 src/python/pants/ivy/bootstrapper.py | 35 ++++++++++++++++++++++++++++++++---
 src/python/pants/ivy/ivy.py          | 15 ++++++++++++---
 2 files changed, 44 insertions(+), 6 deletions(-

Add this in pants.ini:

[ivy]
http_debug: True

When you run again ivy will re-bootstrap itself with more libraries including commons-httpclient.

Then when ivy runs to resolve libraries, you'll see extra messages in the .stderr files in the tool debug
output dir .pants.d/runs/latest/tool_outputs:

00c393ba-15f7-4168-a655-41ab7f573d5e.stderr:2015/03/06 11:07:27:316 EST [DEBUG] header - >> "GET /maven2/antlr/antlr/2.7.7/antlr-2.7.7.jar.sha1 HTTP/1.1[\r][\n]"

I tested with:

PANTS_DEV=1 ./pants compile examples/src/java/com/pants/examples:: -ldebug

If your remove the setting from pants.ini, pants will go back to using ivy without the extra libraries and debug config.

ZU
  1. 
      
  2. src/python/pants/ivy/bootstrapper.py (Diff revision 1)
     
     

    Ideally, this would be an option. This is waiting on the "subsystems" option configuration as proposed by Benjy:

    https://docs.google.com/document/d/1hinJLcTDyx1Rz8FvWU-I1m0WQzt3ynxSdbRHLmhBAek/edit?usp=sharing

  3. 
      
JS
  1. 
      
  2. src/python/pants/ivy/bootstrapper.py (Diff revision 1)
     
     
    IvyResolve already exports the --jvm-options option and here is direct support for _version_or_ivyxml where a custom ivy.xml can be used to include the commons http-client (and not for example all the rest of the optional un-needed deps).  With those 2 things you can setup debugging, proxies etc in the main pants.ini or in a --config-override ini file or in ~/.pants.rc.  Is that good enough for now to prove out that the technique is useful and/or the commons http client is more stable?
    1. Indeed, setting jvm options on [ bootstrapper.bootstrap_jvm_tool ] or something like that is how I proved it out, along with turning off nailgun.

      My impetus behing baking this into bootstrapper is that I want to be able to tell a user to flip a switch and send me the logs the next time this happens to them.

    2. Right - and that _could_ be:
      ```
      [DEFAULT]
      config_override: pants.debug_ivy.ini
      ```
      
      Where pants.debug_ivy.ini is a file with the config settings that's checked into the repo.
    3. This general mechanism is how Twitter beta'd the build cache for trusted tester users for example.
    4. I didn't answer this before, but the only reason for adding the common-httpclient library above is to enable debugging of http headers.

      What I would like to propose is taking this PR and:

      1) Taking out the settings to append specific JVM options. I think we can just document how to do that and leave it.
      2) Using the modifications to bootstrapper and ivy to support the HTTP_PROXY detection in https://rbcommons.com/s/twitter/r/1877/

      I've documented the bits I had to set in order to get Pants setup to use proxies for its traffic. https://rbcommons.com/s/twitter/r/1899/ You have to set HTTP_PROXY or HTTPS_PROXY to use pants in this environment anyway, it is a chore to go through and specify the same options in different ways to Ivy.

      I think it would be a better story to say:

      To use pants behind a firewall, set HTTP_PFOXY or HTTPS_PROXY.

    5. This all makes sense to me.
  3. 
      
ZU
Review request changed

Status: Discarded

Change Summary:

Dropping this in favor of just documenting the steps to enable this debugging, which are now in the troubleshooting page on pantsbuild.github.io. Pieces of this are up for review in https://rbcommons.com/s/twitter/r/1877

Loading...