Added 'http_debug' config option to turn on debugging for HTTP headers in ivy

Review Request #1874 — Created March 6, 2015 and discarded

benjyw, jsirois

Added 'http_debug' config option to turn on debugging for HTTP headers in ivy.

See online discussion!topic/pants-devel/xfrYj0_UB-c

 src/python/pants/ivy/ | 35 ++++++++++++++++++++++++++++++++---
 src/python/pants/ivy/          | 15 ++++++++++++---
 2 files changed, 44 insertions(+), 6 deletions(-

Add this in pants.ini:

http_debug: True

When you run again ivy will re-bootstrap itself with more libraries including commons-httpclient.

Then when ivy runs to resolve libraries, you'll see extra messages in the .stderr files in the tool debug
output dir .pants.d/runs/latest/tool_outputs:

00c393ba-15f7-4168-a655-41ab7f573d5e.stderr:2015/03/06 11:07:27:316 EST [DEBUG] header - >> "GET /maven2/antlr/antlr/2.7.7/antlr-2.7.7.jar.sha1 HTTP/1.1[\r][\n]"

I tested with:

PANTS_DEV=1 ./pants compile examples/src/java/com/pants/examples:: -ldebug

If your remove the setting from pants.ini, pants will go back to using ivy without the extra libraries and debug config.

  2. src/python/pants/ivy/ (Diff revision 1)

    Ideally, this would be an option. This is waiting on the "subsystems" option configuration as proposed by Benjy:

  2. src/python/pants/ivy/ (Diff revision 1)
    IvyResolve already exports the --jvm-options option and here is direct support for _version_or_ivyxml where a custom ivy.xml can be used to include the commons http-client (and not for example all the rest of the optional un-needed deps).  With those 2 things you can setup debugging, proxies etc in the main pants.ini or in a --config-override ini file or in ~/.pants.rc.  Is that good enough for now to prove out that the technique is useful and/or the commons http client is more stable?
    1. Indeed, setting jvm options on [ bootstrapper.bootstrap_jvm_tool ] or something like that is how I proved it out, along with turning off nailgun.

      My impetus behing baking this into bootstrapper is that I want to be able to tell a user to flip a switch and send me the logs the next time this happens to them.

    2. Right - and that _could_ be:
      config_override: pants.debug_ivy.ini
      Where pants.debug_ivy.ini is a file with the config settings that's checked into the repo.
    3. This general mechanism is how Twitter beta'd the build cache for trusted tester users for example.
    4. I didn't answer this before, but the only reason for adding the common-httpclient library above is to enable debugging of http headers.

      What I would like to propose is taking this PR and:

      1) Taking out the settings to append specific JVM options. I think we can just document how to do that and leave it.
      2) Using the modifications to bootstrapper and ivy to support the HTTP_PROXY detection in

      I've documented the bits I had to set in order to get Pants setup to use proxies for its traffic. You have to set HTTP_PROXY or HTTPS_PROXY to use pants in this environment anyway, it is a chore to go through and specify the same options in different ways to Ivy.

      I think it would be a better story to say:

      To use pants behind a firewall, set HTTP_PFOXY or HTTPS_PROXY.

    5. This all makes sense to me.
Review request changed

Status: Discarded

Change Summary:

Dropping this in favor of just documenting the steps to enable this debugging, which are now in the troubleshooting page on Pieces of this are up for review in