Add flag to disable automatic excludes

Review Request #2252 — Created May 21, 2015 and submitted

areitz
pants
3c99030...
pants-reviews
ity, stuhood, zundel

The automatic excludes that Stu added are nice when they do what you want. But occasionally this isn't what you want, so this RB makes this behavior optional (and adds some logging so that you can know when it's happening).

Travis running here: https://travis-ci.org/pantsbuild/pants/builds/63427894

Local test succeeded:

[prometheus pants (areitz/add_auto_exclude_option)]$ PANTS_DEV=1 ./pants test tests/python/pants_test/backend/jvm/tasks:ivy_utils
*** Running pants in dev mode from /Users/areitz/workspace/pants/src/python/pants/bin/pants_exe.py ***
INFO] Detected git repository at /Users/areitz/workspace/pants on branch areitz/add_auto_exclude_option

19:48:22 00:00 [main]
               (To run a reporting server: ./pants server)
19:48:22 00:00   [bootstrap]
19:48:22 00:00   [setup]
19:48:22 00:00     [parse]
               Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> resolve -> compile -> resources -> test
19:48:22 00:00   [bootstrap]
19:48:22 00:00     [bootstrap-jvm-tools]
19:48:22 00:00   [imports]
19:48:22 00:00     [ivy-imports]
19:48:22 00:00   [unpack-jars]
19:48:22 00:00     [unpack-jars]
19:48:22 00:00   [deferred-sources]
19:48:22 00:00     [deferred-sources]
19:48:22 00:00   [gen]
19:48:22 00:00     [thrift]
19:48:22 00:00     [protoc]
19:48:22 00:00     [antlr]
19:48:22 00:00     [ragel]
19:48:22 00:00     [jaxb]
19:48:22 00:00     [wire]
19:48:22 00:00     [aapt]
19:48:22 00:00     [scrooge]
19:48:22 00:00     [spindle]
19:48:22 00:00   [resolve]
19:48:22 00:00     [ivy]
19:48:22 00:00   [compile]
19:48:22 00:00     [compile]
19:48:22 00:00     [jvm]
19:48:22 00:00       [jvm-compilers]
19:48:22 00:00     [python-eval]
19:48:22 00:00       [eval-targets]
19:48:22 00:00     [checkstyle]
19:48:22 00:00     [scalastyle]
19:48:22 00:00     [cpp-compile]
19:48:22 00:00   [resources]
19:48:22 00:00     [prepare]
19:48:22 00:00   [test]
19:48:22 00:00     [run_prep_command]
19:48:22 00:00     [test]
19:48:22 00:00     [pytest]
19:48:22 00:00       [run]
                     ============== test session starts ===============
                     platform darwin -- Python 2.7.6 -- py-1.4.27 -- pytest-2.6.4
                     plugins: cov, timeout
                     collected 10 items 

                     tests/python/pants_test/backend/jvm/tasks/test_ivy_utils.py ..........

                     =========== 10 passed in 0.52 seconds ============

19:48:25 00:03     [junit]
19:48:25 00:03   [complete]
               SUCCESS
AR
TE
  1. Ship It!
  2. 
      
JS
  1. The automatic excludes that Stu added are nice when they do what you want. But occasionally this isn't what you want

    You'll need to help me along here and likely help others with code comments or docs. I see 3 possibilities:
    1. FWICT this feature only doesn't do the right thing if you combine many libraries in one repo, use pants, and yet want each library to only depend on released jars of other libraries in the repo. Doing this globally is antithetical to housing all the libs in one repo. I assume global because you marked the option advanced as-if it were only intended to be set globally in pants.ini.
    2. The next possibility is you do intend the setting to be global, only have some libs that want to depend to jars instead of local source, and so accept the repo-wide penalty of having to apply manual excludes for all the other cases where source -> source is what you want.
    3. You actually intend the flag to be used from the command line in library-specific CI jobs and user-invocations

    Interested in hearing more / seeing more explanation in comments or pantsbuild.github.io docs.

    1. We're trying to merge science and birdcage, and we want to do some experiments with this code disabled. There are some things in pants.ini (scrooge gen, junit bootstrap) that are inserting a source dep on util-core, which is then triggering a massive exclude of the util-core jar in science.

      Since we're finding that useful, I thought it might be useful to have it in pants. But this doesn't need to be here -- I've already produced a hacked PEX to use internally.

      One thing I will say is that I the debug info is useful -- currently, when pants does an auto-exclude, it's totally silent. You don't see it in depmap, resolve, etc. It can be pretty tough to figure out that this feature is causing a target to fail.

    2. A few more points:

      1. This doesn't actually change any behavior -- I'm leaving the default as it is today.
      2. I made the option advanced, so it won't clutter up the help output under the normal use-case.
      3. We are finding this option useful inside of Twitter, and rather than leaving it off to the side in our fork, it would be a lot more convienent to have it as part of pants.
    3. The automatic exclusion is a big win for us. Fortunately, we haven't seen problems like this. There is a little bit of background I'm not aware of. util-core is a target inside of which repo? If there is already a source dep, then why doesn't the code get resolved?

    4. Yeah, sorry -- I explained it in Twitter-speak for John's benefit.

      Let's say that you have two separate git repositories, both configured to use pants. In repository A, the BUILD.tools file contains a target that in it's dependencies, has a source dep on a target within repo A. In repository B, the BUILD.tools file contains a jar dependency on the same target that is source in repo A.

      Now, let's further say that you want to merge these two repositories together. All of the code can be dropped into the new A+B repo cleanly (assuming it all lives in different paths), but you'll have to merge all of the pants support files (pants.ini, BUILD.tools, etc). Now, no matter which BUILD.tools you pick, one half of the targets will be unhappy. Eventually, you'll switch everything to the appropriate source deps, but while you're working that out, the automatic excludes will kick in early, and often, and obscure other work that you need to do.

  2. 
      
ZU
  1. Ship It!
  2. 
      
IT
  1. lgtm!

  2. 
      
AR
Review request changed

Status: Closed (submitted)

Loading...