Implicitly provide source products.

Review Request #4053 - Created July 7, 2016 and submitted

Information
Benjy Weinberger
pants
Reviewers
pants-reviews
nhoward_tw, peiyu, zundel
The ide project gen tasks require the 'java', 'scala' products.
The only reason this works is that the scrooge task happens to
provide those products.  But if we try to deregister the codegen
backend in some repo that doesn't need it, we get a MissingProductError
from the round engine, and a quite unintuitive error message.

This change makes every source root implicitly provide a product
with the same name as that root's language(s).  This makes logical
sense, and also solves the problem above.

While testing this, I noticed an error in how we iterate over all
source roots: We assumed that all fixed source roots come from
the appropriate options. But in fact they might also be added
programmatically.  So this change also fixes that, and adds
appropriate testing.

CI passed on travis: https://travis-ci.org/pantsbuild/pants/builds/142920098

Jenkins seemed to be having issues.

Eric Ayers
Ity Kaul
Eric Ayers
Benjy Weinberger
Benjy Weinberger
Review request changed

Status: Closed (submitted)

Change Summary:

cb78b0ef7bc464fde4dfde2fb5aaecc18900785c

Eric Ayers

Wowzers! After this change was merged to master, the time for shard 4 went up to about 1 hour 30 minutes!

https://travis-ci.org/pantsbuild/pants/builds/143384091

Its not just a one time fluke, its happening on PR builds and more than one master build.

Its not obvious to me why (maybe it has something to do with Travis and not this change?)

I downloaded the previous master build - the output length is nearly identical, but the timestamps we output show a big difference:

Was ~ 20 seconds:

11:23:17 21:21         [chroot]ESC[1m^M                     collecting 0 itemsESC[0mESC[1m^M                     collecting 4 itemsESC[0mESC[1m^M                     collecting 4 itemsESC[0mESC[1m^M                     collected 4 items 
                     ESC[0m
                     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_render_args ESC[32mPASSEDESC[0m
                     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_zipalign_binary ESC[32mPASSEDESC[0m
                     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_zipalign_out ESC[32mPASSEDESC[0m
                     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_zipalign_smoke ESC[32mPASSEDESC[0m

                     ============ slowest 3 test durations ============
                     0.02s call     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_render_args
                     0.00s call     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_zipalign_binary
                     0.00s call     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_zipalign_out
                     ESC[32mESC[1m============ 4 passed in 0.46 seconds ============ESC[0m
                     ESC[1m============== test session starts ===============ESC[0m
                     platform linux2 -- Python 2.7.3 -- py-1.4.31 -- pytest-2.6.4 -- /usr/bin/python2.7
                     plugins: cov, timeout
                     ESC[1m^M                     collecting 0 itemsESC[0mESC[1m^M                     collecting 0 itemsESC[0mESC[1m^M                     collecting 7 itemsESC[0mESC[1m^M                     collecting 7 itemsESC[0mESC[1m^M          
           collected 7 items 
                     ESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_binary ESC[32mPASSEDESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_binary_compile ESC[32mPASSEDESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_binary_with_library ESC[32mPASSEDESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_binary_with_library_compile ESC[32mPASSEDESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_library ESC[32mPASSEDESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_library_compile ESC[32mPASSEDESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_run 
11:23:41 21:45         [chroot]ESC[32mPASSEDESC[0m

Now > 15 minutes:

17:58:43 45:14         [chroot]ESC[1m^M                     collecting 0 itemsESC[0mESC[1m^M                     collecting 4 itemsESC[0mESC[1m^M                     collecting 4 itemsESC[0mESC[1m^M                     collected 4 items 
                     ESC[0m
                     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_render_args ESC[32mPASSEDESC[0m
                     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_zipalign_binary ESC[32mPASSEDESC[0m
                     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_zipalign_out ESC[32mPASSEDESC[0m
                     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_zipalign_smoke ESC[32mPASSEDESC[0m

                     ============ slowest 3 test durations ============
                     0.02s call     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_render_args
                     0.01s call     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_zipalign_out
                     0.00s call     contrib/android/tests/python/pants_test/contrib/android/tasks/test_zipalign.py::TestZipalign::test_zipalign_binary
                     ESC[32mESC[1m============ 4 passed in 0.41 seconds ============ESC[0m
                     ESC[1m============== test session starts ===============ESC[0m
                     platform linux2 -- Python 2.7.3 -- py-1.4.31 -- pytest-2.6.4 -- /usr/bin/python2.7
                     plugins: cov, timeout
                     ESC[1m^M                     collecting 0 itemsESC[0mESC[1m^M                     collecting 0 itemsESC[0mESC[1m^M                     collecting 7 itemsESC[0mESC[1m^M                     collecting 7 itemsESC[0mESC[1m^M                     collected 7 items 
                     ESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_binary ESC[32mPASSEDESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_binary_compile ESC[32mPASSEDESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_binary_with_library ESC[32mPASSEDESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_binary_with_library_compile ESC[32mPASSEDESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_library ESC[32mPASSEDESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_library_compile ESC[32mPASSEDESC[0m
                     contrib/cpp/tests/python/pants_test/contrib/cpp/test_cpp_integration.py::CppIntegrationTest::test_cpp_run 
18:15:19 61:50         [chroot]ESC[32mPASSEDESC[0m
  1. Hmm, that sucks. I have a hypothesis, basically this comment: https://github.com/pantsbuild/pants/blob/master/src/python/pants/source/source_root.py#L103

    Calling SourceRoots.all_roots does do a filesystem walk to look for matching subdirs, with a bunch of excluded dirs (.git, .pants.d etc.) and short-circuiting to make it fast. However that's no guarantee of course, if there are other dirs that we should be excluding but aren't, then that could be a cause of slowness.

    Now that I think of it, I should have been more cautious here. Previously the only things that used all_roots() were Go buildgen, and the 'roots' task. Now every build is invoking this.

    Actions (should this hypothesis turn out to be true):

    A) Replace this logic with simpler logic. Among the options are, in increasing order of complexity:
    1. Don't error when there are no providers of some product, just assume the product is empty.
    2. A milder version of 1: Allow registration of "optional products" that are not required to exist.
    3. Let backends register the languages they implicitly provide, instead of searching for them on the filesystem.

    B) Find the dirs that are causing this slowness and add them to the excluded dirs. Actually, we should probably just exclude all hidden dirs. This is just to make all_roots faster for its other use-cases. I still don't think we should call it on every build.

  2. Oh, but I seem to remember that Square, at least, has sources in hidden dirs? Or something like that?

  3. I don't recall this being something that we do at Square (source in hidden dirs)

Loading...