Add SignApk product and Zipalign task.

Review Request #1737 - Created Feb. 8, 2015 and submitted

Information
Mateo Rodriguez
pants
ed78208...
Reviewers
pants-reviews
dturner-tw, jsirois, patricklaw

Zipalign is the final step in the basic Android app build.
This step is only required on apks signed with release keystores.

I also changed the return type of the KeystoreResolver into a dict.
Since it will usually return more than one keystore, it was easier
to test and use as a dictionary.

This also lays some groundwork for an upcoming refactor and
expansion of the android test harness. This commit adds an
TestAndroidBase class, which provides a couple useful mock
structures common to many Android tests.

./pants test tests/python/pants_test::
Travis passed: https://travis-ci.org/pantsbuild/pants/builds/50042476

Issues

  • 0
  • 5
  • 0
  • 5
Description From Last Updated
John Sirois
Mateo Rodriguez
Mateo Rodriguez
Patrick Lawson
Mateo Rodriguez
John Sirois
John Sirois
John Sirois
Mateo Rodriguez
Mateo Rodriguez
Review request changed

Status: Closed (submitted)

Eric Ayers

I have been having problems running tests locally with about 14 tests failing ./pants test tests/python/pants_test:all

with errros like:

                  src/python/pants/backend/jvm/tasks/ivy_task_mixin.py:221: TaskError
                     -------------- Captured stdout call --------------

                     11:37:00 00:00 [main]
                                    Invalidated 1 target containing 1 payload file.
                     11:37:00 00:00   [bootstrap-checkstyle]
                                      ==== stderr ====
                                      ivy configuration file not found: /private/var/folders/42/8krm7t_50x98_6rxp7p0tv1w000205/T/tmpWkc5IC_BUILD_ROOT/build-support/ivy/ivysettings.xml

I don't understand what about this change could have triggered this, but git bisect traced it back to

commit 3e8f26cfc471a828377621d6872b099aacbb7c3c
Author: Mateo Rodriguez <mateorod9@gmail.com>
Date:   Mon Feb 9 14:14:48 2015 -0500

    Add SignApk product and Zipalign task.

    Zipalign is the final step in the basic Android app build.
    This step is only required on apks signed with release keystores.

    Properly classify the goals tasks are installed into. Final
    .apk is gotten by calling the 'bundle' goal.

    This also lays some groundwork for an upcoming refactor and
    expansion of the android test harness. This commit adds an
    TestAndroidBase class, which provides a couple useful mock
    structures common to many Android tests.

    Testing Done:
    ./pants test tests/python/pants_test::
    Travis passed: https://travis-ci.org/pantsbuild/pants/builds/50042476

    Reviewed at https://rbcommons.com/s/twitter/r/1737/

I've confirmed it a couple of times now locally running this to test:

rm -rf ~/.ivy2 ; devpants clean-all; devpants test ./tests/python/pants_test:all

I tried reverted PR in master and the problem goes away.

I'm not sure what the difference is between my laptop and Travis-CI unless the ivy cache is hiding the issue.

  1. Can you give more context on the error?  If a tmp build root doesn't have build-support/ivy/ivysettings.xml this implies either the "real" build-support/ivy/ivysettings.xml got nuked in your clone (can happen with test errors when a test is being developed).
  2. ... and I've got no or yet, but the details may help - for instance what is the 1st failing test?
  3. The first failing task is test_checkstyle.py.

    test output is:

    ============== test session starts ===============
                         platform darwin -- Python 2.7.5 -- py-1.4.26 -- pytest-2.6.4
                         plugins: cov, timeout
                         collected 646 items 
    
                         tests/python/pants_test/test_binary_util.py ...
                         tests/python/pants_test/test_maven_layout.py ..
                         tests/python/pants_test/test_thrift_util.py ...
                         tests/python/pants_test/android/test_android_config_util.py ..
                         tests/python/pants_test/android/test_android_distribution.py ..
                         ============== test session starts ===============
                         platform darwin -- Python 2.7.5 -- py-1.4.26 -- pytest-2.6.4
                         plugins: cov, timeout
                         collected 646 items 
    
                         tests/python/pants_test/test_binary_util.py ...
                         tests/python/pants_test/test_maven_layout.py ..
                         tests/python/pants_test/test_thrift_util.py ...
                         tests/python/pants_test/android/test_android_config_util.py ..
                         tests/python/pants_test/android/test_android_distribution.py ..
                         tests/python/pants_test/android/test_keystore_resolver.py ........
                         tests/python/pants_test/android/tasks/test_aapt_gen.py ..
                         tests/python/pants_test/android/tasks/test_sign_apk.py ........
                         tests/python/pants_test/android/tasks/test_zipalign.py ....
                         tests/python/pants_test/authentication/test_netrc_util.py ....
                         tests/python/pants_test/backend/codegen/targets/test_java_protobuf_library.py .....
                         tests/python/pants_test/backend/codegen/targets/test_java_wire_library.py ...
                         tests/python/pants_test/backend/codegen/tasks/test_protobuf_parse.py .x..Xxxx...
                         tests/python/pants_test/backend/codegen/tasks/test_wire_gen.py ...
                         tests/python/pants_test/backend/core/test_wrapped_globs.py ............
                         tests/python/pants_test/backend/jvm/test_jvm_debug_config.py ..
                         tests/python/pants_test/backend/jvm/targets/test_jvm_target.py .
                         tests/python/pants_test/backend/jvm/targets/test_unpacked_jars.py ..
                         tests/python/pants_test/backend/jvm/tasks/test_checkstyle.py .FF ........
    

    I commented out all the android tasks tests and now test_checkstyle.py passes so I'll start looking in there.

  4. Commenting out not just one of but both of tests/python/pants_test/android/tasks/test_sign_apk.py and tests/python/pants_test/android/tasks/test_zipalign.py makes the problem go away. Still looking at those tests - they share a new base class.

  5. I repro here.  The key is running something CI does not, `./pants test tests/python/pants_test:all`.  I'll back off investigation since Eric is in the thick of it but give me a yellif you need a hand.
  6. I'm about to give up on this and suggest we revert. Thus far I found these fixups, but they don't solve the problem.

    diff --git a/tests/python/pants_test/android/BUILD b/tests/python/pants_test/android/BUILD
    index d6aafc0..6daa851 100644
    --- a/tests/python/pants_test/android/BUILD
    +++ b/tests/python/pants_test/android/BUILD
    @@ -19,7 +19,7 @@ python_test_suite(
       ]
     )
    
    -python_tests(
    +python_library(
       name = 'android_integration_test',
       sources = [
         'android_integration_test.py',
    @@ -30,12 +30,13 @@ python_tests(
       ],
     )
    
    -python_tests(
    +python_library(
       name = 'android_base',
       sources = [
         'test_android_base.py',
       ],
       dependencies = [
    +    '3rdparty/python/twitter/commons:twitter.common.collections',
         'src/python/pants/backend/android/targets:android',
         'src/python/pants/backend/android/tasks:all',
         'src/python/pants/util:contextutil',
    

    I'll also note that the test dir structure

    tests/python/pants_tests/android
    

    don't match the source package

    src/python/pants/backend/android
    

    which seems inconsistent but I don't think it would case a problem.

    I found a shorter way to reproduce the error too:

    $ PANTS_DEV=1 test tests/python/pants_test/backend/jvm/tasks:checkstyle tests/python/pants_test/backend/android/tasks:sign_apk
    # works
    
    $ PANTS_DEV=1 test tests/python/pants_test/backend/android/tasks:sign_apk tests/python/pants_test/backend/jvm/tasks:checkstyle
    # fails
    
  7. Fix here: https://rbcommons.com/s/twitter/r/1750/
    The only issue with Mateo's change is that it changed the order tests were run in ./pants test tests/python/pants_test:all

Loading...