Avoid using expensive bootstrap artifacts from temporary cache location

Review Request #4368 - Created Nov. 11, 2016 and submitted

Information
Peiyu Wang
pants
4047
Reviewers
pants-reviews
benjyw, nhoward_tw, stuhood

Problem:

shading zinc is expensive, currently master shading takes 30seconds, new zinc jar is 50% larger, so
will take even longer. BaseCompileIT overrides all cache-write-to locations to temp, including
bootstrap, so unless the default bootstrap location already contains the artifacts, every
BaseCompileIT test will end up doing the expensive shading for every its pants run. One
of the tests test_zinc_fatal_warning does 8 pants runs.

The uncached scenario can happen when doing an zinc upgrade like in https://rbcommons.com/s/twitter/r/4342/
or on master, if travis ci shard is not locally cached (from previous travis runs), and schedules
BaseCompileIT tests before other pants runs to bootstrap the default location.

So this fix should help to reduce the extreme slow test cases from BaseCompileIT, sampled some
previous travis runs, that seems to happen from time to time (almost anytime a shard goes > 30minutes)

https://travis-ci.org/pantsbuild/pants/jobs/173113868 test_zinc_fatal_warning 516.30s
https://travis-ci.org/pantsbuild/pants/jobs/173113870 test_zinc_fatal_warning 489.52s
https://travis-ci.org/pantsbuild/pants/jobs/172183627 test_zinc_fatal_warning 505.19s

Solution:

Use a fixed location (buildroot/.cache) for cache-bootstrap and write everything else still to temp
(by taking advantage passing cache-write-to param through override pants.ini instead of cmdline,
the latter takes precedence over all pants.ini cache configurations, the former just merge therefore
keep the default bootstrap cache location)

https://travis-ci.org/peiyuwang/pants/builds/175149101
https://travis-ci.org/peiyuwang/pants/builds/176813195

Before on master:

tw-mbp-peiyu:pants peiyu$ rm ~/.cache/pants/artifact_cache/pants_backend_jvm_tasks_bootstrap_jvm_tools_BootstrapJvmTools/.zinc/d502916b113b472095ea4ca2aa05c97c81106c91-ShadedToolFingerprintStrategy_53c4bb9c6553.tgz 
tw-mbp-peiyu:pants peiyu$ pt tests/python/pants_test/backend/jvm/tasks/jvm_compile/zinc:zinc_compile_integration_with_zjars --no-test-pytest-timeouts -- -k test_zinc_fatal_warning 
...
                     === 1 passed, 11 deselected in 317.94 seconds ====

After:

tw-mbp-peiyu:pants peiyu$ pt tests/python/pants_test/backend/jvm/tasks/jvm_compile/zinc:zinc_compile_integration_with_zjars --no-test-pytest-timeouts -- -k test_zinc_fatal_warning
...
                     ==== 1 passed, 11 deselected in 93.23 seconds ====

tw-mbp-peiyu:pants peiyu$ ls .cache/pants_backend_jvm_tasks_bootstrap_jvm_tools_BootstrapJvmTools/.zinc/d502916b113b472095ea4ca2aa05c97c81106c91-ShadedToolFingerprintStrategy_53c4bb9c6553.tgz 
.cache/pants_backend_jvm_tasks_bootstrap_jvm_tools_BootstrapJvmTools/.zinc/d502916b113b472095ea4ca2aa05c97c81106c91-ShadedToolFingerprintStrategy_53c4bb9c6553.tgz

Issues

  • 0
  • 3
  • 1
  • 4
Description From Last Updated
Peiyu Wang
Peiyu Wang
Benjy Weinberger
Benjy Weinberger
Benjy Weinberger
Peiyu Wang
Benjy Weinberger
Peiyu Wang
Review request changed

Status: Closed (submitted)

Change Summary:

Merged as eb106d9ebcc5a22fd4c439d55c0dd32a93564bc3

Loading...