Avoid using expensive bootstrap artifacts from temporary cache location
Review Request #4368 — Created Nov. 11, 2016 and submitted — Latest diff uploaded
|benjyw, nhoward_tw, stuhood|
shading zinc is expensive, currently master shading takes 30seconds, new zinc jar is 50% larger, so
will take even longer.
cache-write-tolocations to temp, including
bootstrap, so unless the default bootstrap location already contains the artifacts, every
BaseCompileITtest will end up doing the expensive shading for every its pants run. One
of the tests
test_zinc_fatal_warningdoes 8 pants runs.
The uncached scenario can happen when doing an zinc upgrade like in https://rbcommons.com/s/twitter/r/4342/
or on master, if travis ci shard is not locally cached (from previous travis runs), and schedules
BaseCompileITtests before other pants runs to bootstrap the default location.
So this fix should help to reduce the extreme slow test cases from
BaseCompileIT, sampled some
previous travis runs, that seems to happen from time to time (almost anytime a shard goes > 30minutes)
Use a fixed location (
cache-bootstrapand write everything else still to temp
(by taking advantage passing
cache-write-toparam through override pants.ini instead of cmdline,
the latter takes precedence over all pants.ini cache configurations, the former just merge therefore
keep the default bootstrap cache location)
Before on master:tw-mbp-peiyu:pants peiyu$ rm ~/.cache/pants/artifact_cache/pants_backend_jvm_tasks_bootstrap_jvm_tools_BootstrapJvmTools/.zinc/d502916b113b472095ea4ca2aa05c97c81106c91-ShadedToolFingerprintStrategy_53c4bb9c6553.tgz tw-mbp-peiyu:pants peiyu$ pt tests/python/pants_test/backend/jvm/tasks/jvm_compile/zinc:zinc_compile_integration_with_zjars --no-test-pytest-timeouts -- -k test_zinc_fatal_warning ... === 1 passed, 11 deselected in 317.94 seconds ====
After:tw-mbp-peiyu:pants peiyu$ pt tests/python/pants_test/backend/jvm/tasks/jvm_compile/zinc:zinc_compile_integration_with_zjars --no-test-pytest-timeouts -- -k test_zinc_fatal_warning ... ==== 1 passed, 11 deselected in 93.23 seconds ==== tw-mbp-peiyu:pants peiyu$ ls .cache/pants_backend_jvm_tasks_bootstrap_jvm_tools_BootstrapJvmTools/.zinc/d502916b113b472095ea4ca2aa05c97c81106c91-ShadedToolFingerprintStrategy_53c4bb9c6553.tgz .cache/pants_backend_jvm_tasks_bootstrap_jvm_tools_BootstrapJvmTools/.zinc/d502916b113b472095ea4ca2aa05c97c81106c91-ShadedToolFingerprintStrategy_53c4bb9c6553.tgz