[GroupTask] Add OmnivorousZincCompile which consumes both java and scala sources

Review Request #2561 - Created Aug. 3, 2015 and submitted

Information
Sergey Serebryakov
pants
sserebryakov/single_wonderful_task
1830
b70ce32...
Reviewers
pants-reviews
benjyw, nhoward_tw, stuhood, zundel

Expected reviewers: stuhood, nhoward

NOTE:
OmnivorousZincCompile mentioned in the title is just modified ZincCompile (it used to be initially a separate subclass, which was then merged into ZincCompile).

PROBLEM:
Currently, java and scala targets are chunked separately, and (with isolated strategy) a separate execution graph is created for every chunk by JavaZincCompile and ScalaZincCompile. These graphs are then executed sequentially, meaning there are unnecessary artificial barriers in the execution flow, increasing the total compilation time.

Also, when Zinc is run in Nailgun, the static final caches like analysisCache are preserved in Nailgun between Zinc invocations. Since every GroupMember is a NailgunTask, and NailgunTask uses NailgunExecutor parameterized by the task's class name (actually by the executor_workdir, which includes the task's class name), different GroupMembers use different NailgunExecutors, ergo different Nailgun instances. In particular, JavaZincCompile and ScalaZincCompile use two different Nailgun instances, which means storing two separate copies of Zinc's internal state. While not necessarily affecting performance, this increases memory pressure.

SOLUTION:
Define a new GroupMember which subclasses ZincCompile and consumes both java and scala sources. In GroupTask, it will claim all java and scala targets (the whole build graph), which will form a single chunk. When isolated strategy is used, the build graph is translated to the execution graph, which is executed in a single instance of the worker pool. Also, naturally, only one Nailgun instance is used to run Zinc for both languages, therefore only one copy of Zinc caches is stored.

Additionally, --enabled argument is added for jmake-based JavaCompile GroupMember, "True" by default.

RESULT:
The sequence of GroupMembers now is: [AptCompile, JavaCompile, ZincCompile].
1. All AnnotationProcessor Java targets are claimed by jmake-based AptCompile.
2a. When --no-compile-java-enabled is used, JavaCompile becomes a no-op. All remaining Java and Scala targets are claimed by ZincCompile and compiled in a single chunk as described above. Using isolated strategy is recommended here.
2b. When JavaCompile is not disabled, it claims all remaining Java targets. The remaining Scala targets are claimed by ZincCompile, so it behaves like ScalaZincCompile.

One should be able to painlessly test the single-chunk approach vs. multiple-chunks approach, as well as global vs. isolated strategy, comparing the performance.

Soon-to-be-green CI: https://travis-ci.org/pantsbuild/pants/builds/74337800

> ./pants test tests/python/pants_test:all
> SUCCESS

Issues

  • 1
  • 3
  • 2
  • 6
Description From Last Updated
We should also add compile.scala and maybe compile.zinc-java as new entries in migrate_config.py for Nick Howard (Twitter) Nick Howard (Twitter)
Stu Hood
Sergey Serebryakov
Sergey Serebryakov
Benjy Weinberger
Nick Howard (Twitter)
Eric Ayers
Sergey Serebryakov
Sergey Serebryakov
Stu Hood
Benjy Weinberger
Sergey Serebryakov
Review request changed

Status: Closed (submitted)

Change Summary:

Merged as ee99c80794e6b77a724bc17194165b971da0a181

Loading...