[jvm-compile] narrow compile dependencies from full closure to just next nearest invalid compilation targets

Review Request #4136 — Created Aug. 4, 2016 and submitted

nhoward_tw
pants
3750
pants-reviews
benjyw, jsirois, stuhood

I was in this code a couple months ago and noticed that each compile unit in the compile graph depends on the full transitive closure of its invalid dependencies.

This reduces that to just the nearest ones. It makes the debug graphs much more reasonable to render, and should reduce the scheduler overhead of large graphs somewhat.

Tests passed last time, but I've merged master and am rerunnning CI on the associated pull request.

  1. Looks good, but I don't even remember why jvm_compile actually knows or cares about what is invalid upstream from it. Shouldn't target T's dependencies all be valid by the time we get to compiling T?

    1. The dependency collection here is before compilation happens. So, this is determining the graph that is passed to ExecutionGraph. Before this change, the graph's edges were hugely redundant for some trees.

    2. Oh right, this is because we can schedule concurrent jvm compilation jobs, and they have to execute in topological order.

  2. 
      
Review request changed

Status: Closed (submitted)

Change Summary:

Submitted as https://github.com/pantsbuild/pants/commit/c2cd962bc4758a728a2fe30fe951e0ab49f5b433
Loading...