Parallelize isolated compile strategy's chunk execution.

Review Request #2109 - Created April 22, 2015 and submitted

Information
Nick Howard (Twitter)
pants
1443
2aebeba...
Reviewers
pants-reviews
benjyw, davidt, jsirois, stuhood, zundel

Instead of applying compiles in topological sorted order, schedule each compilation unit as part of an execution graph, then
run through the graph submitting each piece of work as its dependencies have successfully finished.

I've introduced an abstraction, ExecutionGraph, around scheduling and executing a graph of dependent work units. I did this instead of just introducing a WorkPool inside compile_chunk. I think being able to generalize parallelizing dependent work will be valuable in other places. I wanted to prepare for being able to do that in other places without having to re-cut abstraction boundaries later.

CI passing at https://travis-ci.org/pantsbuild/pants/builds/59897269

Ran successfully on internal target graphs with interleaved java / scala dependencies, as well as ones with multiple targets per chunk.

Issues

  • 0
  • 13
  • 1
  • 14
Description From Last Updated
Nick Howard (Twitter)
Nick Howard (Twitter)
Nick Howard (Twitter)
Stu Hood
Nick Howard (Twitter)
Stu Hood
Nick Howard (Twitter)
Stu Hood
Nick Howard (Twitter)
Benjy Weinberger
Benjy Weinberger
Nick Howard (Twitter)
Eric Ayers
Benjy Weinberger
Benjy Weinberger
Nick Howard (Twitter)
Review request changed

Status: Closed (submitted)

Change Summary:

https://github.com/pantsbuild/pants/commit/a0a79225f0d735989ded2606d179c32f18a8ca3d

Loading...