[ExecutionGraph] DEMO: Introduce job prioritization

Review Request #2601 - Created Aug. 11, 2015 and submitted

Information
Sergey Serebryakov
pants
sserebryakov/compile_prio
2109
bb005f6...
Reviewers
pants-reviews
gmalmquist, kwlzn, nhoward_tw, patricklaw, stuhood

See googledoc for details: https://docs.google.com/a/twitter.com/document/d/1BGVAR_AdGXsf08MT0nBLh8s-lYenx7NK7Kk7keD_gwg/edit?usp=sharing

PROBLEM:
The jobs are submitted to the worker pool in a sort of insertion order. If the submission order was based on the “size” or “importance” of the job, it could result in less total execution time.

SOLUTION:
Introduce a priority queue where ready-to-be-submitted jobs are waiting for the moment when a worker becomes available, being sorted by “importance”. When a job finishes, its unblocked dependees is not submitted to the worker pool, but rather put in the priority queue. When a worker becomes available (i.e. when any job is finished), the most “important” job is taken from the priority queue. There is an additional latency (before, ready jobs are submitted to the worker pool immediately; after, they are all first put in the priority queue, and only then the most prioritized ones are chosen and submitted).

Additionally, make tracking of ready-to-be-submitted jobs smarter with a counter in status table which is decremented with every finished dependency.

$ ./pants test tests/python/pants_test:all
SUCCESS

CI green: https://travis-ci.org/megaserg/pants/builds/78164959

Issues

  • 0
  • 6
  • 0
  • 6
Description From Last Updated
Kris Wilson
Sergey Serebryakov
Stu Hood
Sergey Serebryakov
Nick Howard (Twitter)
Stu Hood
Sergey Serebryakov
Sergey Serebryakov
Nick Howard (Twitter)
Stu Hood
Sergey Serebryakov
Sergey Serebryakov
Sergey Serebryakov
Review request changed

Status: Closed (submitted)

Change Summary:

Merged as a645a3560ae8743c2bf4459d2855af90198fc3ff

Loading...