[engine] Rule Graph construction perf improvements

Review Request #4281 — Created Oct. 5, 2016 and submitted

kwlzn, stuhood, yujiec

This follows up https://rbcommons.com/s/twitter/r/4251/, cleaning up the perf todos.

It passes through the full graph construction's intermediate states so that they can short circuit subsequent subgraph constructions.

It also moves transitive unfulfillable rule elimination so that it happens once for both full graph and subgraph instead of once per construction.

There's also a bug fix in RuleEdges matching, which was caused by matching being overly broad.

Ran engine tests locally. CI passed at https://travis-ci.org/pantsbuild/pants/builds/165317483

  1. Ship It!
    1. I have a vague idea of what is going on here, but expect it to become more clear when this is actually used to instantiate Nodes.

    2. Mostly it's just eliminating a whole bunch of redundant work.

  1. Ship It!
  1. lgtm!

  2. src/python/pants/engine/rules.py (Diff revision 1)

    nit: indent

Review request changed

Status: Closed (submitted)

Change Summary:

Submitted as https://github.com/pantsbuild/pants/commit/7a79f52f014d838d2178c4e2c08b1bb2b8ce4c15