[engine] Native scheduler implementation

Review Request #4270 — Created Sept. 28, 2016 and submitted — Latest diff uploaded

3821, 4030
benjyw, jsirois, mateor, nhoward_tw, yujiec

This is the preliminary review of a native (in rust) implementation of the inner loop of the v2 engine.

See the Pants Native Engine document for motivation for the use of an additional language.

  • Add a native implementation of most of {scheduler,nodes,selectors}.py, with access to the native binary provided by binary_utils in engine/subsystems/native.py
  • Replace legacy/graph.py's usage of ProductGraph.walk with an explicit request for transitive dependencies implemented via SelectDependencies(.., transitive=True). This avoids leaking the implementation details of the ProductGraph, and keeps the API boundary smaller.

The interface between the native code and python is largely defined in src/rust/engine/src/lib.rs and src/python/pants/engine/subsystem/native.py, which (respectively) define the C-compatible public API, and the CFFI definitions to consume it.


See https://github.com/pantsbuild/pants/labels/native for a list of tests that are being temporarily disabled in order to expedite landing this.