Add baseline functionality for engine experiments.
Review Request #2914 - Created Oct. 1, 2015 and submitted
|benjyw, ity, kwlzn, patricklaw, stuhood|
This introduces a new Addressable/Target parsing system based on named Serializable objects and Addressed pointers. A new Graph implementation is built on these and one example graph is included in three serialized forms. A planner can now be built that consumes example graphs and outputs execution plan graphs. src/python/pants/base/address.py | 2 +- src/python/pants/engine/exp/BUILD | 61 +++++++++++ src/python/pants/engine/exp/__init__.py | 0 src/python/pants/engine/exp/addressable.py | 61 +++++++++++ src/python/pants/engine/exp/graph.py | 112 ++++++++++++++++++++ src/python/pants/engine/exp/mapper.py | 155 ++++++++++++++++++++++++++++ src/python/pants/engine/exp/parsers.py | 248 ++++++++++++++++++++++++++++++++++++++++++++ src/python/pants/engine/exp/serializable.py | 36 +++++++ src/python/pants/engine/exp/targets.py | 126 +++++++++++++++++++++++ tests/python/pants_test/engine/exp/BUILD | 21 ++++ tests/python/pants_test/engine/exp/__init__.py | 0 tests/python/pants_test/engine/exp/examples/graph_test/self_contained.BUILD | 51 +++++++++ tests/python/pants_test/engine/exp/examples/graph_test/self_contained.BUILD.json | 59 +++++++++++ tests/python/pants_test/engine/exp/examples/graph_test/self_contained.BUILD.python | 46 +++++++++ tests/python/pants_test/engine/exp/test_graph.py | 68 ++++++++++++ tests/python/pants_test/engine/exp/test_parsers.py | 280 ++++++++++++++++++++++++++++++++++++++++++++++++++ 16 files changed, 1325 insertions(+), 1 deletion(-)
CI went green here: https://travis-ci.org/pantsbuild/pants/builds/83049418
NB: This is just a basis for writing example input graphs and Planners (new concept) that can process the graphs to produce Task plans that are combined into a schedule by a Scheduler (new concept - like an Engine front-end). Its more polished than it needs to be - pydoc, extra parsers - but these things may grow so I spent a bit of extra time - just don't be deceived! The idea is to move fast and light through some experimental code from here.
Looks great, for what it is. I still fear that this is too complicated. For a large graph this will involve creating a lot of wrapper objects.
How much simpler can we make this if we get rid of the notion of types entirely? I.e., the build graph consists of vertices all of a single type (Target) and edges of a single type (Address), and we handle Config specially in some way. I know I'm not seeing a lot of detail here, but I would really really love to see the build graph implementation be dumb and simple.
I just think that making build graphs really quick to parse and trivial to reason about is a laudable goal.
Do we really need all these micro-targets, one per source file? I'd like us to get away from this as much as possible.
At Foursquare we have a #bikeshed Slack channel for such discussions.