[engine] Implement recursive address walking

Review Request #3485 - Created Feb. 20, 2016 and submitted

Stu Hood
2943, 2944
benjyw, ity, jsirois, kwlzn, patricklaw, peiyu

This review implements the commandline syntax x/: and x/:: for the new engine via recursive directory walks inside the engine.

  • Move the "cast from Struct to literal type" special case out of the Scheduler and into graph tasks. Huzzah!
  • Add additional datatypes to represent recursive and non-recursive directory listings
    • In particular, Addresses is the entry point for both SiblingAddresses and DescendantAddresses. They are bootstrapped in BuildRequest by using DependenciesNode to request goal products for each Address in an Addresses object for their subject.
  • Implement SiblingAddresses by converting an AddressFamily into Addresses
  • Implement DescendantAddresses by converting RecursiveSubDirectories into Addresses, and producing RecursiveSubDirectories via recursive requests for the SubDirectories of Directories.
    • This change introduces one slightly annoying special case, which is that we need to use an identity function to project the Directory field of the DescendantAddresses object in order to begin the recursion. This is pretty minor, but likely an indication that the Selector api still needs more polish.
  • Refresh the src/python/pants/engine/exp/legacy:list command. The performance is approximately on par with the existing engine, but path-excludes/build-ignore-patterns are not yet implemented.
  • Removed self._roots from the Scheduler. Instead, they are a property of a BuildRequest.

A lovely emergent behaviour of this change is that adding a "list" goal was as simple as aliasing 'list': Address. The root result is an Address for every member of the Addresses object... ie, identity. The following run is rendered and attached:

./pants run src/python/pants/engine/exp/examples:viz -- \
  tests/python/pants_test/engine/exp/examples/scheduler_inputs \
  list \



Kris Wilson
Stu Hood
Peiyu Wang
Peiyu Wang
Stu Hood
Review request changed

Status: Closed (submitted)

Change Summary:

Merged as 1e3f7071cf4095abfface461164703106563c459

Timur Abishev

adding a "list" goal was as simple as aliasing 'list': Address


src/python/pants/engine/exp/graph.py (Diff revision 2)

Is it true that this path field actually rel_path? Maybe better to rename?

  1. Yes it is relative... but as of r3488 everything in the fs, graph, and planners module is relative. The only thing aware of the location of the buildroot is the ProjectTree.

Looks like now method returns True/False, not list.

  1. Thanks, will fix in r3488.