[engine] First round of work for 'native' filesystem support

Review Request #3488 - Created Feb. 22, 2016 and submitted

Stu Hood
2946, 2951
ity, jsirois, kwlzn, patricklaw, peiyu

Extract an engine.exp.fs module to handle all filesystem operations. Currently it contains the (existing) support for recursive directory walks, and some new support for requesting the file contents of literal globs (PathGlobs). Support for recursive globs (ie, src/**/*.java) will come in the next half of the review.

This is the first half of #2946, but it shuffles a bunch of modules around, so I wanted to get it out sooner rather than later.

  • Extract nodes, selectors, sources, and fs modules in order to shrink scheduler, and to allow fs to depend on selectors and expose its own tasks.
  • Many things in the fs package will eventually be "native", in that they will be implemented without the end-user task API. But for now it's easiest to experiment with them as tasks.
  • Add PathGlobs, which consumes the (apparently static!) methods of wrapped_globs.(Globs,RGlobs,ZGlobs) to normalize inputs into easily parsed 'filespecs'.
  • Add Sources.path_globs, which can be projected from a Sources object in order to request actual Paths or FileContents.
    • This is used in this review to power all filesystem access for BUILD file parsing and scala dep inference.
  • Remove outdated methods of Sources and Target which executed graph and filesystem walks.
  • Add Locatable interface, to mark a Serializable as needing to receive the 'spec_path' it was parsed from.
    • Structs sometimes do not have a name (and so may not have an Address), but all Structs were parsed from some namespace. The purpose of the interface is to avoid storing the location a Struct was parsed from in cases where it isn't needed later.
  • Add an explicit ProjectionNode to allow SelectProjection to work in cases where a Subject is not already of the necessary input type, but might be after selection.
    • This is a case very similar to Dependencies, where we first select a 'deps_product', and then project its dependencies. There is obviously still room to improve the selection syntax.
  • Add another visualizer entrypoint to support executing the equivalent of "ls" using the engine (and visualizing the result).
    • Since PathGlobs only supports literals at the moment, this basically just serves to tell you whether a file exists. In the next review, this will additionally support expanding globs... see comments in PathGlobs.create.

And: demo time. Attached is a rendering of the following command, which specifies a buildroot of . (ie, the pants working directory), and uses only engine tasks to confirm that examples/README.md exists. Pheew.

./pants run src/python/pants/engine/exp/examples:viz-fs -- . walk examples/README.md




  • 0
  • 2
  • 0
  • 2
Description From Last Updated
Stu Hood
Stu Hood
Kris Wilson
Timur Abishev
Peiyu Wang
Stu Hood
Stu Hood
Review request changed

Status: Closed (submitted)

Change Summary:

Merged as 08bc2add6e98feefdce36f6f094a15e3e598b1e7

Timur Abishev
Ship It!