[engine] Implement BUILD file parsing inside the engine

Review Request #3377 - Created Jan. 26, 2016 and submitted

Stu Hood
2840, 2865
benjyw, ity, jsirois, nhoward_tw, patricklaw, peiyu

Rather than having separate build and product graphs, there is now only one graph. All invalidation of changed values can happen directly in the ProductGraph via invalidation of Nodes that represent filesystem operations.

Additionally, because all file parsing (BUILD or otherwise) happens via the same mechanism, this review also includes a very simple example of dependency inference for scala code (based on looking up imported packages in configured source roots). A series of tasks look for inferred_scala Sources, and parse/infer/request their deps to convert them to scala Sources.

  • Replace the Graph class with tasks that compute Structs by parsing Directories
    • The resolve_unhydrated_struct function produces an UnhydratedStruct object, which declares its additional inline Addressable dependencies. The hydrate_struct function then takes the UnhydratedStruct and its dependencies to hydrate it into a complete value that can be validated.
    • Supports all mergeability/inlineability features of the current implementation (inline Jars, etc), but is much lazier than the current graph in that dependencies of embedded Structs are not walked unless requested by a task.
  • Implement cycle detection in ProductGraph to replace the implementation in Graph
  • Formalize Parsers and SymbolTables as top-level classes, which avoids having to pickle them while multi-processing. Instead, they are passed directly as class references.
  • Remove caching from AddressMapper, since the product graph nodes are normalized to act as their own in-process cache.
  • Add SelectProjection, which allows a Task to request a Product for one field of an input Subject.
    • This allows for de-duplicating within the graph. For example, the hydrate_struct task requests an AddressFamily for a Directory projected from the spec_path field of an Address, which causes AddressFamilies to be cached keyed by their Directory, rather than by an Address.
  • Killed NativeNode in favor of immediately satisfying a SelectNode with the subject value when possible.
  • Add an example of dependency inference for scala code via the extract_scala_imports/reify_scala_sources family of tasks.
    • See the attached dependency-inference.svg rendering for an example.




  • 0
  • 7
  • 4
  • 11
Description From Last Updated
Stu Hood
Nick Howard (Twitter)
Stu Hood
Stu Hood
Peiyu Wang
Peiyu Wang
Stu Hood
Stu Hood
Stu Hood
Stu Hood
Review request changed

Status: Closed (submitted)

Change Summary:

Merged --tbr as 02aad68bd3ee98e864bb22c1fba0b6493bad6b61