[engine] Introduce content-addressability

Review Request #3498 - Created Feb. 25, 2016 and submitted

Stu Hood
2966, 2971
benjyw, ity, jsirois, kwlzn, patricklaw, peiyu

This review converts all Subject values to SubjectKeys, and stores subjects in a dumb content-addressable dictionary. The idea is that in the longer term, subject values go in local databases like sqlite/leveldb/etc (see #2968), in caches, etc.

  • Introduce SubjectKey which is eq'd/hash'd using only the digest it contains. It additionally optionally contains the str representation of the Subject, for debugging purposes.
  • Remove the lazily-memoized hash from Address in order to make it internally immutable, and eagerly hash it. Since Adresses are almost always used in sets/collections, this should not have a performance impact.
  • Add a temporary solution to ignore dotfiles in the fs package (until we dive into #2956).
  • Generate BuildRequests with scheduler.build_request(goals, subjects), which handles storing the initial subjects and generating roots.
  • Remove some of the magic around type_alias, and store it in a Struct whenever it exists.
  • Rename the object that is passed to Node.step to StepContext, and make the NodeBuilder concept private to the Scheduler.
  • Introduce a StatefulPool multiprocessing pool, which allows processes to hold/mutate their own copies of a value between items processed by the pool.
    • This is used here by LocalMultiprocessEngine to have workers with their own copy of the NodeBuilder, and their own Subjects collections. In some regards, this serves to make the LocalMultiprocessEngine even more of a toy, because a real implementation will likely use a local database or shared memory to store.
  • Pass Subjects introduced by Steps back to the Scheduler so that they can be stored in the Subjects instance.


Peiyu Wang
Stu Hood
Kris Wilson
Peiyu Wang
Stu Hood
Stu Hood
Peiyu Wang
Stu Hood
Review request changed

Status: Closed (submitted)

Change Summary:

Merged as 4998c61929a7bdd85d655e0a1702fbed8debf785