[engine] Content address node and state only in engine

Review Request #3615 - Created March 25, 2016 and submitted

Information
Peiyu Wang
pants
3066, 3097
Reviewers
pants-reviews
kwlzn, patricklaw, stuhood

This review conslidates node/state to key and key back to node/state into Storage, all called by engine.

This review alone does not improve performance (see [1]) but it will allow better control by engine what requests are to be CA-ed. By placing the CA logic in one place also cleaned up the interface, which might address some of https://github.com/pantsbuild/pants/issues/3070.

  • No more storage in Scheduler, all it sees are states, nodes, no more keys.
  • Step.__call__ no longer dealing with keys
  • Turning nodes and states into keys and the other way only happens in one code base in Storage, and is only called by Engine
  • Also killed state_key, the 'keyed' version of nodes or states is only for either computing cache keys or multi-process

[1] Manual performance test: cold cache, 30% faster (1.42 sec vs 1.15 sec), warmed-up cache, 64% slower (0.46 sec vs 0.75 sec) is because now we compute keys more, for nodes. Only converting nodes into keys guarantees stable ordering without making nodes support ordering. A lot of keying are repeated so expect https://rbcommons.com/s/twitter/r/3597/ would help.

https://travis-ci.org/peiyuwang/pants/builds/118573674 passed.
https://travis-ci.org/peiyuwang/pants/builds/118612254 passed.
https://travis-ci.org/peiyuwang/pants/builds/118633765 passed.
https://travis-ci.org/peiyuwang/pants/builds/118885399 passed.
https://travis-ci.org/peiyuwang/pants/builds/118898889 passed.

Issues

  • 0
  • 1
  • 0
  • 1
Description From Last Updated
Peiyu Wang
Stu Hood
Peiyu Wang
Peiyu Wang
Stu Hood
Peiyu Wang
Peiyu Wang
Review request changed

Status: Closed (submitted)

Change Summary:

Submitted as 1a1ffdcde387cd87b950db1723b1fdcf63875583

Loading...