[engine] Move storage out of scheduler to engine

Review Request #3554 - Created March 10, 2016 and submitted

Peiyu Wang
kwlzn, patricklaw, stuhood

Storage is part of the execution so it belongs to Engine. In Scheduler now it sees only subject_keys and state_keys (Except for one special case: Noop introduced for cyclic deps). Translations (both from key to subjects/states and the other way around) happen during step call.

This review

  • Move Storage out of Scheduler into Engine, and rename from subjects to storage, since it also stores States.
  • Add serializable's type to Key, this allows 1) validate deserialized object matches its retrieval key's type 2) scheduler needs State's type, this comes in handy
  • Move subject_key to subject look up from scheduler to Step.__call__
  • Add state_key to state look up to Step.__call__
  • Add state to state_key conversion also in Step.__call__
  • Change StepResult.state to StepResult.(state_key|dependencies), just the information needed by scheduler, given type is encoded in the key

Encounted two pickling errors:

  • Need to define __getstate__ and __setstate__ sicne we defined __slot__
  • ConflictingProducersError not pickable due to python bug (TypeError: ('__init__() takes exactly 4 arguments (2 given)')

https://travis-ci.org/peiyuwang/pants/builds/114984044 passed
https://travis-ci.org/peiyuwang/pants/builds/115212183 passed


  • 0
  • 1
  • 0
  • 1
Description From Last Updated
Stu Hood
Kris Wilson
Peiyu Wang
Peiyu Wang
Peiyu Wang
Review request changed

Status: Closed (submitted)