[engine] switch content addressable storage from dict to a embedded db

Review Request #3517 - Created March 1, 2016 and submitted

Peiyu Wang
kwlzn, patricklaw, stuhood

This review switches to use a lmdb db for our content addressable storage.

  • Moved SubjectKey and Subjects to storage module, they serve as the public interface as content addressable storage, the rest in storage are private.
  • Removed SubjectKey.len(), not applicable to the db kvs implementation.
  • Introduce Closable to handle db clean up, more convenient for the long running use case, clean up is done at the end of process. The context management protocol is also provided.
  • No change to StepContext which is still needed for nodes to access subjects, just a different implementation of subjects.
  • No change to StepRequest
  • Simplified StepResult because now there is no need to pass subjects back to the scheduler, as well as the code to handle that.
  • StatefulPool initial state is now bootstrapped via a function that acquires db handle for the process (as opposed to passing the db)

https://travis-ci.org/peiyuwang/pants/builds/112928418 passed
https://travis-ci.org/peiyuwang/pants/builds/113232665 passed
https://travis-ci.org/peiyuwang/pants/builds/114440248 passed


  • 0
  • 4
  • 1
  • 5
Description From Last Updated
Stu Hood
Peiyu Wang
Stu Hood
Timur Abishev
Kris Wilson
Peiyu Wang
Peiyu Wang
Review request changed

Status: Closed (submitted)

Change Summary:

Submitted as a8c1fb3e7eccdae0c4681b861d711a5fe81f9e49