[engine] cache StepResult under StepRequest

Review Request #3494 - Created Feb. 24, 2016 and submitted

Information
Peiyu Wang
pants
3545
Reviewers
pants-reviews
kwlzn, patricklaw, stuhood

This review adds:

  • Cache that caches StepResult for StepRequest by storing the blobs in the main storage like the rest subjects and states, in addition, the request key to result key mappings are stored in a separate lookup table. Therefore two lookups are needed for cache hits.
  • Lmdb support for creating child database, for secondary index use case.
  • Add add_mapping, get_mapping interface to Storage, Cache can choose to have a different location of the subjects storage, or can be the same, but implementation is shared, this uniformation allows for sharing future GC logic for example.
  • Hooks in various execution environment to check cache and save to cache: LocalSerialEngine, LocalMultiprocessEngine, _stateful_pool_loop.
  • Use OrderedDict for Struct.kwargs to ensure consistent hashing.

Notes:

  • Now since the main blobs states and subjects are already in keys, both StepRequest and StepResult are light, so we can afford to do this inside engine before step calls, essentially doing cache checks earliest possible.

TODO:

  • There is some weirdness from tuples, that even for the same tuple for example, ((u'thrift', u'apache_java'),) may produce different hash. I tried if flatten the tuple pickle will be produce consistent result. Still need some work.

https://travis-ci.org/peiyuwang/pants/builds/114562555
https://travis-ci.org/peiyuwang/pants/builds/116460440

Issues

  • 0
  • 7
  • 0
  • 7
Description From Last Updated
Peiyu Wang
Peiyu Wang
Stu Hood
Stu Hood
Peiyu Wang
Yi Cheng
Stu Hood
Peiyu Wang
Stu Hood
Peiyu Wang
Kris Wilson
Peiyu Wang
Peiyu Wang
Review request changed

Status: Closed (submitted)

Change Summary:

submitted as 73642848697b5ee364309bb046f17032c0d97201

Loading...