[engine] turn off pickle memoization

Review Request #3574 — Created March 16, 2016 and submitted

kwlzn, patricklaw, stuhood

I ran into the same issue in https://rbcommons.com/s/twitter/r/3494/ that Stu had earlier in https://rbcommons.com/s/twitter/r/3498

It's due to by default pickle memorizes what has been pickled by ids, which means same object
is not going to be pickled again, (A, A) has less space than (A, A'), A equals A'. For us,
content addressability is based on equality. We need (A, A) to be identical to (A, A').

Switch to fast mode turns off memorization. Long term we still need
https://github.com/pantsbuild/pants/issues/2969. Fast is undocumented in 2 and deprecated
in 3 https://docs.python.org/3/library/pickle.html (there might be a equivalent way in 3)

Ran ./pants test tests/python/pants_test/engine/exp/::
Started https://travis-ci.org/peiyuwang/pants/builds/116415180

  1. Nice find! Please fix the spelling everywhere though: this is "memoization"

  2. src/python/pants/engine/exp/storage.py (Diff revision 1)


Review request changed

Status: Closed (submitted)

Change Summary:

submitted as e6eac05f83be71ece5d89b7630c7cc02d95bdf94