[engine] fix unpickling error in multiprocessing engine tests
Review Request #3751 — Created April 24, 2016 and discarded
|jsirois, kwlzn, patricklaw, stuhood|
Not clear what the root cause of the
UnpicklingErroris, but switching
StringIOseems to fix the flakiness, have run many tests
to confirm, at the cost of significant performance impact.
A few things we know about the failure:
- Those unpickable data appear to be old space has been GC-ed  for sth else.
- Failures only were seen in travis, not on personal mac.
- Switching to use in-memory db instead of
lmdbmakes no difference.
picklealso makes no difference.
StringIOhaven't seen any failures.
Did some search in python bugs with no avail. Our cStringIO usage looks alright. 
Tried to read the cStringIO source , but didn't spend too much time.
The reason it hits travis my guess could be due to 1) multiprocessing has
higher memory overhead, 2) we use #cpu to allocate multiprocessing pool size,
3) travis has more cpus compared to local mac.
This does impact performance significantly, about +50% for the cold cache,
and +100% for the warmed up cache in the list test I did.
 https://github.com/pantsbuild/pants/issues/3149 with examples of the
hexlified and unhexlified unpickable data.
blob is what's reallocated.
 cStringIO https://github.com/python/cpython/blob/2.7/Modules/cStringIO.c
- Test pickle/stringIO combinations:
(with test has a loop for running test 5 times)
cStringIO, cPickle: job 457 to 467 all failed except for 1
cStringIO, pickle: job 469 to 479 all failed
StringIO, cPickle: job 480 to 490 all passed
StringIO, pickle: job 491 to 501 all passed
- This patch vs master:
with this patch: job 502 to 512, 22/22 (restarted once)
without this patch: job 513 to 523, 3/22 failed