[engine] Introduce ProductGraph invalidation.
Review Request #3578 - Created March 17, 2016 and submitted
- Initial pass at ProductGraph invalidation using a predicate-based matching approach as the assumed coupling.
- Add a
ProductGraph.walkfor reusability in backwards (
dependents_of()) graph walks for invalidation.
- Add basic re-entrant locking of
ProductGraphinstance method calls using
ProductGraph.clear()in favor of
- Test coverage.
This synchronization is pretty fine grained... in particular, I'm really curious what the effects of invalidating in the middle of a build would be, and so I'd really kind of prefer that we go much coarser to start.
What about having a Scheduler-level lock and acquiring it for
def invalidateor for
def schedule? That would make the ProductGraph concurrency-oblivious, and the scheduler concurrency aware, which feels more manageable to me.
Doesn't seem like either the enumerate or reversed is necessary here... could return
Rather than doing this check inside the loop, how about selecting the list to use at the beginning of the method?
adjacencies = self._dependents if dependents else self._dependents ... .. = _filtered_entries(adjacencies[node])
So, might it make sense for the purposes of testing to have ProductGraph take a "validation" function as a constructor parameter? I think that that would allow ProductGraph to be decoupled-from/moved-out-of scheduler.py (at some point), and would make it possible to write tests with string keys, for example:
pg = ProductGraph(validator=lambda _: pass) pg.update_state('A', Waiting(['B', 'C'])) self.assertEquals(set('B', 'C'), pg.dependencies_of('A'))
Address Stu's feedback and pull+rebase master.
Revision 2 (+241 -78)
Address Peiyu's feedback.
Revision 3 (+246 -79)