[engine] Isolated Process Execution - First Cut

Review Request #4029 - Created June 28, 2016 and submitted

Nick Howard (Twitter)
2978, 3609
benjyw, jsirois, kwlzn, peiyu, stuhood

This review is for the first cut of isolated process execution in the new engine. It contains the beginnings of snapshotting, isolated workspace preparation and execution. It's not a finished product, but it's a point from which we can start iterating. It includes the beginnings of a process execution model for the engine.

Process Execution:

This first cut is more complicated than it needs to be, but I intend to refine it and make it more transparent to the engine.

How it works:

You define a process execution rule using the SnapshottedProcess rule type. It maybe that this can be decomposed into its components with the user facing pieces broken up a more fine grained way, but again, this is a first cut.

Here's an example of what that looks like from one of the tests.

                     input_selectors=(Select(Files), SelectLiteral(JavaOutputDir('build'), JavaOutputDir)),

The first parameter is the product type produced by running the process. This maps exactly to the product type parameter for the existing task concept.

The input_* parameters communicate what products are needed to run, and how to translate them into a SnapshottedProcessRequest. They are essentially a subclause that's a task rule. The output_conversion is called with the result of running the process and the Checkout which allows it to construct the product.

Internally what happens is that if a SnapshottedProcess task is selected, it creates a ProcessExecutionNode providing a place where before and after steps can be managed.

The execution node
- Creates a task node for creating the process request.
- Creates a checkout directory and dumps snapshots into it based on the contents of the process request.
- Calls the prep_fn on the request with the checkout. I want to get rid of this and come up with a better scheme for preparing for execution for things that are like build output directories, but for this review I punted via this escape hatch.
- Once the checkout is prepared, the process is actually executed using a ProcessExecutionNode.
- Finally, the output is converted into it's final form.

Future steps
- I'm not happy with naming, but for this review, I'm focusing more on a rough structure and I'm planning to iterate on it.
- Cleaning up checkouts. I left cleaning up temp directories as a future step.
- Configuration. Need to store those tar files somewhere. For now, I just create temp directories.
- Documentation. I think this needs to wait on naming and maturing the user facing representation.
- Integration with the planner example.
- Nailgun.

I've got a number of high level tests I've been using to drive development in test_isolated_process.

I've attached a PR with CI running.


  • 0
  • 10
  • 1
  • 11
Description From Last Updated
Stu Hood
Stu Hood
Nick Howard (Twitter)
Stu Hood
Nick Howard (Twitter)
Review request changed

Status: Closed (submitted)

Change Summary:

Submitted as https://github.com/pantsbuild/pants/commit/68b5a6c91ea661f3a6c2fca94b2b9d0234a562c6