[engine] Add native filesystem node type.

Review Request #3550 — Created March 9, 2016 and submitted

kwlzn
pants
kwlzn/engine/native_fs
3025
pants-reviews
peiyu, stuhood
  • Implement FilesystemNode for native filesystem operations to pave the way for invalidation.
  • Plumb ProjectTree instance access to StepContext.
  • Extract create_fs_tasks, create_graph_tasks and create_legacy_graph_tasks into pants.engine.exp.register.
  • Mark test_scheduler.py:test_dependency_inference xfail pending #3024.
  • Misc cleanups and a quick fix to retain .dot files generated by the visualizer for direct analysis.

N.B. this currently omits a prior short-circuiting return in StepContext.task_nodes (which seems to break globs due to the use of Paths) in favor of a more permissive approach for the time being.

https://travis-ci.org/pantsbuild/pants/builds/114725318

  • 0
  • 0
  • 2
  • 0
  • 2
Description From Last Updated
PE
  1. 
      
  2. src/python/pants/engine/exp/nodes.py (Diff revision 1)
     
     
     

    move this to node_builder?

  3. just to comment on this is related to what i am working on moving storage into engine from scheduler: so now engine will have three things to translate: subjects, project_tree, and state

  4. 
      
ST
  1. 
      
  2. As mentioned above, I don't think it goes into Storage.

  3. src/python/pants/engine/exp/nodes.py (Diff revision 1)
     
     
     
     
     
     

    Given that there is an accessor method, can probably make this private.

  4. Rather than taking the key, this should just take the instance, no?

    In the multi-process case, I think the assumption is that each node must have access to the relevant git checkout (or to scoot), and so each of them can have a clone of the ProjectTree.

    Thinking about this more, it seems like the ProjectTree is probably a property of the Engine (ie, the execution environment) rather than the scheduler.

    1. sure - makes sense. fixed and will punt the move of ProjectTree from scheduler->engine for subsequent iteration as discussed.

  5. 
      
KW
ST
  1. 
      
  2. src/python/pants/engine/exp/BUILD (Diff revision 2)
     
     
     

    xx: see below.

  3. src/python/pants/engine/exp/nodes.py (Diff revision 2)
     
     

    Couldn't

    and can drop the "in a FilesystemNode" bit: the type of the Node should always be available.

  4. src/python/pants/engine/exp/register.py (Diff revision 2)
     
     
     
     
     
     
     
     
     
     
     

    These are definitely specific to the legacy subdirectory. I can understand moving things here temporarily in order to break cycles directly in exp, but I'm pretty sure these shouldn't be causing a cycle.

    1. yeah, this one specifically isn't causing a cycle but thought it made sense to consolidate - moved this to a legacy-specific register.py to keep the legacy line distinct.

  5. Maybe rather than is_native, this is is_filesystem_product? There will be other native nodes soonish.

  6. 
      
KW
KW
PE
  1. Ship It!
  2. 
      
KW
Review request changed

Status: Closed (submitted)

Change Summary:

thx gents! submitted @ a56f11bc822f45997c9af154abb1ce512f93888e

Loading...