Prepare for a global --shard flag.

Review Request #3560 — Created March 10, 2016 and submitted

benjyw
pants
pants-reviews
patricklaw, stuhood, zundel

- Refactor shard spec parsing and application to a common utility.

- Rename test.pytest's --shard to --test-shard (and deprecate the
old name) so it won't collide with the global name, and for
uniformity with test.junit's similar flag.

Note that the name --test-shard is slightly awkward, because
'test' is the name of a scope, so using the short form in the
'test.pytest' context won't work. But I stuck with that name
for uniformity with test.junit, and because this is a wider
problem we need to think about anyway.

When global sharding is implemented, the test tasks' sharding will
subshard on top of that. This is useful because sharding the input
targets may not be fine-grained enough (some repos have large numbers
of tests covered by a single target). However global sharding
may be good enough in many cases, and will prevent us from having
to implement sharding over and over again in every task that might
need it.

Note: Replaces https://rbcommons.com/s/twitter/r/3363/, which is now
discarded.

CI passes: https://travis-ci.org/pantsbuild/pants/builds/115140711

  • 0
  • 0
  • 1
  • 0
  • 1
Description From Last Updated
  1. I think this is a great alternative to the previous patch.

    I was surprised there was no code to refactor from junit_run but it looks like all that shard parsing logic is in java (ConsolerRunnerImpl.java)

  2. src/python/pants/base/hash_utils.py (Diff revision 1)
     
     

    In pants, spec usually refers to the string representation of a target and I was a bit confused by the name. Here it means a string that matches [\d]+/[\d]+.

    Could you document the type and exepcted format of 'spec' in pydoc here? Maybe its obvious to other folks, but I was looking around for a type that defined partition() but then it dawned on me that this was a method on string.

    1. Documented, and changed name to 'shard_spec" throughout.

  3. 
      
Review request changed

Status: Closed (submitted)

Change Summary:

d323c65240afc7a47c7c42fdcadb57d70fab1157

  1. Thanks Eric! Submitted as d323c65240afc7a47c7c42fdcadb57d70fab1157.

  2. 
      
Loading...