Reduce number of test shards from 8 to 6 on Travis-ci

Review Request #1804 — Created Feb. 20, 2015 and submitted

zundel
pants
zundel/reduce-travis-shards
1141
a312b30...
pants-reviews
jsirois

Watching the builds late at night, our project never gets more than 5 shards. This means that there is some
tradeoff where fewer longer running shards will actually outperform more shorter ones.

We were up to 12 total shards and the timeline went like this:

*--------->
  ... first 5 shards ...
  *---------->
           *-------->
              ... next 5 shards ...
              *-------->
                     *------->
                       ... last 2 shards ...
                       *------->

I deduced this from watching: https://travis-ci.org/pantsbuild/pants/builds/51516004 and others late at night or early in the morning. In one example

CI is running at: https://travis-ci.org/pantsbuild/pants/builds/51517276

ZU
  1. Not sure if this is actually better or not. On this run it seems like we got 4 shards at a time !?!?

    1. Yeah - even assuming this had run in 2 waves, the long pole in each sums to >19 minutes which is ballpark wall clock time for the 3 wave 12 shard setup from my informal off-hours observations.
  2. 
      
JS
  1. As you see fit - I don't see harm in experimenting and changing the sharding often.
    1. If the algorithm for allocating shards is (# of shards requested /2) - 2, I think I know how to speed this up...

    2. Its definitely not that.  For 12 shards on off hours the shards came in waves of 5.  I really have no clue what their algorithm is.
    3. Sorry - missed the -2 and maybe the joke.
  2. 
      
ZU
Review request changed

Status: Closed (submitted)

Change Summary:

Thanks John, Let's just try this and see. If you want to change it back I'll rubber stamp the change. Commit 51be4e8

Loading...