Fix BootstrapJvmTools to plumb jvm options to Shader.

Review Request #2073 - Created April 13, 2015 and submitted

Information
John Sirois
pants
jsirois/shade/cap_mem
1417
9b32378...
Reviewers
pants-reviews
benjyw, dturner-tw, zundel
This was the root cause of Shader getting killed in TravisCI containers.
In those containers, 4g of memory is available but the jvm detects the
full host ram and so defaults to 16g as the default XmX. With no
jvm_options being plumbed to the Shader (they were already to the
underlying IvyUtils and JarTask) - it happily bombed past the container
memory and was OOMkilled.

 src/python/pants/backend/jvm/tasks/bootstrap_jvm_tools.py | 16 ++++++++++++++--
 src/python/pants/java/jar/shader.py                       |  4 +++-
 2 files changed, 17 insertions(+), 3 deletions(-)

I double-checked in the ./pants server and the Shader spawn was the
only one of 5 jvm tasks (3 resolves, 1 fat-jar creation, 1 shade) to not
have -Xmx.

I rigged a special ci run and found the default MaxHeapSize was above the
container limit:

$ uname -a
java -XX:+PrintFlagsFinal -version | grep HeapSize
./build-support/bin/ci.sh -x ${CI_FLAGS}
Linux testing-worker-linux-docker-29022e06-3351-linux-15 3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 17:53:56 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
    uintx ErgoHeapSizeLimit                         = 0               {product}           
    uintx HeapSizePerGCThread                       = 87241520        {product}           
    uintx InitialHeapSize                          := 989377216       {product}           
    uintx LargePageHeapSizeThreshold                = 134217728       {product}           
    uintx MaxHeapSize                              := 15831400448     {product}           
java version "1.7.0_76"
Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
[== 00:00 CI BEGINS: 'Unit tests for pants and pants-plugins' ==]
...

CI away here:
https://travis-ci.org/pantsbuild/pants/builds/58283618

Benjy Weinberger
John Sirois
John Sirois
Review request changed

Status: Closed (submitted)

Eric Ayers

   

I was going to say that I had been hoping to use subsystem for these cases, we've already got the config file section [jar-tool] and 'jvm-args' if you want to make this more targeted.

https://github.com/pantsbuild/pants/blob/dcb2de22f72596ab848c458d4adafb7471e4a3a5/src/python/pants/backend/jvm/tasks/jar_task.py#L289

In our build we already explicitly tune -Xmx for jar tool. I'm not sure which JVM option will win after this change.

  1. I'm not sure I fully understand your comment/question/suggestion, but:

    In our build we already explicitly tune -Xmx for jar tool. I'm not sure which JVM option will win after this change.
    The [jar-tool]jvm_args option still controls the the jarring done here by the bootstrapper's nailgun (by default)

    The new registered jvm_options are still used by the IvyTaskMixin, they just have a different scope for setting. Although in the pantsbuild/pants pants.ini the setting comes from the DEFAULT section so the end result is no-change.

  2. I don't think there is a problem now, but I don't think this solves the problem the way we want in general. These nailgun tasks probably need to read jvm-options from a Subsystem specific scope, not the one it inherits from the parent task.

    To override the heap setting in jar-tool, I set:

    [jar-tool]
    jvm_args = ["-Xmx300M"]
    

    If I were to just use jvm_options from the parent task, I'm not sure which -Xmx setting I'd end up with. I have the heap sized any where from 256M to 1G for the other tasks that launch jar-tool. Plus, If the shader was launched from multiple tasks, wouldn't the nailgun executor detect different cmdline arguments to the task and constantly shut down the tool and restart it?

  3. but I don't think this solves the problem the way we want in general

    "this" which? "this" as in the current code that is well commented as not ideal or "this" as in where the comment points to the future, using a subsystem for each of Ivy, Jar, Shader and letting the Task pick global or task scope?

    I have a feeling you're preaching to the choir of my comment but I'm not sure.

  4. NM, I am confusing BootstrapJvmTools with NailgunTaskBase, where I toyed with registering --jvm_options a few weeks ago.

David Turner
Ship It!
John Sirois

Although this fix was needed, the BootstrapJvmToolsTest doesn't benefit. Although its JvmToolTaskTestBase base attempts to get real pants.ini options loaded this no longer happens.

I'll have another RB shortly to get -Xmx plumbed in the test to actually really finally squash this issue.

  1. The fix is here: https://rbcommons.com/s/twitter/r/2077/
Loading...