Don't add internal jars to the classpath in a bundle jar.

Review Request #218 — Created April 15, 2014 and submitted

benjyw
pants
pants-reviews
jsirois
Those classes (and resources) are already in the bundle jar itself.

It's actually harmful to add those jars to the classpath, due to
the following OS X issue:

The classloader keeps an open handle to each jar on the classpath.
So if you have N targets then you consume N file descriptors. And
we have over 1000 targets in some binaries.

However Java on OS X cannot stat sockets with fds > 1024. This is
probably a bug, but we're stuck with it. So we end up in the
weird situation of the size of our classpath preventing us from
using network sockets...

PANTS_DEV=1 ./pants goal clean-all
PANTS_DEV=1 ./pants goal bundle src/java/com/pants/examples/pingpong/main

Inspect classpath using unzip -q -c dist/pingpong-bundle/pingpong.jar META-INF/MANIFEST.MF

Verify that this runs without error: java -cp dist/pingpong-bundle/ dist/pingpong-bundle/pingpong.jar
  • 2
  • 0
  • 0
  • 0
  • 2
Description From Last Updated
These are real classpath elements that are now gone for non self.deployjar bundles - sorely this breaks those bundles. JS jsirois
Can you explain further? As far as I can tell: If self.deployjar==False: 1. jar_create.py creates one jar per target. 2. ... BE benjyw
SL
  1. Ship It!
  2. 
      
SL
  1. We believe this is the related OS X bug we are running into:
    http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8035897
  2. 
      
JS
  1. 
      
  2. src/python/pants/tasks/bundle_create.py (Diff revision 1)
     
     
    These are real classpath elements that are now gone for non self.deployjar bundles - sorely this breaks those bundles.
  3. 
      
BE
  1. 
      
  2. src/python/pants/tasks/bundle_create.py (Diff revision 1)
     
     
    Can you explain further? As far as I can tell:
    
    If self.deployjar==False:
    
    1. jar_create.py creates one jar per target.
    2. binary_create.py creates a jar containing the union of all the jars in 1.
    3. bundle_create.py takes that jar, adds 3rdparty jars (and, prior to my change, internal jars), and sets the classpath in the manifest. 
    
    
    If self.deployjar==True then:
    
    1. jar_create.py creates one jar per target.
    2. binary_create.py creates a jar containing the union of all the jars in 1. and all the third-party jars.
    3. bundle_create.py just symlinks to the jar from 2. since that jar is completely self-contained.
    
    In either case I'm not sure what classpath elements would be missing after my change. Can you give an example?
    
  3. 
      
JS
  1. 
      
  2. src/python/pants/tasks/bundle_create.py (Diff revision 1)
     
     
    Aha - your Item 2 in the 'self.deployjar==False' section is correct.  So this was cruft and the removal kills dup classpath elements repeated in the main binary jar.  Internally we had gone away from this form of packaging and I was confused.
  3. src/python/pants/tasks/bundle_create.py (Diff revision 1)
     
     
    Well, in general the idea is to not mutate products - who know what else might be downstream that needs this stuff too.
    1. Right, will change to "a copy of the existing jar". I just expect that would be more efficient for large jars, but I could be totally wrong.
  4. 
      
BE
Review request changed

Status: Closed (submitted)

Loading...