Make monolithic jar slimmer

Review Request #3133 - Created Nov. 16, 2015 and submitted

Information
Peiyu Wang
pants
peiyuwang:peiyu/make-fat-jar-optional2
2577
73411e9...
Reviewers
pants-reviews
fkorotkov, gmalmquist, jsirois, patricklaw, stuhood, zundel

Monolithic jar does not scale, even with --no-deployjar that excludes external
jars, internal dependencies can still be large. In Twitter we saw a few hundred
Mbytes jars to transfer, for remote offices this is more painful. Breaking into
smaller jars can also benefit hadoop folks to better utilize their caching layer [1]

This review makes --no-deployjar no longer bundle internal libraries, instead,
make them symlinks under libs like we do for 3rdparty, the links will go to those
jars under .pants.d, symlinks will have stable names, those stable names are used
as archive contents, they are also to be embeded as the Manifest's Class-Path
attribute.

Current --no-deployjar

app-bundle.jar // this can be large
libs/
libs/3rdparty-x.jar
libs/3rdparty-others.jar

New --no-deployjar

app-bundle.jar // this only sets Class-Path + other Manifest entries
libs/
libs/3rdparty-x.jar
libs/3rdparty-other.jar
libs/internal-library-a.jar // symlinks to .pants.d, unless it's shaded
libs/internal-library-other.jar

---deployjar will remain as is.

[1] Hadoop use case is they use SharedCache distribute jars from HDFS. They have
a caching layer that does file checksums so if we break down into smaller jars
they will be able to transfer less.
http://www.slideshare.net/ctrezzo/hadoop-summit2015-yarnsharedcachev12

https://travis-ci.org/peiyuwang/pants/builds/91439042

Issues

  • 0
  • 2
  • 3
  • 5
Description From Last Updated
Peiyu Wang
Garrett Malmquist
Stu Hood
Eric Ayers
Patrick Lawson
Peiyu Wang
Peiyu Wang
Peiyu Wang
Peiyu Wang
Stu Hood
Peiyu Wang
Patrick Lawson
Peiyu Wang
Review request changed

Status: Closed (submitted)

Change Summary:

Merged as 6285e4f27a0fa35f219449c00395cf78de125a20

Loading...