Split Ivy Resolve into Resolve / Fetch steps

Review Request #3555 - Created March 10, 2016 and submitted

Nick Howard (Twitter)
benjyw, gmalmquist, jsirois, patricklaw, stuhood

This patch contains the first step in the changes to resolution that I've been working on. It breaks up the internals of the IvyTaskMixin into two steps so that we can cache the coordinates of transitive external dependencies and fetch them intransitively during future resolves.

How it works:
When Pants does a ivy resolve, it now generates a json file containing the coordinates of the resolved artifacts and their relationships to the targets that participated in the resolve. This file is cached. If a subsequent resolve successfully hits the cache and gets the json file, instead of doing a full resolve, it instead does a simpler, flat, intransitive resolve using the coordinate information from the full resolve.

  • Resolved versions can't move based on changed dependency information in remote metadata.
  • Flat, intransitive resolves are faster because Ivy no longer has to do version conflict resolution.

In IvyTaskMixin, now instead of having a single fast path, slow path, we have four paths that fall through to the next if their checks fail.

  • Try loading an existing fetch result.
  • Try loading an existing resolve result.
  • Try fetch and load the result.
  • Do a resolve and load the result.

Major Components:

In ivy_task_mixin:

  • Return IvyResolveResult from IvyTaskMixin.resolve

In ivy_resolve:
- Use the result from IvyTaskMixin.resolve to find the report files to generate reports from.
- ResolutionStep, IvyResolutionStep, IvyFetchStep and IvyResolveStep: Encapsulate the load and resolve paths of the two ways of resolving external artifacts using ivy.
- FrozenResolution: Represents the resolved coordinates of all transitive external artifacts and their relationships to targets
- IvyFetchResolveResult: subclass of IvyResolveResult that overrides how target to jar relationships are determined to use FrozenResolutions.

Minor adjustments:

  • Make JarDependencyManager injectable for ivy.xml generation to make testing easier.
  • Move remaining static ivy lock from IvyTaskMixin to IvyUtils
  • Deprecate IvyUtils methods that do things that are specialized by IvyTaskMixin in favor of methods that have larger granularity.

Follow on work:

  • Represent the external artifact dependency graph in the cachable resolution information. https://github.com/pantsbuild/pants/issues/3052
  • Use Pants' buildcache to store the external artifacts that are resolved in a full resolve, falling back to a fetch resolve if the cache is unavailable. https://github.com/pantsbuild/pants/issues/3053
  • Implement the fetch resolve in Python rather than using Ivy. https://github.com/pantsbuild/pants/issues/3054
  • Isolate resolutions for leaf targets. https://github.com/pantsbuild/pants/issues/3055

Design Doc for 3rdparty changes https://docs.google.com/document/d/1RE4Kuf9f2g-ZU2nHKozRZ3P6oUwAEYaREBXLcn83C9w/edit

I added a number of unit tests around ivy resolve behavior around fetch vs full resolves. I've also done some testing within our repo.

CI passed at https://travis-ci.org/pantsbuild/pants/builds/116752107


  • 0
  • 7
  • 3
  • 10
Description From Last Updated
Nick Howard (Twitter)
Garrett Malmquist
Benjy Weinberger
Stu Hood
Nick Howard (Twitter)
Stu Hood
Nick Howard (Twitter)
Stu Hood
Benjy Weinberger
Benjy Weinberger
Benjy Weinberger
Nick Howard (Twitter)
Nick Howard (Twitter)
Nick Howard (Twitter)
Review request changed

Status: Closed (submitted)

Change Summary: