Improved go remote library support.

Review Request #2655 - Created Aug. 18, 2015 and submitted

Information
John Sirois
pants
jsirois/issues/1998
1998, 2013
95aa740...
Reviewers
pants-reviews
fkorotkov, mateor, stuhood, zundel
GoFetch now supports remote libraries with nested packages as well as
implicit targets for transitive dependencies that fall within the same
remote import path root as an explicitly declared remote dependency.

The awkward fit of zip_url to the more generically named GoRemoteLibrary
is fixed by moving the task of fetching fully out to a Fetchers
subsystem that works by delegating based on matches against the remote
import path.  To support this, both GoRemoteLibrary and GoLocalSource
targets gain a consistent notion of `import_path` that matches the same
concept in go.

The existing zip download functionality for remote libraries on github
is extracted to a generic ArchiveFetcher implementation with a default
configuration that knows how to construct and github archive URLs and
unpack them.

Additionally, a 'go_remote_libraries' macro is introduced to allow for
depending on a suite of packages at a single revision without repeating
the version in the BUILD file.

 contrib/go/examples/3rdparty/go/github.com/AdRoll/goamz/BUILD                      |   5 +
 contrib/go/examples/3rdparty/go/github.com/bitly/go-simplejson/BUILD               |   4 +
 contrib/go/examples/3rdparty/go/github.com/cbroglie/mapstructure/BUILD             |   4 +
 contrib/go/examples/3rdparty/go/github.com/gorilla/context/BUILD                   |   3 +-
 contrib/go/examples/3rdparty/go/github.com/gorilla/mux/BUILD                       |   3 +-
 contrib/go/examples/src/go/hello/main.go                                           |   4 +-
 contrib/go/examples/src/go/libA/a.go                                               |   6 +-
 contrib/go/examples/src/go/libB/b.go                                               |   2 +-
 contrib/go/examples/src/go/libC/c.go                                               |   6 +-
 contrib/go/src/python/pants/contrib/go/register.py                                 |   3 +
 contrib/go/src/python/pants/contrib/go/subsystems/BUILD                            |  15 +++
 contrib/go/src/python/pants/contrib/go/subsystems/fetchers.py                      | 365 ++++++++++++++++++++++++++++++++++++++++++++++++++
 contrib/go/src/python/pants/contrib/go/subsystems/go_distribution.py               |   6 +-
 contrib/go/src/python/pants/contrib/go/targets/BUILD                               |   1 +
 contrib/go/src/python/pants/contrib/go/targets/go_local_source.py                  |   4 +
 contrib/go/src/python/pants/contrib/go/targets/go_remote_library.py                |  66 +++++----
 contrib/go/src/python/pants/contrib/go/tasks/BUILD                                 |  23 ++--
 contrib/go/src/python/pants/contrib/go/tasks/go_compile.py                         |  20 ++-
 contrib/go/src/python/pants/contrib/go/tasks/go_fetch.py                           | 281 ++++++++++++++++++++------------------
 contrib/go/src/python/pants/contrib/go/tasks/go_task.py                            |  17 ---
 contrib/go/src/python/pants/contrib/go/tasks/go_test.py                            |  13 +-
 contrib/go/src/python/pants/contrib/go/tasks/go_workspace_task.py                  |  45 ++++---
 contrib/go/tests/python/pants_test/contrib/go/tasks/BUILD                          |  40 +++---
 contrib/go/tests/python/pants_test/contrib/go/tasks/go_tool.py                     |  20 ---
 contrib/go/tests/python/pants_test/contrib/go/tasks/test_go_compile_integration.py |  15 +--
 contrib/go/tests/python/pants_test/contrib/go/tasks/test_go_fetch.py               | 133 ++++++++----------
 contrib/go/tests/python/pants_test/contrib/go/tasks/test_go_fetch_integration.py   |  15 +++
 contrib/go/tests/python/pants_test/contrib/go/tasks/test_go_workspace_task.py      |  38 +++---
 pants.ini                                                                          |   1 -
 src/python/pants/base/address.py                                                   |   2 +-
 src/python/pants/base/build_configuration.py                                       |  11 +-
 src/python/pants/fs/archive.py                                                     |  19 +++
 src/python/pants/goal/context.py                                                   |   8 +-
 33 files changed, 811 insertions(+), 387 deletions(-)

Tested this in Medium's mono-repo in a manual test case that's now
replicated by the GoFetchIntegrationTest.test_issues_1998 test.

Also some ad-hoc testing of the fetcher options like so (flip default
github .tar.gz fetch to .zip):

$ ./pants \
  --fetchers-mapping="{r'github.com/':'ArchiveFetcher'}" \
  --archive-fetcher-matchers="{r'github.com/(?P<user>[^/]+)/(?P<repo>[^/]+)':('https://github.com/\g<user>/\g<repo>/archive/{rev}.zip', 'master', 1)}" \
    compile contrib/go/examples/3rdparty/go/github.com/AdRoll/goamz:dynamodb
...
13:24:38 00:00   [resolve]
13:24:38 00:00     [ivy]
13:24:38 00:00       [cache].
13:24:38 00:00     [go]
                   Invalidated 1 target.INFO] Downloading https://github.com/AdRoll/goamz/archive/51b70b0035cf28c4ca6d719156347ac56a41d922.zip...

                   Invalidated 3 targets.INFO] Downloading https://github.com/bitly/go-simplejson/archive/v0.4.3.zip...

                   Invalidated 1 target.INFO] Downloading https://github.com/cbroglie/mapstructure/archive/25325b46b67d1c3eb7d58bad37d34d89a31cf9ec.zip...
...
$ $ tree .pants.d/compile/go/contrib.go.examples.3rdparty.go.github.com.AdRoll.goamz.dynamodb/pkg/linux_amd64/github.com/AdRoll/goamz/
.pants.d/compile/go/contrib.go.examples.3rdparty.go.github.com.AdRoll.goamz.dynamodb/pkg/linux_amd64/github.com/AdRoll/goamz/
├── aws.a -> /home/jsirois/dev/3rdparty/jsirois-pants4/.pants.d/compile/go/contrib.go.examples.3rdparty.go.github.com.AdRoll.goamz.aws/pkg/linux_amd64/github.com/AdRoll/goamz/aws.a
├── dynamodb
│   └── dynamizer.a -> /home/jsirois/dev/3rdparty/jsirois-pants4/.pants.d/compile/go/contrib.go.examples.3rdparty.go.github.com.AdRoll.goamz.dynamodb.dynamizer/pkg/linux_amd64/github.com/AdRoll/goamz/dynamodb/dynamizer.a
└── dynamodb.a

1 directory, 3 files

CI went green here:
https://travis-ci.org/pantsbuild/pants/builds/76318249

John Sirois
John Sirois
John Sirois
John Sirois
Cody Gibb
Mateo Rodriguez
John Sirois
John Sirois
John Sirois
John Sirois
Cody Gibb
John Sirois
John Sirois
John Sirois
Review request changed

Status: Closed (submitted)

Loading...