[pantsd] Initial implementation of the pants nailgun service.

Review Request #3171 - Created Nov. 24, 2015 and submitted

Information
Kris Wilson
pants
kwlzn/pantsd/pailgun
2617
Reviewers
pants-reviews
benjyw, jsirois, nhoward_tw, stuhood, zundel

This is a python/pants nailgun server implementation that is intended to speed up the startup and runtime of pants by way of keeping warm state in a running daemon and using a client/server fork-on-run model. This utilizes the Nailgun protocol - a generic client/server protocol for running commands and marshalling stdio back/forth - to accomplish this; i.e. it effectively brings the same nailgun mechanics we use to execute code in a resident JVM except against a resident python interpreter with the primary intent of running pants. The current implementation is presented without any intentional state caching optimization (clean runs) and works as-is with the native nailgun client for an already order-of-magnitude speedup of no-op and help-advanced runtime (see testing section below). Two subsequent components are expected to arrive shortly to fully wire this up: the Pants Daemon itself and a closed-loop client implementation that will utilize this service.

  • Initial implementations of DaemonPantsRunner, PailgunService, PailgunServer et al - "Pailgun" is used to disambiguate between the canonical jvm-oriented "nailgun" usage in pants vs the python-oriented implementation.
  • Break-out a new pants.java.nailgun_io module with NailgunStream{Reader,Writer} for general use.
  • Tests!

1) manual E2E testing in the pantsd dev branch.

2) CI green @ https://travis-ci.org/pantsbuild/pants/builds/92886226

3)
[illuminati pants (kwlzn/pantsd/pailgun)]$ ptcm
00:37:48 00:00 [main]
00:37:48 00:00 [setup]
00:37:48 00:00 [parse]INFO] Operating on changed 43 target(s):
* src/python/pants/bin
* src/python/pants/java:executor
* src/python/pants/java:nailgun_client
* src/python/pants/java:nailgun_executor
* src/python/pants/java:nailgun_io
* src/python/pants/java:nailgun_protocol
* src/python/pants/java:util
* src/python/pants/pantsd/service:fs_event_service
* src/python/pants/pantsd/service:pailgun_service
* src/python/pants/pantsd/service:pants_service
* src/python/pants/pantsd:pailgun_server
* src/python/pants/pantsd:process_manager
* src/python/pants/pantsd:watchman
* src/python/pants/pantsd:watchman_client
* src/python/pants/util:contextutil
* tests/python/pants_test/java
* tests/python/pants_test/java:executor
* tests/python/pants_test/java:nailgun_client
* tests/python/pants_test/java:nailgun_integration
* tests/python/pants_test/java:nailgun_io
* tests/python/pants_test/java:nailgun_protocol
* tests/python/pants_test/java:util
* tests/python/pants_test/pantsd/service:fs_event_service
* tests/python/pants_test/pantsd/service:pailgun_service
* tests/python/pants_test/pantsd/service:pants_service
* tests/python/pants_test/pantsd:pailgun_server
* tests/python/pants_test/pantsd:process_manager
* tests/python/pants_test/pantsd:test_deps
* tests/python/pants_test/pantsd:watchman
* tests/python/pants_test/pantsd:watchman_client
* tests/python/pants_test/util
* tests/python/pants_test/util:contextutil
* tests/python/pants_test/util:dirutil
* tests/python/pants_test/util:eval
* tests/python/pants_test/util:fileutil
* tests/python/pants_test/util:filtering
* tests/python/pants_test/util:memo
* tests/python/pants_test/util:meta
* tests/python/pants_test/util:osutil
* tests/python/pants_test/util:socket
* tests/python/pants_test/util:strutil
* tests/python/pants_test/util:timeout
* tests/python/pants_test/util:xml_parser

               Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> jvm-platform-validate -> gen -> resolve -> resources -> compile -> test -> test-changed
00:37:48 00:00   [bootstrap]
00:37:48 00:00     [bootstrap-jvm-tools]
00:37:48 00:00   [imports]
00:37:48 00:00     [ivy-imports]
00:37:48 00:00   [unpack-jars]
00:37:48 00:00     [unpack-jars]
00:37:48 00:00     [unpack-libs]
00:37:48 00:00   [deferred-sources]
00:37:48 00:00     [deferred-sources]
00:37:48 00:00   [jvm-platform-validate]
00:37:48 00:00     [jvm-platform-validate]
00:37:48 00:00   [gen]
00:37:48 00:00     [thrift]
00:37:48 00:00     [protoc]
00:37:48 00:00     [antlr]
00:37:48 00:00     [ragel]
00:37:48 00:00     [jaxb]
00:37:48 00:00     [wire]
00:37:48 00:00     [aapt]
00:37:48 00:00     [scrooge]
00:37:48 00:00     [spindle]
00:37:48 00:00   [resolve]
00:37:48 00:00     [ivy]
00:37:48 00:00     [go]
00:37:48 00:00     [npm]
00:37:48 00:00   [resources]
00:37:48 00:00     [prepare]
00:37:48 00:00     [services]
00:37:48 00:00   [compile]
00:37:48 00:00     [compile]
00:37:48 00:00     [jvm]
00:37:48 00:00       [jvm-compilers]
00:37:48 00:00         [zinc-pre]
00:37:48 00:00         [zinc-post]
00:37:48 00:00     [jvm-dep-check]
00:37:49 00:01     [checkstyle]
00:37:49 00:01     [scalastyle]
00:37:49 00:01     [cpp]
00:37:49 00:01     [go]
00:37:49 00:01     [python-eval]
                   Invalidated 1 target..
00:37:49 00:01           [chroot]
00:37:50 00:02           [eval]

00:37:51 00:03     [pythonstyle]
                   Invalidated 2 targets.
00:37:51 00:03   [test]
00:37:51 00:03     [run_prep_command]
00:37:51 00:03     [test]
00:37:51 00:03     [pytest]
00:37:51 00:03       [run]

00:37:51 00:03         [chroot]
                     ============== test session starts ===============
                     platform darwin -- Python 2.7.9 -- py-1.4.30 -- pytest-2.6.4
                     plugins: timeout
                     collected 276 items

                     tests/python/pants_test/java/test_nailgun_integration.py .
                     tests/python/pants_test/util/test_eval.py ......
                     tests/python/pants_test/pantsd/test_watchman_client.py .
                     tests/python/pants_test/java/test_nailgun_client.py ...........
                     tests/python/pants_test/util/test_memo.py ..............
                     tests/python/pants_test/util/test_meta.py ...
                     tests/python/pants_test/pantsd/service/test_pants_service.py ...
                     tests/python/pants_test/pantsd/test_process_manager.py .................................................
                     tests/python/pants_test/java/test_nailgun_protocol.py ..................
                     tests/python/pants_test/util/test_dirutil.py .................
                     tests/python/pants_test/pantsd/service/test_pailgun_service.py .
                     tests/python/pants_test/util/test_socket.py ....
                     tests/python/pants_test/util/test_contextutil.py .................
                     tests/python/pants_test/util/test_fileutil.py ..
                     tests/python/pants_test/util/test_osutil.py ....
                     tests/python/pants_test/util/test_strutil.py ..
                     tests/python/pants_test/util/test_xml_parser.py ...........
                     tests/python/pants_test/util/test_timeout.py ....
                     tests/python/pants_test/java/test_executor.py .......
                     tests/python/pants_test/util/test_filtering.py ......
                     tests/python/pants_test/pantsd/test_pailgun_server.py .....
                     tests/python/pants_test/pantsd/service/test_fs_event_service.py .........
                     tests/python/pants_test/java/test_util.py ....
                     tests/python/pants_test/java/test_nailgun_io.py ..........
                     tests/python/pants_test/java/distribution/test_distribution.py ..........................................
                     tests/python/pants_test/java/jar/test_manifest.py ....
                     tests/python/pants_test/pantsd/test_watchman.py .....................

                     ========== 276 passed in 26.44 seconds ===========
                     Name                                                                      Stmts   Miss Branch BrMiss  Cover
                     -----------------------------------------------------------------------------------------------------------
                     src/python/pants/java/nailgun_client                                         74      1      8      1    98%
                     src/python/pants/java/nailgun_io                                             61      2     12      1    96%
                     src/python/pants/java/nailgun_protocol                                       98      1     24      1    98%
                     src/python/pants/pantsd/pailgun_server                                       63      5      6      3    88%
                     src/python/pants/pantsd/process_manager                                     214      8     42      2    96%
                     src/python/pants/pantsd/service/fs_event_service                             43      0     12      0   100%
                     src/python/pants/pantsd/service/pailgun_service                              31     10      6      4    62%
                     src/python/pants/pantsd/service/pants_service                                15      0      2      1    94%
                     src/python/pants/pantsd/watchman                                             78      0     20      0   100%
                     src/python/pants/pantsd/watchman_client                                      26     21      8      8    15%
                     src/python/pants/util/contextutil                                            98     15     22      8    81%
                     src/python/pants/util/dirutil                                               137     29     54     19    75%
                     src/python/pants/util/socket                                                 21      0      4      1    96%
                     ...
                     -----------------------------------------------------------------------------------------------------------
                     TOTAL                                                                      2570    134    270     60    93%

00:38:22 00:34     [junit]
00:38:22 00:34     [go]
00:38:22 00:34     [npm]
00:38:22 00:34   [test-changed]
00:38:22 00:34     [test-changed]
00:38:22 00:34   [complete]
               SUCCESS

4) illustrating an ~order-of-magnitude speedup on no-op runs:

[illuminati hammerpants]$ gcc -o ng ng.c
[illuminati hammerpants]$ ln -s ng ./pants
[illuminati hammerpants]$ export NAILGUN_PORT=`cat ~/dev/pants/.pids/pantsd/socket_pailgun`
[illuminati hammerpants]$ time ./pants
[pantsd]
No goals specified.
Use `pants goals` to list goals.
Use `pants help` to get help.

real    0m0.070s
user    0m0.001s
sys     0m0.004s

[illuminati hammerpants]$ time ./pants help-advanced > /dev/null
[pantsd]

real    0m0.077s
user    0m0.001s
sys     0m0.003s

vs normal startup

[illuminati pants (kwlzn/pantsd/base)]$ time ./pants
No goals specified.
Use `pants goals` to list goals.
Use `pants help` to get help.

real  0m0.753s
user  0m0.341s
sys   0m0.213s

[illuminati pants (kwlzn/pantsd/base)]$ time ./pants help-advanced > /dev/null

real  0m1.350s
user  0m0.387s
sys   0m0.586s

Issues

  • 0
  • 9
  • 0
  • 9
Description From Last Updated
Kris Wilson
Benjy Weinberger
Stu Hood
Kris Wilson
Benjy Weinberger
Benjy Weinberger
Benjy Weinberger
Patrick Lawson
Benjy Weinberger
Benjy Weinberger
Kris Wilson
Kris Wilson
Review request changed

Status: Closed (submitted)

Change Summary:

thanks Stu, Benjy & Patrick! submitted @ 56125e3cf0f47b67f1b2e8264445e2696af2867b

Loading...