Initial round of pantsd + new engine + watchman integration.

Review Request #3524 - Created March 2, 2016 and submitted

Information
Kris Wilson
pants
kwlzn/pantsd/engine_integration
2989
Reviewers
pants-reviews
benjyw, ity, patricklaw, peiyu, stuhood
  • Implement a new SchedulerService for managing the online scheduler instance.
  • Revive FSEventService and friends for Watchman integration.
  • Avoid resetting Subsystem options to avoid uninitialized subsystem errors when attempting to use the WatchmanLauncher subsystem in pantsd - this gets stomped on run over run anyhow.
  • Make use of FSEventService and SchedulerService optionable - defaulting to off.
  • Implement a test target in exp to launch pantsd with a copy of the scheduler and listen for file events (currently all this does is log the events - graph invalidation coming soon!).
  • Extend the kill-pantsd goal to also shutdown Watchman when so configured.

CI is away @ https://travis-ci.org/pantsbuild/pants/builds/113015980


while running this in a separate window:

$ tail -F .pants.d/pantsd/pantsd.log .pants.d/watchman/watchman.log

spin up an unconnected pantsd instance using the test target w/ options to enable fs-event-detection:

[illuminati pants (kwlzn/pantsd/engine_integration)]$ ./pants run src/python/pants/engine/exp/legacy:pantsd -q -- src/python/pants/pantsd:: -ldebug --pantsd-fs-event-detection
\*\*\* pantsd launched \*\*\*
DEBUG] acquiring lock: <OwnerPrintingPIDLockFile: u'/Users/kwilson/dev/pants/.pantsd.startup' -- u'/Users/kwilson/dev/pants/.pantsd.startup'>
DEBUG] launching pantsd
DEBUG] purging metadata directory: /Users/kwilson/dev/pants/.pids/pantsd
DEBUG] released lock: <OwnerPrintingPIDLockFile: u'/Users/kwilson/dev/pants/.pantsd.startup' -- u'/Users/kwilson/dev/pants/.pantsd.startup'>
DEBUG] pantsd is running at pid 78563

see log output in the tail window:

D0301 17:45:57.109353 78563 pants_daemon.py:121] logging initialized
I0301 17:45:57.109931 78563 pants_daemon.py:170] pantsd starting, log level is DEBUG
I0301 17:45:57.111968 78563 pants_daemon.py:144] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x106902510>
I0301 17:45:57.112404 78563 pants_daemon.py:144] starting service <pants.pantsd.service.pailgun_service.PailgunService object at 0x1069021d0>
I0301 17:45:57.112715 78563 pailgun_service.py:51] starting pailgun server on port 56898
I0301 17:45:57.112966 78563 pants_daemon.py:144] starting service <pants.pantsd.service.fs_event_service.FSEventService object at 0x106902450>
I0301 17:45:57.116261 78563 watchman_launcher.py:56] watchman is running, pid=70857 socket=/Users/kwilson/dev/pants/.pants.d/watchman/watchman.sock
D0301 17:46:02.122070 78563 watchman.py:147] watchman command_list is: [[u'subscribe', '/Users/kwilson/dev/pants', u'all_files', {'fields': [u'name'], 'expression': [u'allof', [u'type', u'f'], [u'not', [u'dirname', u'dist', [u'depth', u'eq', 0]]], [u'not', [u'match', u'.*', u'wholename']], [u'not', [u'match', u'*.pyc']]]}]]
I0301 17:46:02.265883 78563 watchman.py:153] confirmed watchman subscription: {'subscribe': 'all_files', 'version': '3.1.0', 'clock': 'c:1456869416:70857:1:4433'}
I0301 17:46:02.267790 78563 scheduler_service.py:44] enqueuing 4968 changes for subscription all_files
D0301 17:46:02.292135 78563 scheduler_service.py:75] processing 4968 files for subscription all_files (first_event=True)
D0301 17:46:03.268616 78563 fs_event_service.py:143] callback ID 1 for all_files succeeded

save various src files in my editor, which incur new log events:

I0301 17:49:30.558384 78563 scheduler_service.py:44] enqueuing 1 changes for subscription all_files
D0301 17:49:30.575396 78563 scheduler_service.py:75] processing 1 files for subscription all_files (first_event=False)
D0301 17:49:30.575889 78563 scheduler_service.py:56] file src/python/pants/pantsd/service/fs_event_service.py changed!
D0301 17:49:31.562772 78563 fs_event_service.py:143] callback ID 2 for all_files succeeded
I0301 17:49:33.095844 78563 scheduler_service.py:44] enqueuing 1 changes for subscription all_files
D0301 17:49:33.112721 78563 scheduler_service.py:75] processing 1 files for subscription all_files (first_event=False)
D0301 17:49:33.113214 78563 scheduler_service.py:56] file src/python/pants/pantsd/service/pants_daemon_launcher.py changed!
D0301 17:49:34.098309 78563 fs_event_service.py:143] callback ID 3 for all_files succeeded

shut it all down with:

[illuminati pants (kwlzn/pantsd/engine_integration)]$ ./pants kill-pantsd -ldebug --pantsd-fs-event-detection
18:03:39 00:00 [main]
               (To run a reporting server: ./pants server)
18:03:39 00:00   [setup]
18:03:39 00:00     [parse]
               Executing tasks in goals: kill-pantsd
18:03:39 00:00   [kill-pantsd]
18:03:39 00:00     [kill-pantsd]DEBUG] terminating pantsd
DEBUG] sending signal 15 to pid 82783
DEBUG] successfully terminated pid 82783
DEBUG] purging metadata directory: /Users/kwilson/dev/pants/.pids/pantsd
DEBUG] terminating watchman
DEBUG] sending signal 15 to pid 82696
DEBUG] successfully terminated pid 82696
DEBUG] purging metadata directory: /Users/kwilson/dev/pants/.pids/watchman

18:03:39 00:00   [complete]
               SUCCESS

Issues

  • 0
  • 1
  • 0
  • 1
Description From Last Updated
Stu Hood
Stu Hood
Peiyu Wang
Kris Wilson
Review request changed

Status: Closed (submitted)

Change Summary:

thanks gents! submitted @ e28fe8614a849d0de90af5b6844cf47614a2bb27

Loading...