Initial cut at pants-dev docker environment.

Review Request #2842 — Created Sept. 17, 2015 and discarded

2162, 2214
areitz, davidt
This uses a minimal-ish pre-built image described by Dockerfile pushed
to pantsbuild/pants:latest and just customizes a pantsbuild user to
match the local user's gid/uid so the pants clone volume mount is

 Dockerfile | 11 +++++++++++
 docker-dev | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+)

Local manual testing only so far, these work:
./pants server
./pants test bundle {src,tests}/{java,scala}::

CI went green here:

  • 0
  • 0
  • 3
  • 1
  • 4
Description From Last Updated
  1. This is a bit of a strawman - docs are needed to explain the use mode, etc, but looking for feedback on 3 things:
    1. Is there a better way to deal with portable local host volume mount perms?  I do a crazy secondary image build, feels like this should be easier with a gid/uid map of some sort.
    2. Is there a better way to do port mapping?  I'd love some way to keep `./pants server` using an ephemeral port but I could not figure out a nice way that keeps the user from having to know about `docker ps` and/or `docker port`.
    3. Does this work for you?  Should just be `./docker-dev` and wait a while the 1st time - dropped to a shell that is centered on your pants clone where ./pants commands should all work (mod having to `git clean -fdx` due to platform difference).
      1. Not sure I understand this one (I'm not a big docker person). Why does it matter that uid/gid match between the VM and local machine? Will you be exporting the VM's filesystem via NFS?
      2. Did you play with -P=false? According to the doc, that will publish all exposed ports:
      3. It seems like this is just a fancier vagrant? Is the plan to start using these docker images in the cloud somewhere (i.e. in Travis)?
    1. 1. Nope - its union mounted for raw native speed.  Turns out this is supremely important for pants.  So the image has only administrative users like root, and not a user to match you uid/gid.  As a result, when you volume mount the pants clone from your local host - presumably owned by you, there is no user in the image with gid=1000 uid=1000, for example.
      2. No - but did read about it.  The problem is that it publishes to ephemeral ports on the host side, so you'd see `INFO] Launched server with pid 3244 at http://localhost:60449` and that port number would be a lie on the outside where you have a browser.
      3. Not really, is a kernel container running over a local union mount - no vms.  I tried vagrant/virtualbox but the speed is >2 orders of magnitude slower due to FS slowdown on the vagrant synced/shared folder for the pants clone.  So - this is very useable on linux where its all native and using your kernel, but on OSX I have no clue.  My understanding is docker installs a virtualbox-using shim on OSX, so perf there may be horrible.
    2. Ah, okay. I was thinking solely about Docker on Mac OS X. I can definitely see advantages of running it when the host OS is Linux.

    3. You may have missed the slack conversation.  Peter Schuller was trying to use pants on some AWS instances that were running into issues - no python headers, no proper unicode locale.  The Vagrant or Docker -> Docker dev image to ease on-boarding sprung as a result.  Of course it only eases linux dev and even then just saves you ~2 steps - installing python devel and setting up a sane modern locale, so this may go ~unused.
  2. docker-dev (Diff revision 1)

    You may want to look into the -u arg for docker run. Although, I think the way you have here is the more common practice among the docker community currently.

    1. Huh - I did not expect that to work. I assumed the uid/gid must exist to pick them for the run, but they need not:

      $ docker run -u 1002:1002 -it -v $PWD:/home/pantsbuild/pants pants-dev-local:latest
      groups: cannot find name for group ID 1002
      I have no name!@454547f0e703:~/pants$ id 
      uid=1002 gid=1002 groups=1002
      I have no name!@454547f0e703:~/pants$ 

      It sort-of does work, but I've added a comment to the Dockerfile to explain the shim-image path noting this experiment.

  3. docker-dev (Diff revision 1)

    You may want to consider using --privileged=true & --pid=host given the scope of what this container does.

    1. Makes sense, done.
  2. docker-dev (Diff revision 1)

    Maybe respect XDG_CACHE_HOME, since that's what pants does?

  3. docker-dev (Diff revision 1)

    nit: storing the output of id -ru and id -rg would greatly improve readability

  4. docker-dev (Diff revision 1)

    I think you might need -t, similar to here:

    for extra-mega-bonus points, it would be nifty if there was a that could be sourced that did common things like this.

    1. No bonus points for me today.  Fixed the -t though.
  1. If either of you can find some time to install docker and try out `./docker-dev` I'll move on to writing some docs for this and let it loose.
    1. I can give it a whack on OS X tomorrow. If this needs a Linux host, that... might be more challenging to come by.

    2. I've got the linux end covered, OSX feedback would be great.
    3. Okay, here is what I did:

      1. Downloaded and installed Docker for Mac OS X
      2. Verified that the docker command exists
      3. Patched this RB into my local pants clone
      4. Ran the script. Here is the output:

      [prometheus pants (master)]$ ./docker-dev
      === Building a customized image for areitz:staff (501:20) ===
      Sending build context to Docker daemon 3.072 kB
      Step 0 : FROM pantsbuild/pants:latest
      latest: Pulling from pantsbuild/pants
      6e6a100fa147: Pull complete 
      13c0c663a321: Pull complete 
      2bd276ed39d5: Pull complete 
      013f3d01d247: Pull complete 
      14ee40116a3c: Pull complete 
      8bf608f9153b: Pull complete 
      Digest: sha256:4fbe7ff8c3cf343c4dd2049c44a988613bb6c35944d8d3d32b6e89acc3b8e550
      Status: Downloaded newer image for pantsbuild/pants:latest
       ---> 8bf608f9153b
      Step 1 : RUN groupadd --gid 20 -r staff &&     useradd --uid 501  -r -m -g staff areitz
       ---> Running in 27760359bdb0
      groupadd: group 'staff' already exists
      The command '/bin/sh -c groupadd --gid 20 -r staff &&     useradd --uid 501  -r -m -g staff areitz' returned a non-zero code: 9
      === Image built and tagged as pants-dev-local:latest ===
      Unable to find image 'pants-dev-local:latest' locally
      Pulling repository
      Error: image library/pants-dev-local:latest not found

      I'll do a bit more poking around to see if this is something that I did wrong on my end.

    4. Nothing you did wrong I just assume the user/group mirroring you will not exist in the ubuntu 15.04 image, which is a bad assumption.  I can work with that data - thanks!
    5. After I applied this diff:

      [prometheus pants (master)]$ git diff
      diff --git a/docker-dev b/docker-dev
      index 09d71ea..2a80f25 100755
      --- a/docker-dev
      +++ b/docker-dev
      @@ -36,8 +36,9 @@ FROM pantsbuild/pants:latest
       # /home/${user}/pants volume mount of the pants clone has matching perms and can be used
       # transparently.
      -RUN groupadd --gid ${gid} -r ${group} && \\
      -    useradd --uid ${uid}  -r -m -g ${group} ${user}
      +#RUN groupadd --gid ${gid} -r ${group} && \\
      +RUN useradd --uid ${uid}  -r -m -g ${group} ${user}
       VOLUME /home/${user}/.cache/pants
       VOLUME /home/${user}/.ivy2/pants

      And re-running docker-dev, I was plopped into a running Linux VM, in my pants dir. However, I couldn't run pants:

      areitz@default:~/pants$ ./pants goals
      Bootstrapping pants_deps with requirements:
      rm: cannot remove ‘/home/areitz/pants/build-support/pants_dev_deps.venv/.Python’: Permission denied
      rm: cannot remove ‘/home/areitz/pants/build-support/pants_dev_deps.venv/bin/activate’: Permission denied

      Basically, it looks like it needed to re-make the venv (makes sense, I'll need one with Linux .so files), but didn't have permissions to modify any files in my pants workspace on the Mac. Looks like it's using "vboxsf" filesystem:

      areitz@default:~/pants$ mount|grep areitz
      none on /home/areitz/pants type vboxsf (rw,nodev,relatime)
      none on /home/areitz/.ivy2/pants type vboxsf (rw,nodev,relatime)
      none on /home/areitz/.cache/pants type vboxsf (rw,nodev,relatime)

      Here is my user account in the VM:

      areitz@default:~/pants$ id
      uid=501(areitz) gid=50(staff) groups=50(staff)

      And on my Mac:

      [prometheus pants (master)]$ id
      uid=501(areitz) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),80(admin),33(_appstore),98(_lpadmin),100(_lpoperator),204(_developer),395(,398(,399(,401(

      Let me know if you need more info.

    6. The need to git clean -fdx build-support/ is expected, the perm issues were not. Can you give more detail on the perms for /home/areitz/pants/build-support/pants_dev_deps.venv/.Python for example? Presumably your uid/gid now align with your host udi/gid.

    7. Hmm, that's pretty weird, it's showing uid 1000:

      areitz@default:~/pants$ ls -la /home/areitz/pants/build-support/pants_dev_deps.venv/.Python
      lrwxr-xr-x 1 1000 staff 80 Sep 21 21:58 /home/areitz/pants/build-support/pants_dev_deps.venv/.Python -> /opt/twitter/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/Python
    8. Looks like this is fairly consistent:

      areitz@default:~/pants$ ls -la|head
      total 108
      drwxr-xr-x 1   1000 staff  1326 Sep 22 18:12 .
      drwxr-xr-x 5 areitz staff  4096 Sep 22 18:12 ..
      drwxr-xr-x 1   1000 staff   238 Sep 22 17:44 3rdparty
      -rw-r--r-- 1   1000 staff  1006 May 22 17:08 BUILD
      drwxr-xr-x 1   1000 staff   612 Sep 22 18:00 build-support
      -rw-r--r-- 1   1000 staff  1734 Sep 22 17:44
      drwxr-xr-x 1   1000 staff   374 Sep 22 17:44 contrib
      -rw-r--r-- 1   1000 staff   216 Aug 11  2014
      -rw-r--r-- 1   1000 staff  1656 Sep 22 17:44
    9. How about on the host itself, same mix?  It _should_ be the same mix!
    10. [prometheus pants (master)]$ ls -la |head
      total 208
      drwxr-xr-x+  39 areitz  staff   1326 Sep 22 11:12 .
      drwxr-xr-x+ 128 areitz  staff   4352 Sep 22 10:59 ..
      drwxr-xr-x+  18 areitz  staff    612 Sep 22 11:01 .git
      -rw-r--r--+   1 areitz  staff    103 Jun  5 11:22 .gitattributes
      -rw-r--r--+   1 areitz  staff    525 Jun 22 14:43 .gitignore
      -rw-r--r--+   1 areitz  staff    245 Mar 23  2015 .isort.cfg
      -rw-r--r--+   1 areitz  staff   1339 Sep 22 10:44 .mailmap
      drwxr-xr-x+   8 areitz  staff    272 Sep 17 10:28 .pants.d
      drwxr-xr-x+   3 areitz  staff    102 Sep 17 10:28 .pids
      [prometheus pants (master)]$ ls -la /Users/areitz/workspace/pants/build-support/pants_dev_deps.venv/.Python
      lrwxr-xr-x  1 areitz  staff  80 Sep 21 14:58 /Users/areitz/workspace/pants/build-support/pants_dev_deps.venv/.Python -> /opt/twitter/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/Python

      Where areitz == uid 501.

    11. Huh - ok.  Let me work with this a bit.  The 1000 uid on the ubuntu image is suspicous since the image comes with no such uid IIRC.  When I first got the image up, before adding a custom user, I saw ownership by 1000 (my local host arch uid).  So that was saying jsirois is unkown on the machine.
    12. I added a fix for the pre-existing group issue. The perms issue you're getting with mixed ownership on the docker image though I have no ideas.
      You can at least test the group bit works with:
      rm -rf ~/.cache/pantsbuild/docker/ && docker rmi pants-dev-local:latest && ./docker-dev

      That will reset your local state and rebuild the shim image.

      I may just proceed to doc this up if the group bit works for you but the perms are still off. I can warn this is only tested for linux hosts and then ping Peter. A partial solution is probably better to get out there than none and a motivated OSX person can debug the issues from there.

Review request changed

Status: Discarded