11.4 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
title: Using Guix Without Being root
author: Ludovic Courtès
date: 2017-10-02 15:00:00
tags: pack bundle PRoot container

In [the previous
post](/blog/2017/09/reproducibility-and-root-privileges), we saw that
Guix’s build daemon needs to run as `root`, and for a good reason:
that’s currently the only way to create isolated build environments for
packages on GNU/Linux.  This requirement means that you cannot use Guix
on a cluster where the sysadmins have not already installed it.  In this
article, we discuss how to take advantage of Guix on clusters that lack
a proper Guix installation.

# Setting the stage

So you get access to one of these powerful supercomputers, which will
allow you to run (and/or debug :-)) your number crunching application in
the parallel setting of your dreams.  Now, before you can get there, you
first have to _deploy_ your application, and that’s where things often
start looking muddy: the machine runs an ancient GNU/Linux with a bunch
of sysadmin-provided “environment modules.”

Most likely you’ll end up building your application and its
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74
dependencies by hand, just to notice later that it doesn’t quite behave
the same way as on your lab’s cluster, and yet again differently from
what happens on your laptop.  Wouldn’t it be great if you could use the
exact same software on all these machines?

# Creating bundles with `guix pack`

Earlier this year, [we announced `guix
a command that allows you to create a _binary bundle_ of a set of
packages and all their dependencies.  For instance, we can create a
tarball containing the [hwloc hardware topology
toolkit]( and its dependencies
by running:

$ guix pack hwloc -S /bin=bin


The `-S` switch here instructs `guix pack` to create a `/bin` symlink
inside the tarball that points to hwloc’s `bin` directory (more on
that in [this blog
and in [the
The tarball contains two directories at its root: this `/bin` symlink,
and `/gnu/store`, which contains the actual binaries.

We can send this tarball to the supercomputer:

laptop$ scp `guix pack hwloc -S /bin=bin` supercomputer:hwloc.tar.gz

… and then unpack it on the supercomputer:

supercomputer$ mkdir -p ~/.local
supercomputer$ cd ~/.local
supercomputer$ tar xf ~/hwloc.tar.gz

The problem that we have now is that we cannot run these binaries, first
because `~/.local/bin` is a symlink to `/gnu/store`, which does not
exist on that machine, and second because binaries in general are not

So we need to _map_ `$HOME/.local/gnu/store` to `/gnu/store` in the
76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285
execution environment of hwloc.  We’ve [seen
before](/blog/2017/09/reproducibility-and-root-privileges) that this
normally requires root privileges, so let’s see how we can work around

# User namespaces again

Once again, [user
can save us—when they’re available.  With the
[`unshare`]( and
[`chroot`]( commands, we
can create that environment:

$ unshare -mrf chroot ~/.local /bin/lstopo --version
lstopo 1.11.8

This command creates a new process in a separate mount and user
namespace, in which it runs `chroot` to change the root to
`$HOME/.local` and finally invoke hwloc’s `lstopo`.  In this
environment, `/bin/lstopo` actually corresponds to

Unfortunately, few HPC installations offer this option today, so we need
another solution.

# PRoot

In the absence of user namespaces, we can resort to
[PRoot](/package/proot).  PRoot supports OS resource virtualization in a
way conceptually similar to namespaces.  It does that by tracing
processes with
[`ptrace`](, the
system call that debuggers rely on, which does _not_ require `root`

Guix comes with a [`proot-static`](/package/proot-static) package, which
is a statically-linked version of PRoot.  Because it’s statically
linked, we can build it on our laptop send it to the remote machine and
directly use it there:

laptop$ scp `guix build proot-static`/bin/proot supercomputer:

On the supercomputer, we can now run `lstopo` under PRoot, and tell
`proot` to use `~/.local` as the root file system:

supercomputer$ ./proot -r ~/.local /bin/lstopo
Machine + Package L#0 + L3 L#0 (15MB) + L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0)

Wait, `lstopo` is telling us that there’s a single CPU with a single
core on that machine, what’s wrong?  Well, we also need to map `/proc`
in the execution environment of `lstopo`, since this is where it gets
most of the information from:

supercomputer$ ./proot -r ~/.local -b /proc /bin/lstopo
Machine (126GB)
  Package L#0
    L3 L#0 (15MB)
      L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0)
      L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 (P#2)
      L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 (P#4)
      L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 (P#6)
      L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4 (P#8)
      L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5 (P#10)
    L3 L#1 (15MB)
      L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 (P#12)
      L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 (P#14)
      L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8 (P#16)
      L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9 (P#18)
      L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + PU L#10 (P#20)
      L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + PU L#11 (P#22)
  Package L#1


So now we have a simple way to run an application from a binary bundle
created with `guix pack`, and to selectively expose OS resources such as
`/proc` or `/dev` nodes.

PRoot does not require `root` privileges, but it comes with a
performance hit: the traced process _stops at every system call_ while
`proot` itself interprets and “translates” the system call.  Quite
heavy-handed.  However, for a mostly computational process, it should
not be much of a problem: the I/O phases of the program’s execution will
be slower, but the core of the program’s execution should be largely
unaffected.  An MPI may still be penalized though, but we do not have
benchmarking results for that yet.

# Singularity, Docker, Shifter

Another solution to run your Guix pack is _via_ Singularity, Shifter, or
Docker if one of them is installed on your supercomputer.
are able to load Docker images, and `guix pack` can create such images:

$ guix pack -f docker -S /bin=bin hwloc


You would then send that image to the supercomputer, import it with
`singularity import` or `docker load`.  Then we can run:

$ singularity exec docker-pack.tar.gz /bin/lstopo



$ IMAGE=`docker load -i docker-pack.tar.gz | cut -d' ' -f3`
$ docker run $IMAGE /bin/lstopo


In this sense, Shifter, Docker, and Singularity provide a nice
foundation to run our application bundles.

# Why bother?

At this point, you may be wondering: if we’re doing to use Docker or
Singularity or Shifter to run our bundle, why bother with `guix pack` in
the first place?  These tools also provide commands to provision
container images, after all.

The answer is simple: with Guix, the bundle _is a reproducible byproduct
for which you have the source_.  That is, if you pick a commit of Guix,
`guix pack hwloc` will always give the same result,
[bit-for-bit](  And
it’s not limited to bundles: `guix package -i hwloc` allows you to
install the very same hwloc.

With Docker & co., the bundle is at the center of the stage.  You surely
have a
or a [“bootstrap
recipe”]( which
allows you to rebuild the image.  While they provide a convenient way to
provision an image, `Dockerfile`s and Singularity recipes have several
shortcomings, from a reproducible science viewpoint:

  1. `Dockerfile`s and recipes do _not_ describe a reproducible image
     build process: they typically resort to external tools such as
     `apt-get` or `pip`, whose results depend on the state of the
     Debian or PyPI repository at the time they are run.
  2. They tell only a small part of the story: the core of a Docker
     image is the [“base
     which is an opaque operating system image, and the `apt-get` and
     `pip` commands in the `Dockerfile` or recipe do not really tell us
     how those binaries we’re adding were produced.
  3. They are the wrong abstraction level: when creating an
     _application_ bundle, it makes more sense to think in terms of the
     application and the software it depends on, than to think in terms
     of commands to run to modify the state of the image.

To put it differently, `Dockerfile`s and Singularity recipes _are not
source_, and the binary they lead to are pretty much inert.

In contrast, Guix describes the _complete_ dependency graph of the
application—the [`guix
command allows us to visualize that.  For each package in the graph, we
know not only what packages it depends on, but also which patches,
configure flags, compiler options, and so on are used to build it.

Since Guix knows how to build everything, customizing the package graph
and recreating a new bundle is easy.  For instance, using the
[command-line package transformation
we can produce a pack of hwloc built from a different source tarball,
but otherwise with the same dependency graph and build options:

$ guix pack hwloc -S /bin=bin --with-source=./hwloc-2.0.0rc1.tar.gz

Likewise, we could create a bundle of the [MUMPS sparse
solver](/package/mumps) linked against [PT-Scotch](/package/pt-scotch)
instead of [Scotch](/package/scotch):

$ guix pack mumps -S /bin=bin --with-input=scotch=pt-scotch

If the command-line options are not enough, we can always go further
[using the

# Wrap-up

For systems lacking a Guix installation, `guix pack` provides a
convenient way to provision container images that can be executed either
with Docker, Singularity, or Shifter, or simply with PRoot—without
requiring root privileges.  `guix pack` distinguishes itself from other
approaches to image provisioning by retaining the nice properties of
Guix: it makes bundles _reproducible from source_, provides a high level
of abstraction, and supports customization and experimentation.