12 KB
Newer Older
Ludovic Courtès's avatar
Ludovic Courtès committed
title: Guix-HPC debut!
author: Ludovic Courtès, Roel Janssen, Pjotr Prins, Ricardo Wurmus
Ludovic Courtès's avatar
Ludovic Courtès committed
date: 2017-05-15 14:00
tags: kickoff reproducibility containers singularity shifter easybuild spack
Ludovic Courtès's avatar
Ludovic Courtès committed
5 6 7 8

This post marks the debut of Guix-HPC, an effort to optimize
[GNU Guix]( for reproducible scientific workflows
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
in high-performance computing (HPC).  Guix-HPC is a joint effort between
[Inria](, the
[Max Delbrück Center for Molecular Medicine (MDC)](,
and [UMC Utrecht](  Ludovic Courtès,
Ricardo Wurmus, and Roel Janssen are driving the effort in each of these
institutes, each one focusing specific areas of interest within this
overall Guix-HPC effort.  Our institutes have in common that they are
users of HPC, and that, as scientific research institutes, they have an
interest in using reproducible methodologies for carry out their

Our goals are really twofold: reproducible scientific workflows, and HPC
software deployment.  In this post we will describe what we hope to
achieve in the coming months in these areas.

# Reproducible scientific workflows

[Reproducible research](”—something
that sounds like a pleonasm—is becoming more important.  More and more
experiments rely on computation and software, but relying on software
should not make these experiments non-reproducible.  Unfortunately that
has often been the case, but this is changing.  To give a few examples,
the US National Science Foundation (NSF) is now
[encouraging reproducibility in computing](,
[such as Nature](
insists of the importance of sharing source code and supporting
reproducibility, major conferences such as SuperComputing now have
[reproducibility guidelines](,
and Inria’s upcoming
[strategic plan](
will have a chapter on reproducible science.

We believe that a prerequisite of reproducible science are _reproducible
software environments_.  Of course it takes more than reproducible
software environments to achieve reproducible scientific workflows—for
instance, reproducible numerical recipes and deterministic parallel
programs are crucial in HPC.

In fact there’s a whole spectrum of initiatives around reproducibility:
at one end of the spectrum, efforts like
[Software Heritage]( allow us to preserve
source code.  At the other end, there are efforts like
[ReScience](, a peer-reviewed journal for
computational sciences where everyone can _replicate_ the computational
experiments.  Somewhere in the middle of the spectrum, projects such as
GNU Guix strive to support flexible and reproducible software
deployment, as we described in our
[2015 RepPar paper](, _Reproducible
and User-Controlled Software Environments in HPC with Guix_.

In a nutshell, just like they are able to unambiguously refer to a
research article by its
researchers should be able to unambiguously refer to software artifacts.
Likewise, just like researchers should publish the data they rely on in
their articles so that their peers can assess and question their
66 67 68
work, they should publish the software that led to a particular result.
This necessity is particular acute in areas such as numerical simulation
or the development of run-time systems or compilers for HPC, since the
results that are published depend in large part on software.
70 71 72 73 74 75

# Containers to the rescue?

“Look, we have a solution!”, I hear you say.  “Just use containers!
[Docker](, [Singularity](,
76 77 78 79 80 81
you name it!”  “Even Nature
[had an article](
explaining how containers ‘can boost the reproducibility of data-driven

It is true that “containers”, or, more accurately,
82 83
[“application bundles”](,
solve part of the reproducibility problem.  Without them, users would
build applications by themselves, and, inevitably, their software
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105
environment would be vastly different from yours: different version of
the application, different compiler versions, different libraries,
different compilation options, and so on.  Conversely, an application
bundles contains all the software that you used, the _exact same bits_
that those you run on your machine.  In that sense, application bundles
get users a long way towards truly reproducible software environments.

As we jokingly illustrated in
[a talk at FOSDEM](,
those app bundles are akin to shrink-wrapped computers: they include all
of the operating system and libraries that the application needs.  But
like shrink-wrapped computers, they’re inconvenient to deal with: you
can’t easily unwrap them and add another application in them—another
shrink-wrapped computer; you can’t easily unwrap them and experiment
with a specific component; if you do unwrap them, you still do not know
how they were built.

But is bit-for-bit reproducibility all that a researcher cares about?
Don’t researchers want to go _beyond_ reproducing the exact same bits?
Of course they do.  To pick a concrete example, one of the teams at
Inria develops [StarPU](, a run-time
system to schedule computational tasks on CPUs, GPUs, and accelerators.
107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148
The team next-door develops linear algebra software such as
[Chameleon]( that relies
on StarPU to get good performance.  Researchers working on scheduling in
StarPU might want to fiddle with how Chameleon uses StarPU and see how
this affects performance; likewise, Chameleon developers might want to
fiddle with the scheduling algorithms StarPU uses to see if it could
work better for their workload.  This simple example illustrates the
need for tools that not only support reproducibility, but also _enable

We believe that app bundles fall short on this requirement.  Conversely,
Guix supports both bit-for-bit reproducibility _and_ experimentation.
In Guix,
[package definitions](
express software composition in unambiguous terms.  Given a commit of
the [Guix repository](,
anyone building a set of packages as defined in Guix will get _the exact
same result_, usually [bit-for-bit](
Yet, Guix also allows users to define _package variants_, either
programmatically or
[from the command line](
These properties get us closer to having both reproducible _and_
customizable software deployments.

# Software deployment on HPC systems

Perhaps at this point you’re thinking that this sounds interesting, but
that all you’ve ever seen when it comes to software deployment on HPC
systems is the venerable [“modules”](
tool set.  “Modules” provide a level of customization that’s greatly
appreciated on these big multi-user HPC systems: as a user you get to
choose which versions of the packages to load in your environment, and
you can have separate sessions using different packages and package
versions.  The downside is that modules rely on sysadmins to package the
software.  Thus, different machines may have different modules; modules
may vanish when sysadmins decide it—that could be on the day before the
deadline for your super-important paper; useful packages might be
missing a module; packages available as modules might be configured not
quite they way you’d like; and so on.  In other words, there’s a lack of

This problem has been recognized over the year by the HPC community.
149 150 151
For example, the
[Strategic Research Agenda of the European Technology Platform for HPC](,
Section 5.2.3, calls for more “flexible execution contexts” on clusters.
152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184
The main answers to that, in addition to containers, which are still
mostly unavailable on HPC systems, has been to develop custom package
managers such as [EasyBuild]( and
[Spack](  These tools are comparable to
standard package managers found on GNU/Linux (apt, yum, etc.), but they
are tailored for use by unprivileged HPC users.  Without being root on
the supercomputer, you can now build and install software package by
EasyBuild or Spack.  On top of that, these tools provide ways for you to
customize package recipes.  For example, Spack has
[many command-line and configuration options](
allowing you to customize the build process—choosing the compiler that
is used, choosing the MPI or BLAS implementation, etc.

The main downside, this time, is that the very ability to use these
tools without root privileges hampers reproducibility.  Concretely, what
builds on one machine might fail to build on another one, as illustrated
by reports like
[this one](
[that one](
It is the kind of discrepancy that Guix tries hard to avoid.  It runs
each build
[in isolated environments](
where only the build’s inputs are accessible.  Unless the software or
its build procedure is non-deterministic, this ensures that a given
build process _produces the same result on any machine_.  In other
words, if I managed to build a package on my laptop, I can tell you you
can build the same package on your supercomputer and get the exact same
bits.  That’s a good property to have!

# Stay tuned!

Truth to be said, to our knowledge, Guix is currently deployed only on a
handful of supercomputers today: the MDC runs it, so does UMC Utrecht,
186 187
Inria’s research center in Bordeaux will deploy it on its cluster in the
coming months, and a look at the activity on the `guix-devel` mailing
list suggests that several other HPC deployments are actively used in
189 190 191 192 193 194 195 196 197
academic contexts.  There are several reasons why there aren’t more Guix
deployments on HPC clusters.  One of them is that Guix is only 4 years
old, and it was not specifically written as a tool for HPC and research
scientists—it just happens to have useful properties in this context.
The other reason is the fact that, to achieve build isolation,
[it relies on features of the kernel Linux](
usually only available to system administrators.

The Guix-HPC effort will attempt to address this, both through technical
198 199 200
means (more on that later), by discussing and raising awareness of
the reproducibility and deployment situation in the HPC community, and
by giving talks and running training sessions.  We
201 202 203 204 205 206 207 208 209
will also work on improving the user experience and workflows for
researchers using Guix with HPC and reproducibility in mind.

In future posts, we’ll first describe what the institutes involves in
Guix-HPC expect from this project, and how they take advantage of Guix.
We’ll also describe concrete actions we’ve identified to achieve these
goals.  So please stay tuned, and share your ideas, comments, and
criticism with us on the
[`guix-devel` mailing list](!
210 211 212
If you are already using Guix and would like to join the effort, or if
you have questions about training sessions or deployment on your
cluster, please email us on the `` private alias.