getting-started.md 7.69 KB
Newer Older
1
title: Reproducible software deployment for high-performance computing.
2
frontpage: yes
3 4
---

5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
[GNU Guix](https://www.gnu.org/software/guix/) is a *transactional*
package manager, with support for *per-user* package installations.
Users can install their own packages without interfering with each
other, yet without unnecessarily increasing disk usage or rebuilding
every package.  Users can in fact create as many software environments
as they like—think of it
as [VirtualEnv](https://virtualenv.pypa.io/en/stable/) but not limited
to Python, or [`modules`](http://modules.sourceforge.net/) but not
limited to your sysadmin-provided modules.

The software environments created with Guix are _fully reproducible_: a
package built from a specific Guix commit on your laptop will be exactly
the same as the one built on the HPC cluster you deploy it too, usually
[bit-for-bit](https://reproducible-builds.org/docs/definition/).

20
[We believe](/about) this makes Guix a great foundation for
21 22 23 24 25
reproducible software deployment in high-performance computing (HPC).
Here’s how to get started.

# Installing Guix

26
You can install Guix on your laptop in 5 minutes: just
27
follow
28
[the binary install instructions](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html).
29

30
You’re a cluster sysadmin and would like to have a cluster-wide install?
31
Read [this article](/blog/2017/11/installing-guix-on-a-cluster).
32

33 34
# Installing Packages

35
Say you’re searching for a sparse solver among the
36
[10,000+ packages](/browse) that come with Guix:
37 38

```
39
$ guix search sparse solver
40 41 42
name: mumps
version: 5.0.2
outputs: out
43 44 45 46
systems: x86_64-linux i686-linux armhf-linux aarch64-linux
+ mips64el-linux
dependencies: gfortran-5.4.0 metis-5.1.0 openblas-0.2.19
+ scotch-6.0.4
47 48 49 50
location: gnu/packages/maths.scm:1550:2
homepage: http://mumps.enseeiht.fr
license: CeCILL-C
synopsis: Multifrontal sparse direct solver
51 52 53
description: MUMPS (MUltifrontal Massively Parallel sparse
+ direct Solver) solves a sparse system of linear equations
+ A x = b using Guassian elimination.
54 55 56 57 58 59 60 61
relevance: 12


```

To install it along with the latest GNU compiler tool chain:

```
62
$ guix install mumps gcc-toolchain
63 64
The following packages will be installed:
   mumps	5.0.2	/gnu/store/gg55pn4nk3fl7fvxqqsgqr2w6fds7wa6-mumps-5.0.2
65
   gcc-toolchain	9.1.0	/gnu/store/zs62l7rwvk5180cz3bykjprk2fymsnbs-gcc-toolchain-9.1.0
66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

The following derivations will be built:
   /gnu/store/kipa9k61zkhw4s3frs92w683ps23hpjj-profile.drv
   /gnu/store/73lzwjvr6wx4gb3l9a7vlx6759kcgp7h-fonts-dir.drv
   /gnu/store/3zkyjqg8wsxvypj6ivjpfp9h4lpf9cyg-ca-certificate-bundle.drv
   /gnu/store/37jya2s4kwbkldwfc98x5jvsdynmzzz8-info-dir.drv
   /gnu/store/bihbksjnbhnaxkwbnh03drffmx85vcsm-manual-database.drv
3.1 MB will be downloaded:
   /gnu/store/gg55pn4nk3fl7fvxqqsgqr2w6fds7wa6-mumps-5.0.2



2 packages in profile
The following environment variable definitions may be needed:
   export PATH="$HOME/.guix-profile/bin:$HOME/.guix-profile/sbin${PATH:+:}$PATH"
81
   export CPATH="$HOME/.guix-profile/include${C_INCLUDE_PATH:+:}$C_INCLUDE_PATH"
82 83 84 85 86 87 88 89
   export LIBRARY_PATH="$HOME/.guix-profile/lib${LIBRARY_PATH:+:}$LIBRARY_PATH"
```

# Spawning One-Off Environments

Sometimes all you want is to try out a program without installing it in
your profile.  That’s
where
90
[`guix environment`](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-environment.html) comes
91 92
in.  To create an environment containing Python 3.x, NumPy, and
scikit-learn, run:
93 94

```
95 96 97
$ python3
bash: python3: Command not found
$ guix environment --ad-hoc python@3 python-numpy python-scikit-learn
98 99
The following derivations will be built:
   /gnu/store/2g3mj1xdlq2rj8j0crl4sa68bqhmfsmd-profile.drv
100
The following profile hooks will be built:
101 102 103 104
   /gnu/store/wd0ma3xjq25w2qcnn3x0dgjyrck3dnk0-info-dir.drv
   /gnu/store/n97xqbig6rfliqrw0qbkb1zbnh8v0dis-fonts-dir.drv
   /gnu/store/igdhg9hm5n4npvf41zvznm06c226kx4a-ca-certificate-bundle.drv
   /gnu/store/my4m438264jyq5awk39j20xhdf6symha-manual-database.drv
105 106
building directory of Info manuals...
building database for manual pages...
107 108 109 110 111
[env]$ python3
Python 3.5.3 (default, Jan  1 1970, 00:00:01) 
[GCC 5.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
112 113
>>> import sklearn
>>>
114 115 116 117 118 119
```

# Customizing Packages

Occasionally you’ll want to customize the way packages are
built.
120
[From the command line](https://www.gnu.org/software/guix/manual/en/html_node/Package-Transformation-Options.html),
Roel Janssen's avatar
Roel Janssen committed
121
you can apply transformations, such as replacing one dependency
122 123 124 125 126
with another one in the dependency graph.  The example below replaces
`openmpi` with `openmpi-thread-multiple` in the dependency graph of
`mumps-openmpi`:

```
127
$ guix install mumps-openmpi \
128 129 130 131 132 133 134 135 136 137
     --with-input=openmpi=openmpi-thread-multiple
```

The expressivity of the command line is limited, but you can go further
by writing your own package definitions.

# Defining Packages

To add a package, you
can
138
[generate a template from a third-party repository using `guix import`](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-import.html),
139 140
or you
can
141
[write a package definition](https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html),
142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174
which looks like this:

```scheme
(define-public scalapack
  (package
    (name "scalapack")
    (version "2.0.2")
    (source
     (origin
       (method url-fetch)
       (uri (string-append "http://www.netlib.org/scalapack/scalapack-"
                           version ".tgz"))
       (sha256
        (base32
         "0p1r61ss1fq0bs8ynnx7xq4wwsdvs32ljvwjnx6yxr8gd6pawx0c"))))
    (build-system cmake-build-system)
    (inputs
     `(("mpi" ,openmpi)
       ("fortran" ,gfortran)
       ("lapack" ,lapack)))             ;for testing only
    (arguments
     `(#:configure-flags `("-DBUILD_SHARED_LIBS:BOOL=YES")))
    (home-page "http://www.netlib.org/scalapack/")
    (synopsis "Library for scalable numerical linear algebra")
    (description
     "ScaLAPACK is a Fortran 90 library of high-performance linear algebra
routines on parallel distributed memory machines.  ScaLAPACK solves dense and
banded linear systems, least squares problems, eigenvalue problems, and
singular value problems.")
    (license (license:bsd-style "file://LICENSE"
                                "See LICENSE in the distribution."))))
```

175 176
You can have your own package collection published as a
[channel](https://www.gnu.org/software/guix/manual/en/html_node/Channels.html).
177

178
# Sending Packages to Guix-less Machines
179 180 181

What if the target supercomputer lacks Guix?  You can still enjoy Guix’s
reproducibility and customizability by [sending your package binaries
182 183 184
there](/blog/2017/10/using-guix-without-being-root/), leveraging
[relocatable
binaries](https://www.gnu.org/software/guix/blog/2018/tarballs-the-ultimate-container-image-format/):
185 186

```
187
laptop$ scp `guix pack -RR hwloc -S /bin=bin` supercomputer:hwloc.tar.gz
188 189 190

supercomputer$ mkdir -p ~/.local
supercomputer$ (cd ~/.local; tar xf ~/hwloc.tar.gz)
191
supercomputer$ ~/.local/bin/lstopo
192 193
```

194 195 196
Other options include building [Singularity or Docker
images](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-pack.html).

197 198 199 200 201 202 203 204 205 206
# Learning More

Find the main commands in the [quick reference
card](https://www.gnu.org/software/guix/guix-refcard.pdf).  Learn more
in the reference manual:
[Deutsch](https://www.gnu.org/software/guix/manual/de/html_node) |
[English](https://www.gnu.org/software/guix/manual/en/html_node) |
[español](https://www.gnu.org/software/guix/manual/es/html_node) |
[français](https://www.gnu.org/software/guix/manual/fr/html_node).

207 208
# Joining

209
Read about on-going Guix-HPC developments [on our blog](/blog).
210

211
[Guix-HPC](/about)
212
and [GNU Guix](https://www.gnu.org/software/guix/) are collaborative
213
efforts.  You are welcome to [join](/about)!