README.org 6.78 KB
Newer Older
1
* Guix Jupyter kernel
Ludovic Courtès's avatar
Ludovic Courtès committed
2
3
4

[[https://guix.gnu.org/][GNU Guix]]-enabled kernel to create self-contained, reproducible Jupyter
notebooks.
5

ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
6
7
#+html: <img src="logo.png" />

8
** Status
9

10
#+html: <img src="https://img.shields.io/badge/License-GPLv3+-blue.svg" />
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
11
12
#+html: <img src="https://img.shields.io/badge/Guix-local_package-orange.svg" />
#+html: <img src="https://img.shields.io/badge/System-GNU%2FLinux-green.svg" />
Ludovic Courtès's avatar
Ludovic Courtès committed
13
#+html: <img src="https://archive.softwareheritage.org/badge/origin/https://gitlab.inria.fr/guix-hpc/guix-kernel/">
14

Ludovic Courtès's avatar
Ludovic Courtès committed
15
** Motivation
16

Ludovic Courtès's avatar
Ludovic Courtès committed
17
18
19
20
21
[[https://jupyter.org][Jupyter Notebook]] provides a way for scientists and engineers to share
code along with a narrative providing context and explanations.  It is
no wonder that Jupyter Notebook is now regarded as a tool of choice for
scientists concerned with /reproducible science/ who wish to share their
work.
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
22

Ludovic Courtès's avatar
Ludovic Courtès committed
23
24
25
26
27
28
29
Unfortunately a notebook is a program, and Jupyter leaves it up to you
to set up the right execution environment for the program.  The notebook
itself does not say what its dependencies are.  I wrote my notebook with
Python 3 and SciPy; if you run it on Python 2 or SciPy is missing on
your system, it won’t run; if you have different a different version of
SciPy than the one I have, perhaps it will run, but provide different
results.
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
30

Ludovic Courtès's avatar
Ludovic Courtès committed
31
Guix-Jupyter aims to address this by supporting:
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
32

Ludovic Courtès's avatar
Ludovic Courtès committed
33
34
35
36
  1. Self-contained notebooks that describe their execution environment;
  2. Automatic and reproducible deployment of that environment through
     [[https://guix.gnu.org][GNU Guix]];
  3. Execution of the code in a isolated environment, a “container”.
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
37

Ludovic Courtès's avatar
Ludovic Courtès committed
38
39
40
[[https://hpc.guix.info/blog/2019/10/towards-reproducible-jupyter-notebooks/][Read the initial announcement]] (Oct. 2019) for an in-depth analysis of
the need for “deployment-aware”, self-contained notebooks, and on the
shortcomings of other options.
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
41

Ludovic Courtès's avatar
Ludovic Courtès committed
42
** Usage
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
43

Ludovic Courtès's avatar
Ludovic Courtès committed
44
45
46
47
48
49
Guix-Jupyter is implemented as a /Jupyter kernel/.  Thus, when running
`jupyter notebook`, you will select the “Guix” kernel instead of
selecting, say, the “Python 3” kernel.  This kernel delegates deployment
to Guix itself, so Guix must be installed on your machine and its build
daemon must up and running.  Currently support for Linux unprivileged
user namespaces is also required (see below).
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
50

51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
*** Searching for packages

Before you declare the software environment of your notebook, you might
want to search for packages so you can determine which ones to add to
the environment.  You can do that by entering the following line in a
cell and evaluating it:

#+begin_src scheme
  ;;guix search PATTERN1 PATTERN2
#+end_src

… where PATTERN1 and PATTERN2 and words or regexps to search for.

The result is a (possibly empty) table showing the most relevant
packages for these patterns, including their name, version, and a
synopsis.

Ludovic Courtès's avatar
Ludovic Courtès committed
68
69
70
71
*** Environments

From there on, annotations, aka. /magics/, allow you to create and use
environments.  The general syntax is:
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
72
73

#+BEGIN_SRC scheme
Ludovic Courtès's avatar
Ludovic Courtès committed
74
;;guix environment <env-name> <- <package> …
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
75
76
#+END_SRC

Ludovic Courtès's avatar
Ludovic Courtès committed
77
78
So to create an environment call `python` containing the IPython kernel,
Matplotlib, and NumPy, you would write this in a cell:
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
79

Ludovic Courtès's avatar
Ludovic Courtès committed
80
81
82
#+begin_src scheme
  ;;guix environment my-python <- python-ipykernel python-matplotlib python-numpy
#+end_src
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
83

Ludovic Courtès's avatar
Ludovic Courtès committed
84
85
(Tab completion on package names is supported.)  By default, subsequent
cells will be executed by the IPython kernel in this environment.
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
86

Ludovic Courtès's avatar
Ludovic Courtès committed
87
88
89
You can create as many environments as you like in a notebook.  You can
then switch to a specific environment in a cell by starting it with a
line like this:
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
90

Ludovic Courtès's avatar
Ludovic Courtès committed
91
92
93
#+begin_src scheme
  ;;guix environment <env-name>
#+end_src
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
94

Ludovic Courtès's avatar
Ludovic Courtès committed
95
*** Built-in Guile kernel
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
96

Ludovic Courtès's avatar
Ludovic Courtès committed
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
Guix-Jupyter has a built-in kernel for [[https://gnu.org/software/guile][GNU Guile]] programming.  To use
it, simply create an environment that contains the =guile= executable:

#+begin_src scheme
  ;;guix environment my-guile-env <- guile@2.2 guile-picture-language guile-minikanren
#+end_src

*** Isolation

The kernels select through =;;guix environment= annotations are spawned
/in a container/, isolated from other processes on the machine and from
the network (thus support for Linux /unprivileged user namespaces/ is
required).  This ensures that the notebook results do not depend on
anything but the environments it specifies and the code it contains.
It’s also nice from a security standpoint.

*** Downloads

Since kernels run without access to the network, we have an obvious
problem: how to we populate the environment with things that are not
software packages, such as data files?
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
118

Ludovic Courtès's avatar
Ludovic Courtès committed
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
To address that, Guix-Jupyter provides a =download= magic that can be
used like this:

#+begin_src scheme
  ;;guix download https://ftp.gnu.org/gnu/coreutils/coreutils-8.30.tar.xz e831b3a86091496cdba720411f9748de81507798f6130adeaef872d206e1b057
#+end_src

Here we specify the URL of a file and its expected SHA256 hash.  The
file is downloaded, unless it has already been downloaded before, and is
made available in the current environment.

The download mechanism ensures that the hash matches, again for the sake
of reproducibility.

*** Pinning

It is usually desirable to /pin/ Guix to a specific revision to ensure
that the notebook will keep behaving the same at any point in time,
regardless of changes to the Guix package collection.

To do that, run this magic in a cell:

#+begin_src scheme
  ;;guix pin COMMIT
#+end_src

… where /COMMIT/ is the commit ID (hexadecimal string) of the Guix
commit you want to use.

If you’re unsure what commit to use, run =guix describe= on your machine
or navigate the [[http://data.guix.gnu.org][Guix Data Service]] to find out which Guix revisions
contains the package versions you need.
151

152
** Installation
153
*** Guix
Ludovic Courtès's avatar
Ludovic Courtès committed
154
155
156

This is how you’d build Guix-Jupyter with Guix:

157
158
#+BEGIN_SRC shell
# Clone repo
159
160
$ git clone https://gitlab.inria.fr/guix-hpc/guix-kernel
$ cd guix-kernel
161

162
163
# Build package
$ guix build -f environment.scm
164
165
#+END_SRC

Ludovic Courtès's avatar
Ludovic Courtès committed
166
167
You can install it with:

168
#+BEGIN_SRC shell
169
$ guix package -f environment.scm
170
#+END_SRC
171

Ludovic Courtès's avatar
Ludovic Courtès committed
172
173
To start Jupyter in an environment that contains Guix-Jupyter, run:

174
#+BEGIN_SRC shell
Ludovic Courtès's avatar
Ludovic Courtès committed
175
176
  guix environment --ad-hoc jupyter -l environment.scm guile-git guile \
       -- jupyter notebook
177
#+END_SRC
178

179
** Contributing
Ludovic Courtès's avatar
Ludovic Courtès committed
180
*** Development environment
181
182
#+BEGIN_SRC sh
  guix environment -l environment.scm
183
#+END_SRC
184

185
186
187
188
189
190
191
192
193
194
195
196
*** Using the kernel prior to installation

You can try out the Guix-Jupyter kernel before it’s installed, for
instance by running:

#+begin_src sh
  ./pre-inst-env jupyter notebook --no-browser
#+end_src

This command arranges to set =JUPYTER_PATH= such that the kernel in your
build tree is visible.

Ludovic Courtès's avatar
Ludovic Courtès committed
197
*** Developers
ROUBY Pierre-Antoine's avatar
ROUBY Pierre-Antoine committed
198
199
200
201
Made with *λ* by the *GNU Guix community*.

Based on *[[https://github.com/jerry40/guile-kernel][Guile-kernel]]*.

202
*** Coding style
203
This project try to respect as possible the [[https://www.gnu.org/software/guix/manual/html_node/Coding-Style.html#Coding-Style][Guix coding style]].