Guix packaging for Melissa
Due to the significant efforts made in favor of reproducibility and efficient HPC packaging, it became clear that a complex framework like Melissa will sooner or later need appropriate packaging/building tools to support its diffusion among the HPC community. After a successful experience with Spack and Nix, Guix was finally investigated. This post condenses lessons and basic features learned when deriving Guix recipes for Melissa.
Initially thought as a short tutorial, this document is organized as follows:
- GUIX packaging for Melissa
edit (07/27/23): We integrated some recent changes
- A guix-packages repos has been setup with all package files discussed here (https://gitlab.inria.fr/melissa/guix-packages/-/tree/master/)
- This repo also enable to setup a melissa channel (see README.md for more details)
- The melissa packages have been updated to avoid code duplication by using the 'inherit' feature. As of today the 'melissa-api' is the base package 'py-melissa-core' and 'heat-pde' inherit from. Definition of repo source is only done in 'melisssa-api'
- We added a package for iterative-statistics
edit (06/27/23): the recent addition of the iterative-stats library to the Python requirements of Melissa is likely to make the Guix recipe for the Python components of Melissa section outdated. To solve this issue, the user can either build the SC2023 tagged version of Melissa which does not have this dependency or build a Guix recipe for the iterative-stats library and update the py-melissa-core recipe.
Quick introduction to Guix
As explained on the software website:
GNU Guix provides state-of-the-art package management features such as transactional upgrades and roll-backs, reproducible build environments, unprivileged package management, and per-user profiles. It uses low-level mechanisms from the Nix package manager, but packages are defined as native Guile modules, using extensions to the Scheme language—which makes it nicely hackable.
It comes with a reference manual in english and french as well as with a cookbook also available in both english and french.
In addition, Guix-HPC was introduced with the following objective:
Guix-HPC is an effort to optimize GNU Guix for reproducible scientific workflows in high-performance computing (HPC). Our introductory article explains how we got started and gives an overview of what we want to achieve.
Note: the Guix-HPC website gives a list of clusters it is deployed on as well as several contact means for newcomers.
Finally, a very nice introduction to Guix is also available on gricad documentation.
Setting up Guix on grid5000
Guidelines to install Guix locally on top of a Linux distribution are available on the main website or on system crafters. This may come in handy when debugging recipes but it is not mandatory when a Guix provided platform is accessible. On grid5000 for instance, Guix is available on frontends and nodes.
On any grid5000 site, the user can see that Guix environment variables are automatically set:
$ echo $GUIX_PROFILE
> /home/<username>/.config/guix/current
However, the current
directory is only created after the distribution is updated for the first time i.e. by running:
$ guix pull
It is usually advised to proceed with:
$ guix package -u
> guix package: warning: Consider running 'guix pull' followed by
'guix package -u' to get up-to-date packages and security updates.
guix package: warning: nothing to do
which is a very important command to update installed packages.
As evident from the output message, in the context of a first use of Guix, nothing happens as no package is currently installed in the user profile:
$ guix package -I
>
Note that guix package
is a paramount command (see here):
$ guix package -h
> Usage: guix package [OPTION]...
Install, remove, or upgrade packages in a single transaction.
-i, --install PACKAGE ...
install PACKAGEs
-e, --install-from-expression=EXP
install the package EXP evaluates to
-f, --install-from-file=FILE
install the package that the code within FILE
evaluates to
-r, --remove PACKAGE ...
remove PACKAGEs
...
-h, --help display this help and exit
-V, --version display version information and exit
Report bugs to: bug-guix@gnu.org.
GNU Guix home page: <https://guix.gnu.org>
General help using Guix and GNU software: <https://guix.gnu.org/en/help/>
and the main commands that we will be using in this tutorial are actually aliases of it.
Guix recipe for Melissa API
With Guix, packages are designed as first-class objects implemented in Scheme language. As is often the case, the recipes introduced hereafter were adapted from reference examples (see here and there) but an introduction to Scheme is provided in Guix cookbook in case the reader wants to dig this aspect deeper.
Here is a recipe for the Melissa API in a file melissa-api.scm
:
https://gitlab.inria.fr/melissa/guix-packages/-/blob/master/packages/melissa-api.scm
At a first glance, this looks like a pretty standard recipe structured in two major blocks:
- the first one that lists the necessary
guix modules
already available in the store, - the second one that defines the new package itself:
- the name, version and where/how to fetch it,
- the build system it relies on, its dependencies and how to build it,
- some general information (e.g. home-page, description, license type).
Looking at the very first line of each block, one notices that this recipe consists in a public module which means that it will be installable from anywhere and that it will be referable as a dependency of other packages (see sections 2.2.1 and 2.1.2 of the cookbook) provided that the file is dully placed a the channel folder.
For now for testing, we can work without channel. Just install melissa-api package with:
$ guix package -f melissa-api.scm
You can check that the melissa-api package (with all its dependencies) is installed using:
guix package --list-installed
A handy command when developing is to use this file to have guix install just melissa-api dependencies, not melissa-api itself:
guix shell --check -D -f melissa-api.scm guix
For properly installing a Melissa channel see: https://gitlab.inria.fr/melissa/guix-packages
Examining this recipe in more details, three new questions must be addressed:
- How to correctly specify the package dependencies?
- How to get the package
sha-256
? - What is the
arguments
command for?
Package dependency specification
Melissa API relies on CMake
and the CMakeLists.txt file requiring:
-
gcc
/gfortran
OpenMPI
ZeroMQ
pkg-config
The build system CMake
must be made available to the package via #:use-module (guix build-system cmake)
and (build-system cmake-build-system)
.
On the other hand, the dependencies are listed as a list of inputs in (inputs (list openmpi zeromq gcc-toolchain gfortran-toolchain pkg-config))
and made available to the package thanks to their corresponding module import:
#:use-module (gnu packages mpi)
#:use-module (gnu packages pkg-config)
#:use-module (gnu packages commencement)
#:use-module (gnu packages networking)
To find the appropriate module available in the store, the user can query this information with the search
command For ZeroMQ
for instance:
$ guix search zeromq
> name: zeromq
version: 4.3.4
outputs:
+ out: everything
systems: x86_64-linux i686-linux
dependencies:
location: gnu/packages/networking.scm:1349:2
homepage: https://zeromq.org
license: LGPL 3+
synopsis: Library for message-based applications
description: The 0MQ lightweight messaging kernel is a library which extends the standard socket
+ interfaces with features traditionally provided by specialized messaging middle-ware products. 0MQ
+ sockets provide an abstraction of asynchronous message queues, multiple messaging patterns, message
+ filtering (subscriptions), seamless access to multiple transport protocols and more.
relevance: 30
Guix indicates that the associated module name is networking
. In the same way, we find that the module associated to OpenMPI
is named mpi
.
Note: both gcc-toolchain
and gfortran-toolchain
are imported from commencement
.
If a dependency is incorrectly specified, the user will be prompted with an error message. For instance if python
is listed amongst the inputs but not amongst the modules, Guix will raise an error like this:
...
ice-9/boot-9.scm:1685:16: In procedure raise-exception:
error: python: unbound variable
Package sha-256
Getting the right sha-256
is always a lottery game (same goes for Nix
). The easiest way to get it right is by first providing an absurd value which will trigger the following message:
...
\r:sha256 hash mismatch for /gnu/store/dbn5022l5ksx1756x205wzrn9k0g5qka-git-checkout:
expected hash: 0000000000000000000000000000000000000000000000000000
actual hash: 1zv36r8gbb92v9s6vvzgdvv8ryli6a51b0i3g61k1rr6l3bvh6g3
hash mismatch for store item '/gnu/store/dbn5022l5ksx1756x205wzrn9k0g5qka-git-checkout'
...
this indicates that 1zv36r8gbb92v9s6vvzgdvv8ryli6a51b0i3g61k1rr6l3bvh6g3
is the right hash.
Passing arguments to the build system
As explained on Guix documentation about build systems:
Each package definition specifies a build system and arguments for that build system (see Defining Packages). This build-system field represents the build procedure of the package, as well as implicit dependencies of that build procedure.
Similarly to what is done for a gnu-build-system
, for a cmake-build-system
which inherits from the latter (see cmake-build-system sources), the following command sequence will be performed at build:
unpack && ./configure && make && make check && make install
For Melissa API however, there is no check
phase which is why trying to follow such sequence won't work. In order to turn the check
phase OFF, the #:tests? #f
argument must be introduced.
Note: this is an error very hard to track on grid5000 as the build log is not accessible when the build is handled by the Guix daemon on a dedicated server. By being attentive to the terminal, the user may still be able to spot the failing phase of the build.
At this point, the user should have melissa-api
available amongst their packages:
melissa-api 1.0 out /gnu/store/<some-character-sequence>-melissa-api-1.0
Guix recipe for the Python components of Melissa
Similarly to the way Melissa packaging was handled with Spack, a separate recipe is now designed to build the Python components of Melissa.
Given how volatile the Python dependencies can be (lots of them are required for specific features), it would be possible to build many variants for Melissa e.g. for sensitivity analysis, for deep-learning either with torch
or tensorflow
, etc. To keep things as simple as possible, this part will only focus on how to build a functional version of Melissa for sensitivity-analysis.
Here is what the recipe looks like: https://gitlab.inria.fr/melissa/guix-packages/-/blob/master/packages/py-melissa-core.scm
Note that this time, in addition to the test phase which relies on extra-dependencies, the sanity check phase also needs to be removed for the package to successfully build:
#:phases (modify-phases %standard-phases (delete 'sanity-check))))
The user should now be able to call the Melissa executable commands:
$ melissa-launcher -h
> /gnu/store/rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16/bin/bash:
warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
$! ---------------------------------------------- $!
__ __ ______ _ _____ _____ _____
| \/ | ____| | |_ _|/ ____/ ____| /\
| \ / | |__ | | | | | (___| (___ / \
| |\/| | __| | | | | \___ \\___ \ / /\ \
| | | | |____| |____ _| |_ ____) |___) / ____ \
|_| |_|______|______|_____|_____/_____/_/ \_\
$! ---------------------------------------------- $!
usage: melissa-launcher [-h] [--config_name CONFIG_NAME] [--version] [--print-options]
Melissa Launcher
options:
-h, --help show this help message and exit
--config_name CONFIG_NAME, -c CONFIG_NAME
User defined configuration file. Path can be relative or absolute.
--version, -v show the Melissa version
--print-options Show the available configuration options
In order to run the heat-pde sensitivity analysis, the next objective is to compile the heat-pde executable in the right environment and with the proper dependencies.
Guix recipe for the heat-pde example
The heat-pde use-case relies on the executables
folder whose sources are compiled with CMake
.
As opposed to Spack which makes built packages available by loading them, with Guix, a built package lies in the store and automatic adjustments to the PATH
makes executables available. For example following this tutorial until this point should yield the following:
$ guix package -I
> melissa-api 1.0 out /gnu/store/<some-character-sequence>-melissa-api-1.0
py-melissa-core 1.0 out /gnu/store/<some-character-sequence>-py-melissa-core-1.0
$ echo $PATH
> /home/<user-name>/.config/guix/current/bin:/home/<user-name>/.guix-profile/bin:/usr/local/bin:/usr/bin:/bin:/grid5000/code/bin:/opt/puppetlabs/bin
$ ls /home/<user-name>/.guix-profile/bin
> melissa-launcher melissa-monitor melissa-server melissa_set_env.sh
Note: with Guix, the melissa_set_env.sh
script generated when building melissa-api
is useless and will never be sourced.
To compile the heat-pde sources however, one needs to adjust LD_LIBRARY_PATH
which is the point of the melissa_set_env.sh
script when Melissa is built manually. In addition, the modules used to compile the API must be consistent with the ones used to compile the heat-pde
sources. To meet both requirements, the solution is to build a new recipe for the heat-pde use-case with melissa-api
as input. Here is the resulting recipe heat-pde.scm
: https://gitlab.inria.fr/melissa/guix-packages/-/blob/master/packages/heat-pde.scm
This time, the unpack phase (see Build Phases) was preceded by a custom phase to move inside the executables
folder before build.
Note: because CMakeLists.txt
wrongfully requires Python (this should be fixed in the future), it was added amongst the inputs.
As expected, the executables are now available in /home/<user-name>/.guix-profile/bin
:
ls /home/<user-name>/.guix-profile/bin
> heatc heatf melissa-launcher melissa-monitor melissa-server melissa_set_env.sh
To start a study the user can clone Melissa repository and move to the heat-pde-sa
use-case:
$ git clone https://gitlab.inria.fr/melissa/melissa.git
$ cd melissa/examples/heat-pde/heat-pde-sa
The config_mpi.json
can be modified to fix the path to the executable:
"client_config": {
// FIXME: the executable command needs to be replaced with the appropriate path
"executable_command": "heatc 100 100 100",
...
}
The study can then be launched on any grid5000 machine:
$ oarsub -I -l host=1
$ melissa-launcher -c config_mpi
Using Guix shell to compile sources manually
Instead of building a recipe, the user could have chosen to use the Guix shell command to compile the sources manually:
$ git clone https://gitlab.inria.fr/melissa/melissa.git
$ guix shell --pure cmake make melissa-api openmpi gcc-toolchain gfortran-toolchain python pkg-config
$ cd melissa/examples/heat-pde/executables
$ mkdir && cd build
$ cmake ..
$ make
$ exit # to leave Guix shell
Note: in this case of figure, executable_command
in config_mpi.json
must be adjusted accordingly to the executable path.
A surprisingly successful execution
Taking a step back to what we have done so far, we have:
- built the api by installing
melissa-api
- built the Melissa executables by installing
py-melissa-core
- built the heat-pde executables by installing
heat-pde
.
For 1 and 3 however, we have made sure to use the OpenMPI
provided by Guix-HPC (currently openmpi@4.1.5
). Running a study in the default grid5000 environment will execute all jobs with the default OpenMPI
(currently openmpi@4.1.0
). This breaks the consistence between the heat-pde executable and the OpenMPI
version. Nevertheless, due to OpenMPI
flexibility, the study still manages to go through (probably in suboptimal conditions though). Another way to go would be to identify the path to the appropriate mpirun
binary and use it instead of the default one.
To do so, the path to mpirun
can be identified either by checking CMakeCache.txt
if heat-pde was built manually from guix shell
or proceed as follows:
$ guix search openmpi
> name: openmpi
version: 4.1.5
outputs:
...
$ ls /gnu/store/ | grep openmpi-4.1.5
> 35sy493l8569sfcqxmsddq99xz672ily-openmpi-4.1.5.drv
am9igjyrd41k5bj6v52n8zbs3cjx1qnd-openmpi-4.1.5.drv
bfa8dijdr905n5rhjbxm04lyl6apw3vl-openmpi-4.1.5.tar.bz2.drv
dqbqn4zbbf2x9y3dar1z5rlzv0d9pvlk-openmpi-4.1.5-builder
hdls97c5xb4y3s4k00wfq8ih8xzk91ri-openmpi-4.1.5.tar.xz.drv
l2cb7yiarzspxf32c4hgfwclw8wwlnhb-openmpi-4.1.5.drv
p84w0m7qi3w768gvvvy68dfqlx5y2mmx-openmpi-4.1.5
q2jy1yiq1fxyj9sdqginmwmlhbzrjc4p-openmpi-4.1.5.drv
v5l5b1viqyzzhdhy2jd99n1sq3cich88-openmpi-4.1.5.tar.xz-builder
w7za1rg0ssxbhkq1v32pakibd12wx5ws-openmpi-4.1.5-builder
zwhkfjgym8q5l0kv65r5v8bndlg3v120-openmpi-4.1.5
Then, any binary amongst one of the two candidates can be specified in config_mpi.json
:
"launcher_config": {
...
"scheduler_client_command": "/gnu/store/p84w0m7qi3w768gvvvy68dfqlx5y2mmx-openmpi-4.1.5/bin/mpirun",
"scheduler_server_command": "/gnu/store/p84w0m7qi3w768gvvvy68dfqlx5y2mmx-openmpi-4.1.5/bin/mpirun"
}
Acknowledgements and final comments
In comparison to Spack, Guix seems like a much more subtle packager which is definitely harder to take in hand. Indeed, the documentation and the error messages are not always easy to decipher which is why it is important to have people to turn to. I personally had the chance to get in touch with Pierre-Antoine Bouttier who has been particularly helpful but as mentioned in the first section of this document, other means exist to ask questions to Guix experts.
Finally, for simplicity's sake, this tutorial focused on the sensitivity analysis variant of Melissa. In real conditions however, a user may be interested in the deep learning flavour of Melissa. With Spack, variants were defined for the deep learning server (with torch
or tensorflow
). The same could be done with Guix which provides an easy way to define package variants by inheriting from the base one.
Regarding Nix, recipes designed by Jonathan Bleuzen and Adrien Faure already exist for the API (see melissa), the Python components (see melissa-launcher) and the heat-pde use-case (see melissa-heat-pde). Although these are not up to date with the last Melissa version, those recipes provide a good entry point into the Nix logic.