From 82374c5b7ebef4d6c0efa8c708ec38efac8145da Mon Sep 17 00:00:00 2001 From: Fernando Ayats <ayatsfer@gmail.com> Date: Tue, 17 Jun 2025 09:49:30 +0200 Subject: [PATCH] Finish binary cache post --- docs/content/hpc-env/bin-spack/index.md | 69 +++++++++++++++++++++++-- 1 file changed, 66 insertions(+), 3 deletions(-) diff --git a/docs/content/hpc-env/bin-spack/index.md b/docs/content/hpc-env/bin-spack/index.md index 640f9af..a334a7c 100644 --- a/docs/content/hpc-env/bin-spack/index.md +++ b/docs/content/hpc-env/bin-spack/index.md @@ -1,8 +1,8 @@ --- title: Binary deployments with Spack hidden: false -draft: true -date: 2025-06-13 +draft: fase +date: 2025-06-17 --- Spack (https://spack.io) is a package manager for Linux, aimed at making it easy to deploy scientific software to computers. @@ -220,4 +220,67 @@ operation you can do is to **shrink** a string. By using `padded_length`, Spack artificially create paths with a number of characters of padding, such that if the number is big enough, we can assume that they will be always shrank. For example, it will install packages to `/home/runner/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/bin/cmake` -instead (I simplified the actual installation paths for clarity). \ No newline at end of file +instead (I simplified the actual installation paths for clarity). + +Regarding the `spack.lock` file, to be able to have users download easily from the cache, I +would recommend committing and sharing the `spack.lock` with users of the cache. The two +possibilites are: + +- CI builds some packages, but the `spack.lock` is not preserved: when users concretize the environment + on their ownm they may get the same versions and hashes that are actually cached, or may not, + as a `spack.yaml` doesn't guarantee reproducibility. + +- CI builds some packages, and pushes back the `spack.lock` into the environment: users won't have + to re-concretize the environment, so they will get the hashes for the packages that were built + and cached by CI. + +To preserve the `spack.lock` from CI, your caching workflow might look like the following: + +1. Setup Spack. +2. Concretize the environment, rewriting the `spack.lock` if needed. +3. Build and push the environment. +4. Commit and push the `spack.lock` back into the repository. + + +--- + +Finally, when populating a Spack binary cache, it is important to take into consideration the Linux +distribution to target. As Spack doesn't bootstrap itself from glibc, it actually links its packages +to the system's glibc. This adds an implicit dependency on the system's, and is reflected on the +package spec. For example, if I concretize some package: + +``` +$ spack spec -L cmake | grep cmake + - gq5bqyhsvdlqs6qqjvs6useiq6kzhakf cmake@3.31.6~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-ubuntu24.04-icelake %c,cxx=gcc@13.3.0 +``` + +This one is concretized for `ubuntu24.04`. Therefore, if you want to deploy pre-built Spack packages +for some OS like Debian 11, you have to make sure to concretize and build the environment on Debian 11 +as well. + +This can be easily done with a container, and both GitHub and GitLab provide means of running some action +steps under a container: + +```yaml +# github workflow +jobs: + main: + container: debian:11 + #... +``` + +```yaml +# gitlab pipeline +main: + image: debian:11 + # ... +``` + +## Need more help? + +If you need more help with creating your Spack binary cache to accelerate +deployments on HPC centers, please get in contact with us at NumPEx WP3: + +- Mailing list: numpex-pc5-wp3@inria.fr +- Fernando Ayats (Author of the post): fernando.ayats-llamas@inria.fr + -- GitLab