Add the slurm virtual cluster to the CI

This MR adds steps for testing the slurm schedulers in the pipeline including slurm slurm-semiglobal and slurm-openmpi.

This is done by adding a new lxd runner that sits on Maiko. The creation of the cluster and the activation of the runner is semi-automated but still needs a user to manually register the runner with gitla.inria.fr. All instructions for this process and source files are added in a new repository located here.

• setup new virtual cluster with instructions
• add new lxd runner accessible by gitlab CI
• create new CI stages employing LXD for slurm, slurm-semiglobal and slurm-openmpi
• create a virtual environment that can be reused to speed up CI stages
• create instructions for rebuilding the cluster and adding the runner
• ensure fault tolerance is tested in slurm
• add an update script to the virtual-cluster-ci repository to make updating dependencies easier than rebuilding the entire cluster.
• test new iterative stats checkpointing

marked the checklist item setup new virtual cluster with instructions as completed

marked the checklist item add new lxd runner accessible by gitlab CI as completed

marked the checklist item create new CI stages employing LXD for slurm, slurm-semiglobal and slurm-openmpi as completed

marked the checklist item create instructions for rebuilding the cluster and adding the runner as completed

marked the checklist item create a virtual environment that can be reused to speed up CI stages as completed

marked the checklist item ensure fault tolerance is tested in slurm as completed

added 1 commit

• 075b095c - try to improve ci

Compare with previous version

changed the description

added 1 commit

• 437a469c - revert example files

Compare with previous version

added 1 commit

• 85a75f25 - ci: use vc_slurm.json

Compare with previous version

added 1 commit

• 898b6eca - ci: use vc_slurm.json

Compare with previous version

added 20 commits

• 898b6eca...bf035b87 - 18 commits from branch develop
• 84f84aef - Merge remote-tracking branch 'origin/develop' into slurm-in-ci
• d12d938b - feat: add failing sa server to test checkpointint

Compare with previous version

marked the checklist item add an update script to the virtual-cluster-ci repository to make updating dependencies easier than rebuilding the entire cluster. as completed

added 1 commit

• 55e5fb07 - chore: fix flake8

Compare with previous version

added 1 commit

• 7dfb6745 - ci: fix

Compare with previous version

added 1 commit

• b338551f - ci: fix

Compare with previous version

added 1 commit

• e74f21b6 - make a failiing dl server

Compare with previous version

added 1 commit

• 91191108 - fix: last checkpoint doesnt need a batch number

Compare with previous version

added 1 commit

• 81e4e24e - fix: last checkpoint doesnt need a batch number

Compare with previous version

added 1 commit

• e365ef51 - ci: do sa checkpointing with openmpi

Compare with previous version