selecting the right CUDA module depending on the node
Some old sirocco nodes have old GPUs that don't work with CUDA12. If you load the CUDA 12 module there, it will fail to work with the CUDA 11 driver. An easy way to load the appropriate cuda module is:
cudaversion=$(nvidia-smi -q | grep CUDA | awk {'print $4'})
module load compiler/cuda/$cudaversion
Should be in the doc but I don't know where.