Commit 301b84b8 authored by GILLES Sebastien's avatar GILLES Sebastien
Browse files

#1715 Refactor and rewrite in large parts the documentation about CI...

#1715 Refactor and rewrite in large parts the documentation about CI (especially considering it has changed much since the documentation was first written, with for instance the shared runners now fully functional).

Also overhaul and document better the way to neutralize some CI steps.
parent 89891cf9
# How to add new members to both MoReREM CI projects?
## Connect on ci.inria.fr
Admistrator login is m3disim-ci@inria.fr
Password is the usual administration password of the team followed by the two characters ';_' (usual one was not accepted by the interface).
## Dashboard
* Click on _Dashboard_; you should see two projects __MoReREM__ and __MoReFEM-gitlab-ci__.
* Click on _Manage project_ on one of them.
## Add members
* Click on _Add members_, and then specify user's email in the block (this email must already been registered on the platform).
* Refresh the page until the email address is modified with (dot) and (at) instead of . and @.
* Click on _Slave Admin_.
* And do the exact same procedure for the second project.
<!-- toc -->
- [Introduction](#introduction)
- [Basic set up of the VM](#basic-set-up-of-the-vm)
* [VM choice on CI platform](#vm-choice-on-ci-platform)
* [Update Ubuntu](#update-ubuntu)
* [Connect the additional disk](#connect-the-additional-disk)
* [Install and set-up Docker](#install-and-set-up-docker)
* [Install and set-up gitlab-runner](#install-and-set-up-gitlab-runner)
* [Fix communication error](#fix-communication-error)
<!-- tocstop -->
# Introduction
The given procedure here explains how to set up a basic virtual machine on which a Docker executor may be installed.
Currently 6 such VMs are created on the [CI platform](https://ci.inria.fr) provided by Inria; at some point a Docker farm may be deployed in which case we won't need anymore to set up our own machines to run Docker executors.
# Basic set up of the VM
## VM choice on CI platform
The choice is made to take the most beefy configuration provided by the CI service: 4 processors with 12 Go RAM (FYI the immediately inferior one is 1 processor with 8 Go) with an additional 100 Go storage (all Linux VMs are provided only 20 Go so we would reach it in the very first use of the executor).
I chose to use Ubuntu 16.04 as system, but as the code itself will be run into a Docker container it doesn't matter much (I found Docker installation is slightly easier in Ubuntu than in Fedora).
## Update Ubuntu
Once the VM is created, log onto it and update it.
````
sudo apt update
sudo apt upgrade
````
## Connect the additional disk
````
sudo fdisk /dev/vda
````
Choose 'n', 'p', validate twice, and 'w'.
Then type:
````
sudo mkfs -t ext4 /dev/vda
sudo mkdir /media/suppl_drive
sudo nano -Bw /etc/fstab
````
In this file, add line:
``
/dev/vda /media/suppl_drive ext4 defaults 0 2
``
Then type:
````
sudo mount -a
sudo chown -R ci:ci /media/suppl_drive
````
## Install and set-up Docker
````
sudo apt install docker.io
````
Remove the default Docker directory and create a new one on the additional disk:
````
sudo rm -rf /var/lib/docker
cd /var/lib
mkdir /media/suppl_drive/docker
sudo ln -s /media/suppl_drive/docker docker
sudo systemctl restart docker
sudo systemctl enable docker
````
## Install and set-up gitlab-runner
````
sudo wget -O /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
sudo chmod +x /usr/local/bin/gitlab-runner
mkdir /media/suppl_drive/gitlab-runner
sudo gitlab-runner install --user=ci --working-directory=/media/suppl_drive/gitlab-runner
sudo gitlab-runner start
````
Once this is done, you may register your runner through:
````
sudo gitlab-runner register
````
(you may have a look at the [official documentation](https://docs.gitlab.com/runner/register/index.html)).
## Fix communication error
There is currently an issue with Inria configuration, so you need to add an option in config.toml file to make things run smoothly (see [here](https://gitlab.inria.fr/siteadmin/doc/wikis/faq#using-a-docker-executor) for the explanation).
````
sudo nano /etc/gitlab-runner/config.toml
````
Add ``network_mode = "host"`` in _[runners.docker]_ section.
[TOC]
# Introduction
<!-- toc -->
MoReFEM uses up Gitlab-CI for its continuous integration (shortened as 'CI'): the highlight of this solution is that the setup is very easy to put on motion and the git repository versions directly the Yaml files used to run the CI.
- [Introduction](#introduction)
- [Set up the Yaml file to use](#set-up-the-yaml-file-to-use)
- [Runners in Gitlab interface](#runners-in-gitlab-interface)
* [Runner with docker executor](#runner-with-docker-executor)
+ [Introduction](#introduction-1)
+ [Setting up your Docker executors](#setting-up-your-docker-executors)
- [Activate specific runner](#activate-specific-runner)
- [Create new Docker runners on MoReFEM VMs](#create-new-docker-runners-on-morefem-vms)
* [Example: on VM Docker1](#example-on-vm-docker1)
* [The 5 others VMs](#the-5-others-vms)
* [Runner with shell executor (on macOS)](#runner-with-shell-executor-on-macos)
The configuration files related to CI are all put in the `ExternalTools/Gitlab-CI` directory - I didn't respect the usual convention to put a `gitlab-ci.yml` file at the root of the repository, as I used several files to gain in clarity. This adds an additional step in setting up Gitlab-CI for your own fork - see dedicated section below.
<!-- tocstop -->
# Installation
# Introduction
CI is properly set for the [main MoReFEM project](https://gitlab.inria.fr/MoReFEM/CoreLibrary/MoReFEM).
Gitlab-CI relies on a relatively simple configuration:
If you want to do so on a fork of the project, you have three steps to follow, two of which are straightforward.
- A Yaml file is provided within the Git repository, and is used by Gitlab to describe the continuous integration tasks.
- Runners which take up the jobs described in the Yaml and run them.
## Setting up the Yaml file in Gitlab
Gitlab offers a way to share runners between several projects... unfortunately currently not activated in the Inria instance.
As we do not use the conventional `.gitlab-ci.yml` file, we need to tell Gitlab where is the file to use.
So I have defined very basic VMs (with the procedure described [here](Internal/SetUpVMForDockerExecutor.md)) onto which you may define your own runners.
Go in your fork's `Settings > CI/CD > General pipelines` and choose `ExternalTools/Gitlab-CI/gitlab-ci.yml` in the field `CI/CD configuration file`.
As you shall see, the procedure is rather fast: you should have 7 (Docker) runners defined in around 10 minutes
If `CI/CD` doesn't appear in your interface, go to `Settings > General > Visibility, project features, permissions` and check `CI/CD` is properly enabled.
# Set up the Yaml file to use
If you don't want to bother with macOS runners, `ExternalTools/Gitlab-CI/gitlab-ci-no-macos.yml` instead.
By default, if CI is activated for your projects Gitlab assumes it is at the root of the projet and is named .gitlab-ci.yml. In MoReFEM I chose to put it elsewhere (so that they appear more explicitly in the project and several may be defined) so you need to specify the path:
* Go in _Settings_ / _CI/CD_ / _General pipelines_.
* In _Custom CI config path_, choose the one you wish.
## Setting up shared runners (to run Docker containers)
The most usual is _ExternalTools/Gitlab-CI/gitlab-ci.yml_, but if you do not have access to a macOS machine _ExternalTools/Gitlab-CI/gitlab-ci-no-macos.yml_ is more fitting.
Go in your fork's `Settings > CI/CD > Runners` and ensure `Enable shared runners for this project` is enabled.
# Runners in Gitlab interface
One it's done, you are able to run all the CI jobs that are using up Docker containers.
The most usual runner choices are _docker_ and _shell_. I will describe briefly both of them: Docker ones will be useful for our Linux builds, and shell ones for macOS ones.
Using these shared runners is much more easier that former procedure that required to deploy VM and run personal runners on them (the procedure was dropped from MoReFEM tutorial in May 2022).
## Runner with docker executor
## Setting up macOS (shell) runners
### Introduction
You need to define a shell runner on a macOS machine (in M3DISIM team typically Atlantis and Nostromo MacPro are used to do so).
The idea of a docker executor is to run the job inside a Docker image.
The complete procedure to do so from scratch is detailed [here](SetUpmacOSVM.md); you need to dispose of the administration rights to be able to do all of them.
This is a very appealing idea because:
If you're from M3DISIM team, the best is to ask the credentials for the CI-MoReFEM account and register your runners on Atlantis and Nostromo with this account (in which case you may go directly to [the registration of the runner procedure](SetUpmacOSVM.md#as-ci-user) and don't have to update yourself the third party dependencies).
- Maintainance is much easier: you need to just maintain few Docker images that are then deployed on as many machines as you wish.
- Ressources may be used more efficiently: a same machine may be used for thoroughly different projects. There is an experimentation at Inria to provide many machines able to run such Docker executors.
There are however few drawback a well:
# Content of MoReFEM CI
- Debugging is harder: if the log doesn't give you the reason of an eventual failure, you can't easily (or at least I didn't manage to) grab the container to play with it. The official documentation [alludes briefly](https://docs.gitlab.com/runner/executors/) to this issue but doesn't provide more explanation. You may of course run locally in Docker what was run remotely during CI to play with it and understand the issue.
- Docker is really tightly coupled to Linux and you can't run macOS (or Windows but we don't support it anyway in MoReFEM) images.
MoReFEM CI is composed of many jobs (more than twenty of them...) but not all of them are built each time. We will detail each target and when they are intended to be run (or course in the end it is the yaml files which are right if there is a discrepancy...)
The base images used in MoReFEM are available in [ThirdPartyCompilationFactory registry](https://gitlab.inria.fr/MoReFEM/ThirdPartyCompilationFactory/container_registry).
Please notice that the `master` branch of the main project gets a very special status and don't run much tests: the idea is that only the integration manager should push toward it and does so only when the `develop` branch (which on the other hand runs plenty of tests) is completely validated.
## Stages build_and_test and check_warnings
### Setting up your Docker executors
Those two first stages are the ones that are most often run: basically they are at each push toward gitlab.
In your own fork of MoReFEM (you need to get a high enough status to be able to reach settings of a project, and for your fork you obviously have it!), go in the __Settings__ tab, choose __CI/CD__ and then __Runners__.
### build_and_test
#### Activate specific runner
There are two different types of jobs here:
There is an on-going test to provide a Docker farm at Inria; so far three machines are provided in the experimentation phase:
- Compilation ones
- Doxygen ones
![SpecificRunners](Images/SpecificRunners.png)
#### Compilation and tests
Choose _INRIA QLF-CI_ large which is provides enough horsepower to run our job in a reasonable time.
The library is compiled on three differents OS (macOS, Ubuntu and Fedora) with different configurations:
#### Create new Docker runners on MoReFEM VMs
- clang, gcc or Apple Clang as compilers.
- Static or shared library.
- Debug or release mode.
- MoReFEM compiled as one big library or 10 smaller ones.
While the Docker farm is not available, we need to define our very own VM to run the tasks, which is a waste (we are sitting over Go of RAM that are used only once in a while).
Of course all combinations aren't tested, but currently 8 of them are nonetheless.
First make sure your [SSH credentials](ssh.md) are properly registered.
If you want to skip entirely this stage, use **noci** or **nobuild** in the name of the branch pushed to Gitlab.
You also need to make a request to _M3DISIM-CI@inria.fr_ to be added as members to the MoReFEM CI projects; this alias encompasses currently Philippe Moireau, Dominique Chapelle and Sébastien Gilles. The procedure to add new members is described [here](Internal/AddMembersToCIProjects.md) (might be useful as Sébastien is currently the only one aware of it).
#### Doxygen
So connect on each of them and set-up your own runner:
The three levels of documentation (see [here](../../Doxygen/README.md#levels-of-infirmation)) are generated.
##### Example: on VM Docker1
### check_warnings
````
ssh ci@docker1.ci
````
The purpose of this second stage is to sift through the logs of the jobs of the first stage to identify whether there are warnings (be it from compilation or from Doxygen).
(ask the password at m3disim-ci@inria.fr).
There is the same number of jobs as in first stage; the analysis is made by custom scripts that are in `Scripts/Tools ` directory (respectively `find_warning_in_compilation_log.py` and `find_warning_in_doxygen_log.py`).
On this machine:
These jobs don't result on complete failure (the tests are in orange in this case) but they must pass completely before being integrated in a full release.
````
sudo gitlab-runner register
````
It should be noticed that Gitlab has evolved and it is now possible to start a job in the second stage without having to run all the jobs in first stage (thanks to the `needs` instruction in Yaml files).
If you want to skip entirely this stage, use **noci** or **nodox** in the name of the branch pushed to Gitlab.
## Stage Valgrind
At this stage, all the model instances are run with Valgrind to check there are no memory leaks (or more precisely that there are no memory leaks that are knowingly related to the third party library used - see [this page](../../../ExternalTools/Valgrind/README.md) for more details about the command used).
The base Docker image used here is generated using the dedicated [AnalysisTools](https://gitlab.inria.fr/MoReFEM/analysistools) project; the idea here is just to add Valgrind to one of the image used in previous stage (namely the Fedora image with clang compiler).
This stage is intended to be run much less often; one of the three conditions must be met:
- The branch pushed is `develop` on the main project (thus concerns only the integration manager)
- The branch pushed includes **valgrind** in its name.
- The branch pushed includes **full_ci** in its name.
## Stages analysis and Sonarqube
These stages are akin to Valgrind one and are run only when one of the three conditions below is met:
- The branch pushed is `develop` on the main project (thus concerns only the integration manager)
- The branch pushed includes **sonarqube** in its name.
- The branch pushed includes **full_ci** in its name.
Then choose:
Analysis stage runs some analysis tools using specific Docker images created in the [AnalysisTools](https://gitlab.inria.fr/MoReFEM/analysistools) project; the results from these tools are then aggregated by Sonarqube.
- Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
These steps may be very long: cpp checks runs over two hours and sonar-scanner takes at least half an hour.
Copy/paste password the choice on ![Gitlab page](Images/RunnerCredential.png).
- Please enter the gitlab-ci token for this runner:
The result is published on Inria Sonarqube instance at [this link](https://sonarqube.inria.fr/sonarqube/dashboard?id=m3disim%3Amorefem%3Adevelop) (please don't bother about the branch name: I started working on this branch without realizing it couldn't be renamed later...)
The token given in Runner page (see image above).
- Please enter the gitlab-ci description for this runner:
You may just repeat the name of the machine... or really put whatever you want (you may change it afterwards anyway).
- Please enter the gitlab-ci tags for this runner (comma separated):
Skip that: we don't use tags so far.
- Please enter the executor:
Choose _docker_.
- Please enter the default Docker image
Type ``alpine``
Your runner should appear shortly on ther gitlab runner pages with a green dot.
## (Former) stage Linter
Now there is currently an issue with Inria configuration, so you need to add an option in config.toml file to make things run smoothly (see [here](https://gitlab.inria.fr/siteadmin/doc/wikis/faq#using-a-docker-executor) for the explanation).
This stage... no longer exists. I indicate it here to explain why.
Two jobs were associated to it: one to run include what you use, and the other to run clang-format.
Unfortunately, it proved to be very unwieldy and time consuming for the integration manager: even when ensuring the version used for these softwares were the same, the output did not match exactly all the time.
The jobs have therefore be removed and both those softwares are run locally by the integration manager at least once before each release.
If you want to run them at some point:
### Include-what-you-use
A README explain how to work with it [here](../../../ExternalTools/IncludeWhatYouUse/README.md).
You may force some behaviour with comments that look like:
````
sudo nano /etc/gitlab-runner/config.toml
#include <cstddef> // IWYU pragma: keep
````
and add ``network_mode = "host"`` in _[runners.docker]_ section of your runner.
There are other commands than `keep`; check the [IWYU documentation](https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/IWYUPragmas.md) to find out existing pragmas.
### clang-format
Make sure clang-format is installed on your system, and then run from root directory:
##### The 5 others VMs
````
python Scripts/Tools/run_clang_format.py
````
(preferably starting from a clean git status to be able to check what was changed).
Repeat the exact same procedure on the following VMs (I've not been very consistent on names...):
You may deactivate it in some portions of the code with
ssh ci@morefem-gitlab-docker-2.ci
ssh ci@morefem-gitlab-docker-3.ci
ssh ci@morefem-gitlab-docker-4.ci
ssh ci@docker5.ci
ssh ci@docker6.ci
## Runner with shell executor (on macOS)
````
// clang-format off
````
which neutralize it until the opposite
`````
// clang-format on
````
macOS can't run in a Docker container, so we will need to use shell executors on machines. A shell executor simply means the job will run directly on the machine; so you need to connect yourself regularly on this machine to update it.
is written.
There is currently only one such executor on M3DISIM machine _Nostromo_; you need to be part of the team to be able to use it. If you don't have access to it, you may use the _gitlab-ci-no-macos.yml_ provided which skips the related job in your Gitlab project (see [here](#set-up-the-yaml-file-to-use)).
If you have credential to _Nostromo_:
* Log in with your account.
* Change account and choose CI (same credential as for Linux VM).
* Run _gitlab-runner_ (without _sudo_) and choose:
- Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
Copy/paste password just as for the Linux case.
- Please enter the gitlab-ci token for this runner:
Same as Linux case.
- Please enter the gitlab-ci description for this runner:
You may just repeat the name of the machine... or really put whatever you want (you may change it afterwards anyway).
- Please enter the gitlab-ci tags for this runner (comma separated):
Choose _macos_.
- Please enter the executor:
Choose _shell_.
......
<!-- toc -->
- [Install Miniconda environment](#install-miniconda-environment)
- [Install third party libraries](#install-third-party-libraries)
- [Install gitlab-runner](#install-gitlab-runner)
* [As an administrator:](#as-an-administrator)
* [As CI user:](#as-ci-user)
<!-- tocstop -->
[TOC]
# Install Miniconda environment
......
# SSH access
<!-- toc -->
- [Generate a pair of public and private keys](#generate-a-pair-of-public-and-private-keys)
- [Register this key in CI platform](#register-this-key-in-ci-platform)
- [Shortcut to ease connexion](#shortcut-to-ease-connexion)
<!-- tocstop -->
To connect yourself to the CI Inria VMs, you need first to follow several steps documented [here](https://wiki.inria.fr/ciportal/Slaves_Access_Tutorial).
To put in a nutshell:
## Generate a pair of public and private keys
For instance for the M3DISIM-CI account, I have generated a key with:
````
mkdir ~/.ssh/m3disim-ci
ssh-keygen -t rsa -C "m3disim-ci@inria.fr" -b 4096 -f ~/.ssh/m3disim-ci/id_rsa
````
## Register this key in CI platform
On the [CI platform](https://ci.inria.fr), log yourself in and click on your login:
![screenshot login](Images/login.png)
and choose "My account".
There, copy the *public* key generated above (i.e. the content of the file ~/.ssh/m3disim-ci/id_rsa.pub).
## Shortcut to ease connexion
As indicated on the [CI tutorial](https://wiki.inria.fr/ciportal/Slaves_Access_Tutorial), you should edit ~/.ssh/config and write inside:
````
Host *.ci
ProxyCommand ssh mci001@ci-ssh.inria.fr "/usr/bin/nc `basename %h .ci` %p"
````
(replace mci001 by your CI login if you're not using M3DISIM-CI account).
......@@ -17,7 +17,7 @@ cppcheck:
only:
- develop@MoReFEM/CoreLibrary/MoReFEM
- /(sonarqube)/
- /(full_ci_analysis)/
- /(full_ci)/
rats:
......@@ -36,7 +36,7 @@ rats:
only:
- develop@MoReFEM/CoreLibrary/MoReFEM
- /(sonarqube)/
- /(full_ci_analysis)/
- /(full_ci)/
clang-static-analysis:
......@@ -55,7 +55,7 @@ clang-static-analysis:
only:
- develop@MoReFEM/CoreLibrary/MoReFEM
#- /(sonarqube)/
- /(full_ci_analysis)/
- /(full_ci)/
sonarqube:
stage: generate_sonarqube
......@@ -78,4 +78,4 @@ sonarqube:
only:
- develop@MoReFEM/CoreLibrary/MoReFEM
- /(sonarqube)/
- /(full_ci_analysis)/
- /(full_ci)/
......@@ -19,6 +19,7 @@ variables:
except:
- master
- /(nobuild)/
- /(noci)/
tags:
- ci.inria.fr
- large
......@@ -57,6 +58,7 @@ variables:
except:
- master
- /(nobuild)/
- /(noci)/
cache:
key: "cache_${CI_PROJECT_ID}_${CI_COMMIT_REF_SLUG}_${OS}-${COMPILER}-${MODE}-${LIB_NATURE}-${IS_ONLY_ONE_LIB}"
untracked: true
......@@ -79,7 +81,8 @@ variables:
.run_doxygen_template: &run_doxygen_template
stage: build_and_test
except:
- /(nobuild)/
- /(nodox)/
- /(noci)/
tags:
- ci.inria.fr
- linux
......@@ -102,6 +105,7 @@ variables:
except:
- master
- /(nobuild)/
- /(noci)/
tags:
- ci.inria.fr
- linux
......@@ -121,6 +125,7 @@ variables:
except:
- master
- /(nobuild)/
- /(noci)/
# dependencies: in the instantiations!
script:
- python Scripts/Tools/find_warning_in_compilation_log.py --log build/compilation.log
......@@ -131,7 +136,8 @@ variables:
stage: check_warnings
except:
- master
- /(nobuild)/
- /(nodox)/
- /(noci)/
tags:
- ci.inria.fr
- linux
......
......@@ -20,7 +20,7 @@
only:
- develop@MoReFEM/CoreLibrary/MoReFEM
- /(valgrind)/
- /(full_ci_analysis)/
- /(full_ci)/
allow_failure: true
......
[TOC]
# How to set up include-what-you-use
Rather obviously, first install https://include-what-you-use.org/ on your computer...
......
[TOC]
# Valgrind call
Valgrind should be run only in a Linux environment, unless its developer manage to provide a working version in Mac (currently even a very basic function such as ls returns plenty of errors in Mac).
There are many leaks stemming from third party libraries; most of them are suppressed with the help of the files in this directory.
......
......@@ -152,8 +152,7 @@ Already mentioned, but you should really have a look at this [tutorial](Documen
## Coding standards
Coding standards are provided in Documentation/CodingStandards. They are mostly inspired by those of [Verdandi](verdandi.sourceforge.net).
Coding standards are provided in _Documentation/CodingStandards_. They are mostly inspired by those of [Verdandi](verdandi.sourceforge.net).
# Getting help
......@@ -167,37 +166,3 @@ You may also contact engineers currently involved (part-time) on the project, bo
If you're part of M3DISIM team, please ask your team leader or Jérôme to add you in the internal Mattermost channel dedicated to the project.
# Continuous integration