faq.md

@@ -263,6 +263,78 @@ shm_size = 0
 [runners.cache]
 ```
 
+### Cleaning the pipelines (artifacts, logs)
+
+In the course of time the size of data generated for gitlab-ci pipelines may grow quickly and sometimes reaches dozens of GB.
+The quantity of data stored on the gitlab's server for pipelines can be checked in the Settings -> Usage Quotas panel of the project, or in Build -> Artifacts.
+For projects with more than 10GB of artifacts there is certainly something to do to reduce the disk storage:
+
+1) By cleaning old artifacts.
+2) By disabling the "Keep artifacts from most recent successful jobs" in the Settings -> CI/CD -> Artifacts if not necessary because it keeps all job's artifacts (build, test, etc) of all the git refs (branches, tags, merge requests, ...) and this can cost a lot.
+3) By changing the gitlab-ci jobs definitions, for example:
+
+   - use the [expire_in](https://docs.gitlab.com/ee/ci/yaml/#artifactsexpire_in) keyword to reduce the expiration time (30 days by default) of artifacts,
+   - use [cache](https://docs.gitlab.com/ee/ci/yaml/#cache) instead of artifacts when possible when jobs with dependencies can be executed on the same machine,
+   - generate fewer archives and lighter archives, keeping only what is strictly necessary.
+
+To view and delete artifacts and logs there is for example this tool [gitlab-storage-analyzer](https://gitlab.com/gitlab-de/use-cases/gitlab-api/gitlab-storage-analyzer).
+This tool use the Gitlab's REST API to execute tasks on the project, thus you must get:
+
+- a [Personal Access Token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) with __api__ access,
+- the gitlab's project __id__, see the number in front of Project ID on the main page of your project.
+
+Then you can clone it locally, install python-gitlab with pip
+```sh
+git clone https://gitlab.com/gitlab-de/use-cases/gitlab-api/gitlab-storage-analyzer.git
+cd gitlab-storage-analyzer/
+
+pip install python-gitlab
+```
+
+and start playing.
+```sh
+export GL_SERVER=https://gitlab.inria.fr
+export GL_TOKEN=
+export GL_PROJECT_ID=
+python gitlab_storage_analyzer.py
+```
+This will list all the artifacts.
+
+To only look for archives and exclude logs (called trace here) use `GL_EXCLUDE_FILE_TYPE`
+```sh
+export GL_EXCLUDE_FILE_TYPE="trace,metadata"
+```
+
+To only look for logs
+```sh
+export GL_EXCLUDE_FILE_TYPE="archive,metadata"
+```
+
+To only look for artifacts older than a number of seconds use `GL_THRESHOLD_AGE_SEC`
+```sh
+export GL_THRESHOLD_AGE_SEC=604800 # i.e. older than 1 week
+```
+
+Then to actually delete the filtered artifacts use `GL_DELETE_MODE`
+```sh
+export GL_DELETE_MODE=threshold # none by default
+```
+Unfortunately this script cannot be used to delete logs (only archives) associated with pipelines and which, yet, can be responsible for a large amount of the storage quota.
+
+To remove logs of a pipeline there is no other way than deleting the pipeline.
+The script [cleanup-gitlab-pipelines.sh](https://gitlab.inria.fr/siteadmin/doc/-/snippets/897) allows to delete 100 (max pagination size) pipelines older than a certain date:
+```sh
+export GL_SERVER=https://gitlab.inria.fr
+export GL_TOKEN=
+export GL_PROJECT_ID=
+export GL_DELETE_BEFORE="2023-01-01"
+./cleanup-gitlab-pipelines.sh
+```
+To delete more than 100 pipelines one has to repeat this operation several times.
+
+After having doing that the storage quota can be recalculated on the page Settings -> Usage Quotas, button "Recalculate repository usage" and wait several minutes to get a refreshed page up-to-date (the result is not instantaneous).
+
+
 ## Git-LFS is not activated
 
 The git-lfs feature is not activated. To version data or binary files, an option is to use git-annex on Inria Forge. Please have a look at the following links: