faq

FAQ - Frequently Asked Questions

FAQ - Frequently Asked Questions

Good usage of the gitlab?

Gitlab is initially and mainly aimed at providing a set of tools for collaborative work on software development projects. Additionally, it can also be used for collaborative work on projects not directly related to software development, such as collaborative edition of scientific publications, since source code or text edition are similar.

Keep in mind that the design and sizing of gitlab have some limitations:

Git is designed to keep a complete history of operations. It is not expected to change this history and doing so is rather convoluted (see related faq entry). This means that if you put a file under git and later remove it, the file will be removed only from the current and following revisions, not from the old ones. Disk space is not reclaimed. If you commit different versions of a single file, each version will be stored and uses disk space.
Git is best used to store text files (source code, latex source, etc.), because they compress well.
Git is designed initially to version source code. The size of a git repository such as the linux kernel repository is in the order of 100 to 200 MB (and this is a rather large project). Repositories with sizes wich are one order of magnitude or more greater will be slow.
The worst case is storing big binary files (they don't compress well)
Additionally, having huge repositories will impact not only git, but also gitlab as a whole:
- The web interface will be slow (eg. looking at big commits in web interface will timeout)
- Continuous Integration will be impacted (eg. need to have big runners, build artifacts will be huge, and jobs may fail or timeout)

Hence, some gitlab usages are rather inappropriate:

Using source control to store a lot of binary data (either lots of medium to big files, or a few enormous files).
In particular, it is bad usage to use the git to publish and share big experimental datasets (inputs or outputs), virtual machine images, and more generally any kind of "big data". You should, instead, version the code for generating this data or vm images.

Alternative solutions:

If you need to publish huge volumes of data, you need to find some network storage outside of the gitlab. See with your IT contacts for solutions, or find a funding for your own servers / space. The gitlab is not a Dropbox alternative for big data!
git-lfs may be a good solution but it is not yet activated for the Inria gitlab.
An alternative to git-lfs is git-annex.

The frontier between good and bad usage of the gitlab is fuzzy, but as there are a lot of users and projects on the gitlab, if too much people abuse the system we will have to put hard limits and restrictions on the gitlab usage, which will be annoying for everyone.

How to not have to type a password for every git operation

First of all to avoid having to type login/password each type, the best is to use git over ssh rather than git over https (see below for git over http)

On your local computer:

git relies on ssh for the connexion to the gitlab's git server
ssh uses your ssh private key to authenticate to the gitlab's git server
your ssh private key is stored in your home, on your local computer's hard disk. This ssh private needs to be crypted, so that if your local computer is breached or stollen, the key cannot be used. This is the reason why it is crypted with a so-called "passphrase". This passphrase is actually a password, but you are encouraged to use something longer and more complex than a password (beacuse if this passphrase is "cracked", it can lead to a breach on many servers, so from the security point of view, it is critical)

Up to now, everything works correctly, but you have to type your ssh passphrase each time you connect by ssh, thus at each git operation involving the remote git server (push, pulls, ...)

The ssh-agent's role is to avoid that: it is a software running in background on your local computer. Each time your local ssh client needs to connect to a server, if it detects a running ssh-agent, it first tries to get the decrypted ssh private key from it. The first time, the ssh-agent asks for the passphrase to decrypt the key, then keeps the decrypted key in memory as long as you don't log out of your local computer.

This is the ssh-agent that allows not having to enter the passphrase at each ssh connection.

By default ssh client is installed with a command line ssh-agent, but modern operating systems / graphical environments come with integrated "keyrings" which can act as ssh-agents (and can as well manage other kind of secrets, such as passwords, gpg keys, etc.):

for OS X: Keychain
for linux / gnome: gnome-keyring
for linux / kde: kde-wallet
for other linux desktop environments, or for windows: ?

These "keyrings" work as follows: When you login on your local computer, your login password is automatically used to unlock the secrets stored in the keyring, of which the ssh keys are part.

Thus to be able to perform connected git operations (push, pull, ...) without having to type any password you need:

to use git over ssh
to have a ssh keypair protected by a passphrase
to use a command line or graphical ssh-agent

Note: it is also possible to do passwordless operations with git over http by using a personal access token:

create a personal access token in your gitlab user settings, with permissions "read_repository" (only these permissions).
when cloning the repository, use customize the URL as follows: https://:@gitlab.inria.fr/<end_of_url>

How to convert a user account from internal to external

When a user is leaving Inria, the gitlab account cannot be used anymore. Two situations can occur:

The user is still at Inria

Before leaving Inria:

open a ticket requesting to convert the account, providing the external email that will be used as identifier and for password recovery.
the user ask for password recovery using the new identifier when the account has been converted and provide a new password.

The user has already left Inria

Only an Inria administrator of a project the user was contributing to can ask for the account conversion.
The administrator open a ticket requesting to convert the account, providing the external email that will be used as identifier and for password recovery.
the user ask for password recovery using the new identifier when the account has been converted and provide a new password.

How to activate a commit email hook

Gitlab's model is that instead of setting up a centralized commit email hook per project, it is rather up to the users to choose his/her notification level per project. All users can set their 'global' notification level setting in their account settings. Then they can override this default global notification level per project on top of the project details page, with the little bell (This way, a project cannot impose a notification level to its members).

If you still want something similar to the old commit email hooks, then in your project configuration, section "Integrations/Project services", activate the "Emails on push" service. Then, unfortunately, you need to enter the explicit list of recipients. We haven't found a way to configure this such that the list of project members is automatically used as the recipients list

How to rewrite history

As documented in https://gitlab.inria.fr/help/user/permissions project members with roles Developer or Master or Owner are allowed to force push to non protected branches. If you want to rewrite history of a protected branch, You need first to switch the branch to non-protected (only Master or Owner can do that), do the force-push, then revert the branch to protected.

How to add users of an INRIA ildap group (eg. team members) to a gitlab group

Open a ticket, specifying the INRIA ildap group name, the gitlab group name, and the access level (GUEST, REPORTER, DEVELOPER, MASTER, OWNER) that you wish to give to added members. We will run a script which adds existing gitlab users matching the ildap group members to the gitlab group. Be aware that inria members need to connect at least one time to gitlab for their account to be created.

Provided you are in the INRIA network and have access to ldap://ildap.inria.fr, you can search for a groupname with a command such as:

$ ldapsearch -x -H "ldap://ildap.inria.fr" -b "ou=groups,dc=inria,dc=fr" | grep -i <team name>

For verification purposes you can check the list of ildap group members with a command such as:

$ ldapsearch -x -H "ldap://ildap.inria.fr" -b "ou=People,dc=inria,dc=fr" "(inriaGroupMemberOf=cn=<ldap group name>,ou=groups,dc=inria,dc=fr)"

How to use the continuous integration (CI) service?

Besides using Inria's Continuous Integration platform ci.inria.fr, you can use gitlab's integrated CI pipelines. This tool is more suited to the execution of simple and lightweight tasks, than for complex deployment pipelines.

You will need to set up a CI runner (small tool in charge of running the CI tasks themselves) and then register the runner with your project.

Enabling CI on a gitlab project

Go to your project's settings
In the General Permissions tab, search for "Pipelines", and check the permission is not "Disabled"
Then a CI / CD tab should now appears in the settings
In this CI / CD tab, you will find the url and the registration token you will need later, when adding a specific runner.

Installing runners

To perform the actual build, you need to install GitLab Runner. Runners can run anywhere : you can use a virtual machine on ci.inria.fr to host your GitLab runner.

See :

https://inria-ci.gitlabpages.inria.fr/doc/page/web_portal_tutorial/ to create a slave on Inria's CI platform and access it (you can ignore the jenkins related parts)
https://docs.gitlab.com/runner/install/ for the official documentation to install the runner on the vm(s) you've created.

Installation example on a GNU/linux slave from ci.inria.fr ("ci" user account)

sudo wget -O /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
sudo chmod +x /usr/local/bin/gitlab-runner
sudo gitlab-runner install --user=ci --working-directory=/builds
sudo gitlab-runner start
sudo gitlab-runner status # should return "service is running"

Installation example on a macOS slave from ci.inria.fr ("ci" user account)

Limitations on macOS : The service needs to be installed from a Terminal window logged in as your current user (i.e., "ci" user account).

sudo curl --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-darwin-amd64
sudo chmod +x /usr/local/bin/gitlab-runner
# Run the following commands as the "ci" user
gitlab-runner install --working-directory /builds
gitlab-runner start
gitlab-runner status # should return "service is running"

Installation example on a Windows slave from ci.inria.fr ("ci" user account)

# Follow https://docs.gitlab.com/runner/install/windows.html then when installing the service prefer the following to use the existing "ci" user account
gitlab-runner install -u ".\ci" -p "ci" -d "C:\Users\ci"
gitlab-runner start
gitlab-runner status # should return "service is running"

Register a runner in your project

On the virtual machine where you installed the gitlab CI runner, run (as root or sudo if the gitlab-runner program has been installed with sudo, remove sudo from the following if not) :

$ sudo gitlab-runner register

Use the gitlab URL and the registration token found in the "Pipelines" page when asked.

In order to answer when asked

Please enter the gitlab-ci tags for this runner (comma separated):

it is important to know that "gitlab-ci tags" are NOT git tags. The tags you enter here are useful to specify conditional execution in the configuration file .gitlab-ci.yml.
See :

for more details.

The next question

Whether to run untagged builds [true/false]:

becomes clear when the "gitlab-ci tag" concept is understood.

When asked

Please enter the executor:

you should answer shell (or docker but then see the related subsection below)

At the end of this step, your runner should appear in the Settings > Pipelines tab of your project.

Use existing runners

If you don't want to manage runners yourself you can use some existing ones provided by ci.inria.fr where jobs can be executed in docker containers. Please contact the support team for more information.

Configurating CI tasks

Finally, configure the tasks to run by creating a .gitlab-ci.yml file at the root of your project.
Follow the official documentation at https://docs.gitlab.com/ce/ci/yaml/ to create this file.

Using a docker executor

It has been reported that there are DNS issues with docker running on the INRIA CI's VMs (due to bad interaction between dnsmasq and docker, see https://stackoverflow.com/questions/49998099/dns-not-working-within-docker-containers-when-host-uses-dnsmasq-and-googles-dns). This can be solved by adding network_mode = "host" in the configuration. Eg. /etc/gitlab-runner/config.toml:

concurrent = 1
check_interval = 0

[[runners]]
name = "ci.inria"
url = " [ https://gitlab.inria.fr/ | https://gitlab.inria.fr/ ] "
token = "..."
executor = "docker"
[runners.docker]
network_mode = "host"
tls_verify = false
image = "alpine:latest"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]

Git-LFS is not activated

The git-lfs feature is not activated. To version data or binary files, an option is to use git-annex on Inria Forge. Please have a look at the following links:

Why isn't git-LFS activated? We observe that many users are trying to use git/gitlab to store huge volumes of data such as experimental datasets, experimental results, binary blobs, big data, etc. Git is not designed to handle such usage, and git-lfs would be the good answer, but because we do not have yet a working, production ready, quota system for gitlab, and because we do not yet have a clear strategy about how to manage such storage, we do not want to encourage such usages, and thus we decided to not enable git-lfs.

You are welcome to tell us if git-lfs is a need for your usage. Your requests will be taken in account for future decisions on this matter.

Merge Request development model and external contributors

This Inria Gitlab instance offers accounts both to Inria members (who login using their Inria LDAP credentials) and to external users (who can be invited/sponsored by Inria members through the external account portal: https://external-account.inria.fr ). The external (non-inria) users are limited to join existing projects, but cannot create projects on their own.

This limitation for external users has an important result: from the Gitlab point of view, forking a project is creating a project, thus external users cannot fork projects.

As a result, the well-known MR (Merge Request) development model cannot be used directly with external users, but some workarounds can be used in some situations.

The MR development model works as follows:

A restricted core development team is member of a project and has commit access to the development tree
Some contributor starts contributing by forking the project ("fork" button). When their contribution is ready, they open a new merge request (in the "new" menu).
Someone from the core development team can then accept (or refuse) the Merge Request to be merged into the main development tree

This development model has several benefits:

It limits the access to the main development tree to a small group of trusted core developers.
It allows the core development team to control finely which contributions to merge or not.
In particular, it shields the project and forces all contributions to pass through a validation workflow, such as ensuring that the Continuous Integration passes, or that all intellectual property constraints are satisfied, before merging some development.
With tools such as Github or Gitlab, the web user interface for merge requests is very user friendly.

So, here are some possible workarounds to this development model when using the Inria Gitlab with external contributors:

Use pull-requests mechanisms at the git level, rather than at the Gitlab level. This is the way that the Linux kernel has been developed for a long time now. Instead of submitting a MR with the Gitlab interface, contributors send patches by email.
For projects with few known external contributors, it is possible for Inria members to create forks on behalf of the external contributors, then transfer ownership of these forks to the contributors. Here is a possible worklow:
create a new group to handle the forks of your external collaborators
take care to remain the only owner/master of the group in order to control the creation/destruction of forks
create a fork from the original project with as destination the newly created group. By default the name of the project will be the same as the original project, e.g. https://gitlab.inria.fr/ForksOfFoo/bar
then rename this forked project through Settings -> General -> Advanced Settings, e.g. https://gitlab.inria.fr/ForksOfFoo/bar-jdoe
finally add your collaborator master on that fork
For projects with few known and trusted external contributors, these contributors can be added to the project but develop in their own branches. There are some basic mechanisms in Gitlab to ensure they don't override their permissions: if these users have role developer, they won't be allowed to push to protected branches (master, by default). Details on Gitlab permissions can be found here: https://gitlab.inria.fr/help/user/permissions.md.

If these workarounds do not fit well with what you need and you absolutely want to use the standard MR development model with external contributors, the only solution is to use another tool than the Inria Gitlab.

Gitlab-Pages

Gitlab-Pages are activated on our gitlab instance. Documentation can be found here: https://gitlab.inria.fr/help/user/project/pages/index.md
Gitlab pages allow only static html content to be published (it means: no database, no php, etc.). The pages are generated as part of a continuous integration step, so you need a gitlab runner for the pages generation. As there is currently (April 2020) no shared runner on our instance, a workaround is to ask the support team, you'll be added to a special project which allows you to use a (beta testing) docker runner to build your pages instead of having to build and manage one yourself.
There is a basic gitlab-pages example project here: https://gitlab.inria.fr/siteadmin/pages-example The generated web page is here: https://siteadmin.gitlabpages.inria.fr/pages-example/
You can also have a look at https://gitlab.inria.fr/help/user/project/pages/getting_started/new_or_existing_website.md note that there are templates for many website generators.
Inria gitlab does not support custom domains and certificates. They are required if you want to have a custom domain that you own point to a gitlab pages generated site (optionnaly in https) but Inria has explicitely forbidden this functionnality.
The first time Gitlab-Pages are activated, a question is asked: "Authorize GitLab Pages to use your account? An application called GitLab Pages is requesting access to your GitLab account. Please note that this application is not provided by GitLab and you should verify its authenticity before allowing access. This application will be able to: Access the authenticated user's API, Grants complete read/write access to the API, including all groups and projects, the container registry, and the package registry.". This is strange, but normal. The Gitlab-Pages server is a separate server, and for situation where pages are restricted to project members, the Gitlab-Pages server must use the authentication infrastructure of the Gitlab server. So you need to answer yes.

Broken Links with Uploaded Files in a Project Wiki

The symptom is that when clicking on links to uploaded resources, an error 404 is thrown. The link follows the pattern (/uploads/4d0b0eecbe6f0bd95a96f3b90cf64fe3/my_file.pdf). In that case, check that the file uploads/4d0b0eecbe6f0bd95a96f3b90cf64fe3/my_file.pdf is not in the git repository of the wiki by:

cloning the wiki repository (button at the upper right of the wiki);
checking with ls uploads/4d0b0eecbe6f0bd95a96f3b90cf64fe3/my_file.pdf whether the file is stored in the repository. If it not, then you are in the situation addressed by this FAQ entry.

Explanation

The way the linked files are stored in the wiki has changed (cf. https://docs.gitlab.com/ee/user/project/wiki/#attachment-storage): previously to gitlab 11.3 release, the linked files of all the wikis were uploaded in a global /uploads directory. Since 11.3, they are uploaded to a uploads directory pertaining to the git repository of the wiki.

But the files that were uploaded previously to 11.3 have not been moved to the wiki repositories, causing:

404 errors since it appears the global repository is not used anymore when gitlab follows the links inside the wiki;
the files are not present when cloning the wiki repository, making the clone incomplete.

Workarounds

If the file is not stored in the git repository of the wiki, two solutions exist:

Quick and Dirty Workaround

Change the (/uploads/4d0b0eecbe6f0bd95a96f3b90cf64fe3/my_file.pdf) links to (https://gitlab.inria.fr/uploads/4d0b0eecbe6f0bd95a96f3b90cf64fe3/my_file.pdf).

Long-term Solution

It requires to download and the upload again each linked file:

the file can be downloaded using the link given in the previous section;
then upload the file using the "attach a file" button at the bottom right of the wiki editor.