Skip to content

GitLab

D
doc
Last edited by Olivier Rochel
Page history

Forge to Gitlab migration FAQ

Planning the migration

INRIA Forge project administrators and/or users need to coordinate to decide when and how to perform these migrations. It's also important to decide what exactly needs to be migrated, since you may not need to keep everything.

Coordination is paramount because if, as a project admin, you migrate for example the source control repository without telling the users, some of them might still use the old repository on the forge for checkouts or commits.

Communication with forge project members

As there is no simple way for a forge project administrator to get the list of emails of project members (to communicate with everyone and be sure to forget nobody), we can provide these lists on demand to forge project administrators. Open a ticket for this.

Preventing project members from commiting / pushing

Send a Helpdesk ticket, "Submit an IT request -> Service request -> Services for research -> Request for Gitlab", to ask administrators to add a pre-commit hook.

Subversion

Add a pre-commit hook which always returns false. For example:

ssh scm.gforge.inria.fr
echo -e '#!'"/bin/bash\necho \"This repository is read-only\"\nexit 1" > /svnroot/<project unix name>/hooks/pre-commit
chmod a+x /svnroot/<project unix name>/hooks/pre-commit

Git

Add a pre-receive hook which always returns false. For example:

ssh scm.gforge.inria.fr
echo -e '#!'"/bin/bash\necho \"This repository is read-only\"\nexit 1" > /gitroot/<path_to_git_server_repo>/hooks/pre-receive
chmod a+x /gitroot/<path_to_git_server_repo>/hooks/pre-receive

New access policy for external users (from inria)

Inria has decided in june 2020, for security matters and due to spams, to disable account creation for non-inria users. External users can have access to this gitlab but with a sponsorship system managed by the IS department (DSI in french) and hosted here https://external-account.inria.fr/. This means that an Inria member must create an account through this portal to let external members get an account. This sponsorship must be renewed every year.

Available documentation and help

The gitlab documentation is available here: https://gitlab.inria.fr/help

Beware that gitlab is both a software (gitlab) and a free hosting platform (https://gitlab.com). In many places in the documentation, the links given point to https://gitlab.com, you have to replace these by links to our instance: https://gitlab.inria.fr

Migration support is available through the inria helpdesk, in section "IT request", subsections "Ask for a fix" or "Service request" / "Services for research " / "GitLab" or "Inriaforge"

A mattermost channel is also available https://mattermost.inria.fr/devel/channels/gforge-gitlab (but remember that the preferred way to ask for support is through the helpdesk because support tickets can be correctly tracked).

For specific technical details on gitlab, git, git svn... there are numerous resources easily available on the web. In many situations, moving from subversion to git is pretty easy as long as you stick with basic operations:

  • svn checkout -> git clone
  • svn update -> git pull
  • svn commit -> git add + git commit + git push

There are cheat sheets with basic equivalence, such as this one.

To better understand git concepts, you can for example have a look at Anthony Baire tutorials

Gitlab project setup

A gitlab project needs to be created. It's up to you to choose between private / internal / public project (see https://gitlab.inria.fr/help/public_access/public_access.md). You can choose between a project in a user namespace or in a group namespace (see https://gitlab.inria.fr/help/user/group/index.md#namespaces)

Projects members need to have an account on the gitlab. If they don't have an account, they need to create one. It will then be possible for them to request access to the project (but only for public projects) or the opposite, an administrator can manually add them.

Project name and user names do need necessarily need to be the same between gforge and gitlab (though it may be simpler, if they are the same).

Also don't forget that to be able to actually do something (like cloning, pushing...) users need to setup their account correctly. If they want to use ssh to avoid entering their username/password on each action, they should have a working ssh setup and upload their ssh public key to the gitlab (the same as with the gforge)

Migration of source control repositories

INRIA gitlab is restricted to git (no subversion anymore), so for subversion users unfamiliar with git, you should take the time to read and understand the basic differences between subversion and git. If you want to stay with subversion, there is no alternative at INRIA, but you can go to SourceSup https://sourcesup.renater.fr/, which is actively maintained by Renater.

You need to decide if you want to migrate just the tip of your development history (last version on the repository) or if you want to migrate the whole commits / branching / tags history. Note that in several situations, there is no real need to migrate the whole history. For example if you keep a project only for historical reference, you may not need all the commit history.

In the following examples, we assume that the gitlab project is already created, correctly setup, and empty. The user performing the migration has a gforge account <forge_user>, a gforge project with unixname <forge_project>, a gitlab account <gitlab_user> and a gitlab project in the <gitlab_user> namespace, with name <gitlab_project>.

Migrating only the last version of the repository

Get the last version of the development tree (git or subversion), and import this in a new gitlab repository. All history is lost.

Migrating the whole (or part of the) history

From Subversion

Command-line git-svn method

Handling of branches and tags

In subversion, branches and tags are kind of symlinks to snapshots of the development history. They can be put anywhere in the repository directory hierarchy, but usually it is advised to gather them in specific directories, and the recommended best practice is the standard layout of three top directories in the subversion repository: trunk (main development tree), tags, branches.

By default git-svn just clones the repository 'as-is', (you'll end up with a migrated repository in git with exactly that structure, trunk, tags, branches, and no git branches nor git tags).

But you almost always want to convert subversion branches and tags. git-svn takes options to set the directories for branches and tags. If your subversion repository has the classic standard layout structure, you can use the shorthand option --stdlayout to convert all svn branches in branches and all svn tags in tags to git branches (if your repository has a non standard layout, there are options to instruct git-svn of the layout, see git-svn documentation).

git-svn command

Checkout the subversion repository with git-svn (https://git-scm.com/docs/git-svn). It creates a git repository from the subversion repository (with all the history). You can then add your gitlab project as a git remote and push to that project:

git svn --stdlayout clone svn+ssh://<forge_user>@scm.gforge.inria.fr/svnroot/<forge_project>/
cd <forge_project>
git remote add origin git@gitlab.inria.fr:<gitlab_user>/<gitlab_project>.git
git push --set-upstream origin master

In this example we only push the master branch. When there are other branches and/or tags, see below section Post-processing for branches and tags

See also how to change authors of commits and clean commits.

This directory can then be used to work from now on, or if you want to be sure that everything is correct, you can try a new checkout from scratch from the gitlab in another directory:

cd ..
mv <forge_project> <forge_project>.migration
# (may be removed later when you're sure the migration was correct
git clone git@gitlab.inria.fr:<gitlab_user>/<gitlab_project>.git
cd <gitlab_project>
# check that all is correct
Post-processing for branches and tags

git-svn by default puts all subversion branches and tags in remote branches pointing to the original subversion repository, because git-svn is bi-directional, you can push to the svn repo. But in our case, let's say that:

  • we want to migrate to git once and for all
  • we want all subversion branches to end up as regular git branches
  • we want all tags to end up as regular git tags

For this, there is a bit of post-processing involved before pushing to the new git repository:

checkout branches

Iterate all remote branches of the newly checked-out git-svn repository, filter out the tag branches, and checkout these branches, so that they are included when pushing to the new git remote:

for D in $(git branch -a | grep '^\s*remotes/' | grep -v '/tags/') ; do
     git checkout -b "$(basename "$D")" "$D"
done

convert tags

Iterate all remote branches of the newly checked-out git-svn repository, if they are tag branches then create a git tag:

git for-each-ref refs/remotes/origin/tags |
    while read REF ; do
        SHA1=$(echo $REF | awk '{print $1}')
        TAG=$(echo $REF | awk '{print $3}' | cut -d / -f 5- )
        git tag "$TAG" "$SHA1"
    done

push everything

Once all branches and tags are ready, instead of just pushing the master branch, push all branches and tags:

git push --force -u origin --all
git push --force -u origin --tags

Alternative methods

Other methods are described here: https://gitlab.inria.fr/help/user/project/import/svn

From Git

Command-line mirror method

A convenient way to migrate everything (all branches, all tags) in a single operation is to use a mirror checkout:

git clone --mirror git+ssh://<forge_user>@scm.gforge.inria.fr/gitroot/<forge_project>/<forge_project>.git
cd <forge_project>.git
git push --mirror git@gitlab.inria.fr:<gitlab_user>/<gitlab_project>.git

This directory cannot be used as a local working copy since the --mirror option implies --bare. You need to checkout a new working copy:

cd  ..
mv <forge_project>.git <forge_project>.git.migration
# (may be removed later when you're sure the migration was correct
git clone git@gitlab.inria.fr:<gitlab_user>/<gitlab_project>.git
cd <gitlab_project>
# check that all is correct

Gitlab import method

An alternative method is to directly choose the "Import project" tab when creating the project in gitlab, and then choose "Git repo by URL". Enter the URL of the project's git repository on the forge, such as: https://scm.gforge.inria.fr/anonscm/git/<forge_project>/<forge_project>.git (anonymous access) or https://<forge_user>@scm.gforge.inria.fr/authscm/<forge_user>/git/<forge_project>/<forge_project>.git (https developer access). Anonymous access is only available for public repositories. For https developer access you'll need to provide forge username and passsword for gitlab to be able to retrieve and import the project from the forge. It is not possible to use a git+ssh URL scheme for this method.

Filtering history

If needed, you can filter out part of the history before pushing it to the new gitlab git repository, with tools such as git-filter-repo, bfg-ish, BFG Repo Cleaner.

Mapping subversion / git committers to gitlab users

Git itself has no knowledge of forge / gitlab users. From git point of view, the author of a commit is a string Gitlab maps commit authors to gitlab users accounts with the git .mailmap file. It allows solving situations where a forge svn or git repository is migrated to gitlab and some users don't have the same username between the forge and the gitlab. It also allows solving situations where some users have committed with varying names/addresses during the life of the project.

Subversion / Git Hooks

Classic subversion or git hooks, implemented as executable in the hooks directories of the subversion or git server repositories, are not possible on the gitlab. So the usual email on subversion commit / git push has no direct equivalent. Gitlab has the concept of notifications, where this is the user who decides what he/she wants to be notified for. But git push are not part of the notified events. So the best equivalent of the email on commit/push is to activate "Emails on push" in the gitlab project integrations.

For other, more custom / advanced hooks usage, gitlab provides web hooks https://gitlab.inria.fr/help/user/project/integrations/webhooks.md

Forge Forums and mailing lists

There is no equivalent to forge forums or forge mailing lists. An alternative is to migrate both forge forums and mailing lists to the INRIA mailing lists server https://sympa.inria.fr/sympa.

migration of mailing lists users

To get the list of users of a forge mailing list: Point a browser to https://lists.gforge.inria.fr/mailman/roster/<listname>

To import this list of users in a https://sympa.inria.fr mailing list: got to "Add/delete subscribers" on the list page, then click "Multiple add" and copy/paste the list of email addresses

migration of mailing lists archive

If you need to transfer ML archives from InriaForge to Sympa INRIA mailing lists server:

  • You need to create the sympa mailing list beforehand
  • You need to be admin of both the InriaForge and the Sympa mailing lists
  • Submit a ticket, indicating both mailing lists names, on InriaForge and on Sympa (they may be the same or different)

Forge Trackers

The equivalent to forge trackers are gitlab issues. The equivalent to having multiple trackers is to use issue labels. D. Vojtisek has contributed a script to help migrate trackers to gitlab: https://gitlab.inria.fr/sed-rennes/gforge-to-gitlab-scripts

Forge Tasks

One can use a combination of gitlab milestones and/or issue boards to manage the development.

Forge Documents, File Release System (FRS)

Forge Documents and File Release System are two tools with similar functionality: managing files.

gitlab

gitlab releases

Gitlab is lacking a full equivalent. Since v8.2, there is a release functionality (https://about.gitlab.com/releases/2015/11/22/gitlab-8-2-released/#releases) and since 11.11 there is guest access to releases (https://about.gitlab.com/releases/2019/05/22/gitlab-11-11-released/#guest-access-to-releases). There are some limitations, as pointed in https://stackoverflow.com/questions/29013457/how-to-store-releases-binaries-in-gitlab. The most important limitations are that parts of the functionality can only be controlled by the API, and though releases can be removed, file attachments cannot. There is an open issue upstream regarding this feature: https://gitlab.com/gitlab-org/gitlab/-/issues/16229. Current up-to-date documentation can be found here: https://gitlab.inria.fr/help/user/project/releases/index

gitlab generic package registry

Another option that has been added to gitlab recently is the use of the Generic Packages Registry (https://gitlab.inria.fr/help/user/packages/generic_packages/index.html), part of the Gitlab Package Registry (https://gitlab.inria.fr/help/user/packages/index) that can be used to host various packages systems for a project (maven, pypi, npm,...).

In order to manually publish a binary release in such a registry you'll need to use the gitlab API, the procedure can roughly be summarized as follows:

  • enable "Packages" for the project, in Settings -> General -> Visibility, project features, permissions
  • create an authentication token (project access token, or personal/group access token; do not use a deploy token, they don't work (yet) with generic packages)
  • use you favorite tool to upload a file, e.g. with curl something like:
curl --header "PRIVATE-TOKEN: <token>" \
    --upload-file <local file to publish> \
    https://gitlab.inria.fr/api/v4/projects/<id>/packages/generic/<package name>/<version>/<name of published file>

where <id> is the project id as it appears just below the project name, on the main project page. Gitlab documentation says that the url-encoded path to the project can be used as well, but it does not work (yet).

The files will then appear in the Package Registry part of the project.

Static files storage

The static file storage which can be used for complete websites can also be used to store individual blobs like release tarballs or any static files.

see Forge Web Hosting / Static files storage

Forge News

No equivalent on gitlab

Forge Project Activity

The activity page of gitlab projects lists events such as pushes, merge request comments, issues.

Statistics

No direct equivalent in gitlab, but one should be able to compute many kind of custom statistics by using git itself (for the git repositories) or the gitlab API (for gitlab specific stuff).

Forge mediawiki

A wiki may be associated with each project in gitlab. The markup language is slightly different from mediawiki, though. When planning the migration of a wiki, you also need to handle attached files (images, etc.).

Forge project admins can open a support ticket asking for the following:

  • get a tgz with a full dump of a wiki (pages and attached files), including history
  • get a tgz with a full dump of a wiki (pages and attached files), without history (it's a dump of the current state of the wiki)
  • we have a tool for converting a full forge wiki to a GFM (github flavored markdown) / GLFM (gitlab flavored markdown, the format for gitlab wikis) wiki, including file attachements. The history is lost, only the current state of the forge wiki is converted. This tool is not 100% perfect, there are various corner cases which may be imperfectly handled, especially with links (to internal pages, attached files) or embedded images, and also rare minor text formatting glitches. Some links may need to be fixed manually, but in any case, all attached files are included in the converted wiki. A manual check pass is needed on the converted wiki. We can provide you a tgz of the converted wiki, that you can then directly put in a gitlab wiki git repository.

Forge Project Hierarchy

Gitlab groups are the alternative to Forge projects hierarchy, and they are actually more powerful and easy to use.

Forge Web Hosting

There are several INRIA alternatives to the forge web hosting, depending on the needs:

Wordpress hosting for INRIA Teams websites

https://team.inria.fr/

Wordpress hosting for projects in general (research projects, collaborations)

https://project.inria.fr/

Site addresses are in the form https://project.inria.fr/<sitename>. It supports virtual hosting, meaning that it can be configured to serve page for an alternate URL, in the inria domain (for example https://<project>.inria.fr or not (for example https://<project>.org). For non inria domains, you need to rent the domain name.

Static files storage

https://wiki.inria.fr/support/Espace_web

It is intended to store static files or web sites served with https. It is accessible with webdav, so it's possible to setup automated rsync to this place. The documentation says that it's intended only for static files needed by another website, but we had confirmation that it's officially supported to host full static web sites.

URLs are in the form https://files.inria.fr/<sitename>. It supports virtual hosting, meaning that it can be configured to serve page for an alternate URL, in the inria domain (for example https://<project>.inria.fr or not (for example https://<project>.org). For non inria domains, you need to rent the domain name.

To request a space on files.inria.fr, submit a ticket. Provide the name of the space you want, and the ildap logins of the people who should be able to have write access. To update this space, any webdav client can be used. For example, on linux + gnome you can access the space in the file manager with a location such as davs://<ildap user name>@files.inria.fr:8443/<space name>. You can also mount this space in the file system with tools such as davfs2 or gvfs-fuse. For example, on debian linux + gnome, install package gvfs-fuse, and when the space is "mounted" in the gnome file manager, it will also be mounted in the filesystem at location /run/user/<local unix user numeric id>/gvfs/dav:host=files.inria.fr,port=8443,ssl=true,user=<ildap login>/<space name>/. You can then rsync your forge site to the files.inria.fr site with a command such as: rsync --delete --exclude=.htupdate -rl <forge username>@forge-scm2:/home/groups/<forge unix project name>/htdocs/ /run/user/<local unix user numeric id>/gvfs/dav:host=files.inria.fr,port=8443,ssl=true,user=<ildap login>/<space name>/. If using option --delete make sure to not delete .htupdate since it controls who has write access. Also do not use -a, it doesn't work with webdav, use -rl instead.

gitlabpages

Gitlab includes gitlab-pages. Gitlab pages can only serve static content, so this means no PHP (unlike the forge) or database. For example it is not possible to host a dokuwiki (which is php based). The generation of the pages is done with the same mechanisms as gitlab continuous integration. Any static site generator can be used (and there are templates for many). Gitlab pages are served over HTTPS. A gitlab runner is needed to build gitlab pages. A gitlab runner is a server/VM/docker container running an agent and which can act as an executor for gitlab continuous integration jobs (and gitlab pages are a special case of gitlab continuous integration). As it is annoying for every project to setup a runner just for being able to publish a website, there is a work in progress to add shared runners to the inria gitlab. In the meantime, all projects requiring a runner for quick tasks like gitlab pages generation can ask to join the qlf-ci-gitlab-runner which provides such pseudo-shared docker runners.

See https://gitlab.inria.fr/siteadmin/doc/-/wikis/faq#gitlab-pages for more information, there is also an example project showing how to setup gitlabpages here: https://gitlab.inria.fr/siteadmin/pages-example

One point worth noting is that contrary to previously, INRIA has decided that a team website has URL .inria.fr and nothing except teams can have such inria subdomains. Therefore, it has been decided that custom domains / certificates are disabled in gitlab-pages. It means that it is not possible to alias a DNS name to a gitlab-pages website.

Other options

It is also possible to move forge websites to non-INRIA hosting.

maintaining old URLs pointing to resources on gforge web hosting

We are currently evaluating the possibility to offer the possibility to redirect the web hosting of forge projects to new URLs.

Forge Shell Access

Not available in gitlab

Using the forge API (SOAP)

The forge API ( https://fusionforge.org/soap/# ) can be used to help retrieve some data programmatically (list of users for a project, releases names and metadata, ...).

Examples in python (using zeep to help build the SOAP client: https://pypi.org/project/zeep/ )

# print names of users associated with a given project
from zeep import Client

client=Client("https://gforge.inria.fr/soap/index.php?wsdl")
session=client.service.login("<username here>","<password here>")

t=client.get_type("ns0:ArrayOfstring")
gid=client.service.getGroupsByName(session,t(["<insert project name here>"]))[0]["group_id"]
users=client.service.getGroupUsers(session,int(gid))
for user in users:
  print(user["firstname"],user["lastname"])
client.service.logout(session)
# list all releases of a given project
from zeep import Client

client=Client("https://gforge.inria.fr/soap/index.php?wsdl")
session=client.service.login("<username here>","<password here>")
t=client.get_type("ns0:ArrayOfstring")
group_id=int(client.service.getGroupsByName(session,t(["<insert project name here>"]))[0]["group_id"])
packages=client.service.getPackages(session,group_id)
for p in packages:
  print("# Package: {} (id={})".format(p["name"],p["package_id"]))
  releases=client.service.getReleases(session,group_id,p["package_id"])
  for r in releases:
    print("# Release: {} ({})".format(r["name"],time.ctime(r["release_date"])))
    print("https://gforge.inria.fr/frs/download.php/zip/{}/{}-{}.zip".format(r["release_id"],p["name"],r["name"]))

Be aware that the forge API is known to be hard to use, and prone to bugs. userGetGroups does not work at all (you'll get an error 500 if you try).

Alternatives to INRIA gitlab

  • If you really need a close equivalent to INRIA forge, RENATER SourceSup might be an good alternative. It is based on FusionForge (like INRIA forge), offers git and subversion hosting, and is currently actively maintained. https://sourcesup.renater.fr/ INRIA members can directly create accounts on this forge through the Renater identity federation (french research and education organizations). Non INRIA members belonging to an organization of this identity federation can also directly create accounts. External users who don't have an account in this identity federation (for example from a foreign country) can create a CRU account, then login through the identity federation. There are a few notable differences between INRIA forge and SourceSup:

    • There is no shell access on SourceSup
    • Thus, to import a subversion dump, one needs to ask SourceSup support
    • mailing lists manager is Sympa, not mailman. An interresting feature is the ability of the SourceSup Sympa forge plugin to automatically synchronise members of a mailing list with project members and/or administrators. This can be choosed at mailing list creation (but not sure if you can later change it)
    • There is no svn+ssh protocol for subversion, only webdav is offered

  • If you want to archive a public project (source code only, using git or subversion), you may use the Software Heritage portal, see https://save.softwareheritage.org