ALLGO containers

Overview

A minimal deployment of allgo consists of 6 docker images:

• allgo/redis : the redis application server
• allgo/django : the django application server
• allgo/mysql : the mysql database server
• allgo/controller: the manager for user docker containers
• allgo/ssh : the ssh frontend (giving access to the sandboxes)
• allgo/toolbox : an image containing a set of commands (scp, vi, nano, less, ...) to be mounted in the user sandboxes

These images may be deployed multiple times to implement multiple independent environments (eg: production, qualification, ...).

Additionally there are two images that are meant to be deployed only once (they may serve multiple environments)

• allgo/registry: the docker registry
• allgo/nginx: the frontal http server

There is an extra image used only in development:

• allgo/smtpsink: a SMTP server that catches and stores all incoming messages into a single mailbox

Each environment has its own docker network. The nginx container is connected to all these networks.

License

GNU AFFERO GENERAL PUBLIC LICENSE https://www.gnu.org/licenses/agpl-3.0.html

Conventions

All docker images use the following conventions.

External volumes

Their data are stored in:

• /vol/rw for persistent data
• /vol/ro for persistent data in read-only access
• /vol/cache for cache data (persistent data that may be destroyed at any time without any consequence)
• /vol/log for the logs

These paths are expected to be mounted as external volumes, either separately (typical for a production deployment) or with a single mount at /vol (typical for a development environment). The owner of external volumes must be the same as the uid used for the app inside the container.

Admin scripts

Each container may contain a set of scripts for admin purpose (especially for managing the content of external volumes)

• /dk/container_init initialise the content of the external volumes (eg: create and seed a database, write a default config, ...)
• /dk/image_upgrade apply security upgrades to the image. This command is expected to exit with 0 if successful and to output something on stdout/stderr when something was upgraded an nothing if nothing was upgraded (thus if the output is empty, it is not necessary to commit a new image).

Constraints

• The deployment repository has to be named allgo, because docker-compose names its network with <dir_name> prefix, and we used this network name in some parts.

Development environment

The development environment is managed with docker-compose. There are 2 important files:

• docker-compose.yml the docker-compose configuration
• bootstrap the bootstrap script

It provides 8 containers:

• dev-controller
• dev-mysql
• dev-nginx
• dev-docker
• dev-reddit
• dev-registry
• dev-smtpsink
• dev-ssh

All external volumes are stored in /data/dev/ (the path is absolute because it is tricky to use a relative path with the allgo/docker image).

For convenience, all containers not running as root (django, mysql, registry) have their user overridden to the UID:GID of the developer running docker-compose. This is managed with the DOCKERUSER environment variable set in the .env file by prepare.sh.

For convenience (again), there is an extra external volume for dev-django, dev-controller and dev-ssh so that the source directory of the app is mounted inside /opt/ (in fact it overrides the actual application files provided by the docker image). The purpose is to avoid rebuilding a new docker image for each development iteration.

Getting started

The sources are located in one repository:

• allgo: the deployment repository

To set up the development environment, run:

1. get the sources

   git clone git@gitlab.inria.fr:allgo/allgo.git
   cd allgo
   

2. (as root) create /data/dev and make it owned by the developer

   sudo mkdir -p    /data/dev
   sudo chown $USER: /data/dev
   

3. bootstrap the environment

   ./bootstrap
   

   This command will run the /dk/init_container in every container that needs it, then start the container.

   The first run takes a few minutes because all images are built from scratch. You may have enough time for a short coffee break.

   Note by default bootstrap works on all containers. It is possible to give an explicit list of containers instead. Example:

   	./bootstrap dev-mysql dev-django
   

4. for convenience, you may want to alias docker-compose as fig (because fig is much faster to type than docker-compose and you will have to type it a lot). Somewhere in your .bashrc you should add:

   alias fig=docker-compose
   

5. after bootstrap, in development the db is initialised with three users (all with the password allgo):

   • admin@localhost which is superuser
   • devel@localhost which is the owner of a webapp named sleep
   • guest@localhost

Common commands

The official doc for docker-compose is available at: https://docs.docker.com/compose/

• start all containers (in the background)

   fig up -d
   

• start all containers (in the foreground, i.e interactively, when you hit Ctrl-C all containers are stop)

   fig up
   

• soft cleanup (stop and remove all containers)

   fig down
   

• hard cleanup (remove images too)

   fig down --rmi local
   

• restart a container

   fig restart dev-django
   

• restart a container using a new docker image (if the image has been rebuilt since the last start)

   fig up dev-django
   

• rebuild an image

   fig build dev-django
   

• Note: most commands work on every container by default (eg: up down start stop restart ...) they can be use on an individual container too:

   fig restart dev-controller dev-django
   

• run a container with an arbitrary command (eg: to have access to the django console)

   fig run --rm dev-django bash
   

  Note: containers created by fig run have the same parameters as the referenced containers but their name is different (eg: allgo_dev-ssh_run_1), which means that this container is not reachable by the others (this may be an issue for example if you want to run the mysqld server manually: fig run dev-mysql mysqld -> this container won't be reachable by the ssh and django containers)

• follow the output of all containers:

   fig logs --tail=1 --follow
   

Troubleshooting

Fedora OS: docker.sock usable by root user only

The docker socket /run/docker.sock for interacting with the docker daemon is owned by root:root with 0600 permissions. Only the root user can use it

To use it as an ordinary user, create the docker group, then the docker engine will automatically make its socket owned by root:docker. Users who are members of the docker group will be able to use it.

To do so you must run the following commands :

sudo groupadd docker

Edit the group to add the relevant users to the docker group (add them at the end of the line starting with docker:, separated by commas)

sudo vigr

Re-open your session (to be effectively in the docker group)

dev-mysql initialisation fails

The prepare.sh script (also sourced by the ./bootstrap script) creates the /data/dev/*/ with the ownership set to the calling user. Be sure not to run it as root, otherwise it will be owned by root and you may have errors like:

mkdir: cannot create directory ‘/vol/rw’: Permission denied
mkdir: cannot create directory ‘/vol/log’: Permission denied

If somehow you skipped this step, you can reset the ownership to the current user:

sudo chown    $USER: /data/dev
sudo chown -R $USER: /data/dev/{registry,mysql,django}

If you are completely lost, you can just restart the initialisation from scratch:

fig down
sudo rm -rf /data/dev
. prepare.sh
./bootstrap

Production environment

• TODO unicorn/nginx integration
• TODO use capistrano too ?

Images design

registry

Hosts docker registry with a nginx configured as a reverse proxy. It listens to 4 ports:

• :8000 (production) access limited to /v2/allgo/prod/
• :8001 (qualification) access limited to /v2/allgo/qualif/
• :8002 (development) access limited to /v2/allgo/dev
• :5000 (legacy production) access limited to /v2/allgo(which is mapped to /v2/allgo/prod/)

mysql

Hosts a mysql server listening on port 3306 with two databases: allgo and allgo_test and two users: allgo and ssh.

• allgo has read/write access to both databases
• ssh has read only access to allgo

django

Hosts three daemons for running the allgo web server:

• a nginx frontend for buffering the HTTP requests/responses and routing them to the other daemons. It also serves static files directly
• the gunicorn server (running the django application)
• the allgo.aio server (serving the asynchronous requests)

Running the django server manually

TODO ?

[comment]: # ( - run the dev-rails container and open a shell:

fig up -d
docker exec -t -i  dev-rails bash

• then, inside the container, run:

   supervisorctl stop rails
   rails server
   

)

ssh

Hosts the ssh front server for accessing the sandboxes (ssh WEBAPP@sid.allgo.irisa.fr). Each allgo webapp is mapped to a system user (using Glibc NSS) starting at uid 2000.

• /etc/passwd and /etc/group are overriden so as to contain only the two users (root and sshd) and one group (nogroup) required to run the ssh server

• Extra users are obtained from the mysql database (using libnss-mysql-bg) and mapped as follows:

    name  = webapps.docker_name
    uid   = webapps.id
    gid   = 65534 (nogroup)
    gecos = webapps.name
    shell = /bin/allgo-shell
    

• The ssh server is configured to accept key-based authentication only. The list of public keys is obtained from the (using an AuthorizedKeysCommand).

• The forced shell (allgo-shell) connects to the webapp sandbox (if running).

• The connection to the sandbox is made though a unix socket and a set of pipes in the filesystem.

controller

Hosts the docker-controller which manages all docker operations (run, stop, rm, commit, pull, push, ...) on behalf of the django container.

Technically speaking this container has root privileges since it has access to the docker socket.

The proxy script enforces restrictions (according to the current environment: eg prod/qualif/dev) on:

• the registry (for pulling/pushing)
• the paths of external volumes
• the container names (ENV-user-XXXX)

nginx

Hosts the frontal nginx server, its purpose is to:

• give access to one or more allgo instances
• manage TLS encryption

smtpsink

Hosts a SMTP server (port 25) and an IMAP server (port 143) for development/qualification

Its purpose is to channel all outgoing mail (received on port 25) into a single mailbox.

The mailbox is accessible with IMAP as user sink (password sink).

NOTE: in the development environment, django's default is to dump outgoing e-mails to the console. Thus this container is only useful in the qualif setup.