Commit e05a61d3 authored by BAIRE Anthony's avatar BAIRE Anthony
Browse files

initial docker setup

PREFIX = allgo/
ALLGO containers
A minimal deployment of allgo consists of 4 docker images:
- **allgo/rails**: the rails application server
- **allgo/mysql**: the mysql database server
- **allgo/docker**: the manager for user docker containers
- **allgo/ssh**: the ssh frontend (giving access to the sandboxes)
These images may be deployed multiple times to implement multiple independent
environments (eg: production, qualification, ...).
Additionally there are two images that are meant to be deployed only once (they
may serve multiple environments)
- **allgo/registry**: the docker registry
- **allgo/nginx**: the frontal http server
There is an extra image used only in development:
- **allgo/smtpsink**: a SMTP server that catches and stores all incoming messages into a single mailbox
Each environment has its own docker network. The nginx container is connected
to all these networks to that it can connect to the rails servers.
All docker images use the following conventions.
### External volumes
They data is stored in:
- `/vol/rw` for persistent data
- `/vol/ro` for persistent data in read-only access
- `/vol/cache` for cache data (persistent data that may be destroyed at any time without any consequence)
- `/vol/log` for the logs
These paths are expected to be mounted as external volumes, either separately (typical for a production deployment) or with a single mount at `/vol` (typical for a development environment). The owner of external volumes must be the same as the uid used for the app inside the container.
### Admin scripts
Each container may contain a set of scripts for admin purpose (especially for managing the content of external volumes)
- `/dk/container_init` initialise the content of the external volumes (eg: create and seed a database, write a default config, ...)
- `/dk/image_upgrade` apply security upgrades to the image. This command is expected to exit with 0 if successful and to output something on stdout/stderr when something was upgraded an nothing if nothing was upgraded (thus if the output is empty, it is not necessary to commit a new image).
Development environment
The development environment is managed with docker-compose. There are 3 important files:
- `docker-compose.yml` the docker-compose configuration
- `` a shell script which is meant to be sourced in the shell before
working with the environment
- `bootstrap` the bootstrap script
It provides 7 containers:
- `dev-docker`
- `dev-mysql`
- `dev-nginx`
- `dev-rails`
- `dev-registry`
- `dev-smtpsink`
- `dev-ssh`
All external volumes are stored in `/data/dev/` (the path is absolute because
it is tricky to use a relative path with the allgo/docker image).
For convenience, all containers not running as root (rails, mysql, registry)
have their user overridden to the UID:GID of the developer running
docker-compose. This is managed with the `DOCKER_USER` environment variable set
by ``.
For convenience (again), there is an extra external volumes for `dev-rails`,
`dev-docker` and `dev-ssh` so that the source directory of the app is mounted
inside `/opt/` (in fact it overrides the actual application files provided by
the docker image). The purpose is to avoid rebuilding a new docker image for
each development iteration.
### Getting started
The sources are located in two repositories:
- *rails-allgo*: the rails application repository
- *allgo*: the deployment repository
To set up the development environment, run:
1. get the sources
git clone
git clone
cd allgo
**Note:** the *allgo* repository has a symbolic link to the
*rails-allgo* repository. They should be cloned at the same place (or
the link will be broken).
2. *(as root)* create `/data/dev` and make it owned by the developer
sudo mkdir -p /data/dev
sudo chown USER: /data/dev
3. source the `` script (to be run every time before you start working)
4. bootstrap the environment
This command will run the `/dk/init_container` in every container that
needs it, then start the container.
The first run takes a very long time because all images are built from
scratch (especially the rails image which builds ruby source).
You have enough time for a coffee break.
**Note** by default `bootstrap` works on all containers. It is possible
to give an explicit list of containers instead. Example:
./bootstrap dev-mysql dev-rails
5. for convenience, you may want to alias `docker-compose` as `fig` (because
`fig` is much faster to type than `docker-compose` and you will have to
type it a lot). Somewhere in your `.bashrc` you should add:
alias fig=docker-compose
### Common commands
The official doc for docker-compose is available at: []()
- start all containers (in the background)
fig up -d
- start all containers (in the foreground, i.e interactively, when you hit Ctrl-C all containers are stop)
fig up -d
- soft cleanup (stop and remove all containers)
fig down
- hard cleanup (remove images too)
fig down --rmi local
- restart a container
fig restart dev-rails
- restart a container using a new docker image (if the image has been rebuilt since the last start)
fig up dev-rails
- rebuild an image
fig build dev-railf
- **Note:** most commands work on every container by default (eg: up down
start stop restart ...) they can be use on an individual container too:
fig restart dev-docker dev-rails
- run a container with an arbitrary command (eg: to have access to the rails console)
fig run dev-rails bash
**Note:** containers created by `fig run` have the same parameters as
the referenced containers but their name is different (eg:
*allgo_dev-ssh_run_1*), which means that this container is not
reachable by the others (this may be an issue for example if you want
to run the mysqld server manually: `fig run dev-mysql mysqld` -> this
container won't be reachable by the ssh and rails containers)
- follow the output of all containers:
fig logs --tail=1 --follow
Production environment
- TODO unicorn/nginx integration
- TODO use capistrano too ?
Images design
## registry
Hosts docker registry with a nginx configured as a reverse proxy. It listens to 4 ports:
- :8000 (*production*) access limited to `/v2/allgo/prod/`
- :8001 (*qualification*) access limited to `/v2/allgo/qualif/`
- :8002 (*development*) access limited to `/v2/allgo/dev`
- :5000 (*legacy production*) access limited to `/v2/allgo`(which is mapped to `/v2/allgo/prod/`)
## mysql
Hosts a mysql server listening on port 3306 with two databases: `allgo` and
`allgo_test` and two users: `allgo` and `ssh`.
- `allgo` has read/write access to both databases
- `ssh` has read only access to `allgo`
## rails
Hosts three daemons for running allgo:
- the rails server
- the sidekiq queue manager
- the redis db server
(TODO add unicorn+nginx for production)
## ssh
Hosts the ssh front server for accessing the sandboxes (`ssh`). Each allgo webapp is mapped to a system user
(using Glibc NSS) starting at uid 2000.
- `/etc/passwd` and `/etc/group` are overriden so as to contain only the two users (*root* and *sshd*) and one group (*nogroup*) required to run the ssh server
- Extra users are obtained from the mysql database (using libnss-mysql-bg) and mapped as follows:
name = webapps.docker_name
uid =
gid = 65534 (nogroup)
gecos =
shell = /bin/allgo-shell
- The ssh server is configured to accept key-based authentication only. The
list of public keys is obtained from the (using an AuthorizedKeysCommand).
- The forced shell (`allgo-shell`) connects to the webapp sandbox (if running).
- The connection to the sandbox is made though a unix socket and a set of pipes
in the filesystem.
## docker
Hosts the *docker-allgo-proxy* which manages all docker operations (run, stop,
rm, commit, pull, push, ...) on behalf of the rails container.
Technically speaking this container had root privileges since it has access to
the docker socket.
The proxy script enforces restrictions (according to the current environment: eg prod/qualif/dev) on:
- the registry (for pulling/pushing)
- the paths of external volumes
- the container names (*ENV*-user-*XXXX*)
## nginx
Hosts the frontal nginx server, its purpose is to:
- give access to one or more allgo instances
- manage TLS encryption
## smtpsink
Hosts a SMTP server (port 25) and an IMAP server (port 143) for
Its purpose is to channel all outgoing mail (received on port 25) into a single
The mailbox is accessible with IMAP as user *sink* (password *sink*).
FROM debian:jessie
ADD apt-getq /usr/local/bin/
RUN sed -i 's/; s/main *$/main contrib non-free/' /etc/apt/sources.list &&\
apt-getq update &&\
apt-getq dist-upgrade &&\
apt-getq install vim-tiny locales patch realpath logrotate python3
COPY . /tmp/context
RUN ["sh", "/tmp/context/"]
set -e
for p in "$@"
echo "`basename $p`"
patch -p0 <"$p"
export DEBIAN_FRONTEND=noninteractive
exec apt-get -qq -y --no-install-recommends "$@"
cmd="`basename $0`"
cat >&2 <<EOF
usage: $cmd show list existing diversions
$cmd add ORIG DEST add a diversion (WARNING: erases ORIG)
$cmd rm ORIG remove a diversion
exit 1
echo "error: $*" >&2
exit 1
# load the current diversions
if [ -e "$ETC" ] ; then
grep -q "^declare -g -A div=" "$ETC" || die "invalid config file $ETC"
. "$ETC"
declare -g -A div=()
declare -p div > "$ETC.tmp"
sed -i "s/^declare/declare -g/" "$ETC.tmp"
mv "$ETC.tmp" "$ETC"
local value="${div["$1"]}"
[ -n "$value" ] || die "unknown key: $1"
echo "$value"
[[ "$1" =~ ^/ ]] || die "path must be absolute: $1"
touch "$1" # create the file in case it does not exist yet
local rp="`realpath "$1"`"
[ "$1" = "$rp" ] || [ "$1" = "$rp/" ] || die "path is not a real path: $1 (resolves to $rp)"
[ "$rp" != "/" ] || die "you should never try to divert /"
echo "$rp"
[[ "$1" =~ ^/ ]] || die "path must be absolute: $1"
echo "$1"
local key="$1"
local value="`get "$1"`"
rm -rf "$key"
ln -s "$value" "$key"
local key="$1"
if [ -L "$key" ] ; then
rm "$key"
[ ! -e "$key" ] || die "unable to disable diversion: $key is not a symbolic link"
set -e
case "$CMD" in
[ $# = 0 ] || print_help
printf "%-32s %s\n" "PATH" "DIVERTED TO"
for k in ${!div[@]}
printf "%-32s %s\n" "$k" "${div["$k"]}"
[ $# = 2 ] || print_help
KEY="`validate_key "$1"`"
VALUE="`validate_value "$2"`"
[ -z "$previous" ] || die "$KEY is already diverted to $previous"
enable_diversion "$KEY"
[ $# = 1 ] || print_help
VALUE="`get "$KEY"`"
unset div["$KEY"]
disable_diversion "$KEY"
[ $# = 0 ] || print_help
for k in ${!div[@]}
"$CMD"_diversion "$k"
set -e
for CANDIDATE in /vol/ro/logrotate.d /dk/logrotate.d
if [ -d "$CANDIDATE" ]
cat > "$TMPFILE" <<EOF
# see "man logrotate" for details
# rotate log files weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
# uncomment this if you want your log files compressed
# packages drop log rotation information into this directory
include $CANDIDATE
logrotate "$TMPFILE" -s "$STATEFILE"
exit 0
# container upgrade script
# The output must be empty when no upgrade is needed.
set -e
diversions disable
apt-getq update
apt-getq upgrade -u
# restore our symbolic links in case they are overwritten by a package
diversions enable
import atexit, collections, os, subprocess, sys, traceback
def die(fmt, *k):
sys.stderr.write("error:%s\n" % (fmt % k))
class Failure(Exception):
class migration:
__instances = collections.defaultdict(lambda: {})
def __init__(self, previous, next):
self.previous = previous = next
self.func = None
assert next not in self.__instances[previous], ("duplicated migration: %s -> %s" % key)
self.__instances[previous][next] = self
def __call__(self, func):
assert self.func is None, "migration function already set"
self.func = func
def __str__(self):
return "migration %r -> %r" % (self.previous,
def run(cls, from_version, to_version):
# find the shortest migration path for each version
# version -> (migration, ...)
routes = {from_version: ()}
queue = collections.deque([from_version])
while queue:
version = queue.popleft()
for migration in cls.__instances[version].values():
assert migration.previous == version
if not in routes:
routes[] = routes[migration.previous] + (migration,)
assert len(routes[]) <= len(routes[migration.previous]) + 1
fmt_route = lambda r: from_version + "".join(" -> %s" % for m in r)
if routes:
print ("possible migrations from version %s" % from_version)
for version in sorted(routes):
print(" %-8s %s" % (version, fmt_route(routes[version])))
route = routes.get(to_version)
if not route:
die("no migration path from version %s to %s", from_version, to_version)
print("migration path: %s" % fmt_route(route))
for migration in route:
"┃ applying %-33s ┃\n"
"┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛" % migration)
# flush stdout/stderr because we may have some interleaving
except subprocess.CalledProcessError as e:
sys.stderr.write("error: %s\n" % e)
die("migration %s failed", migration)
except SystemExit as e:
if e.code:
die("migration %s failed", migration)
except Exception as e:
die("unexpected error in migration %s", migration)
def run():
if len(sys.argv) != 3:
die("usage: %s SRC_VERSION DST_VERSION", sys.argv[0])*sys.argv[1:3])