initial docker setup

.gitignore

+.*.sw[op]
+.stamp.*
+.deps.*

Makefile

+
+PREFIX = allgo/
+
+include docker.mk
+
+
+

README.md

+ALLGO containers
+================
+
+
+Overview
+--------
+
+A minimal deployment of allgo consists of 4 docker images:
+
+- **allgo/rails**: the rails application server 
+- **allgo/mysql**: the mysql database server
+- **allgo/docker**: the manager for user docker containers
+- **allgo/ssh**: the ssh frontend (giving access to the sandboxes)
+
+These images may be deployed multiple times to implement multiple independent
+environments (eg: production, qualification, ...).
+
+Additionally there are two images that are meant to be deployed only once (they
+may serve multiple environments)
+
+- **allgo/registry**: the docker registry
+- **allgo/nginx**: the frontal http server
+
+There is an extra image used only in development:
+
+- **allgo/smtpsink**: a SMTP server that catches and stores all incoming messages into a single mailbox
+
+Each environment has its own docker network. The nginx container is connected
+to all these networks to that it can connect to the rails servers.
+
+
+Conventions
+-----------
+
+All docker images use the following conventions.
+
+### External volumes
+
+They data is stored in:
+
+  - `/vol/rw` for persistent data
+  - `/vol/ro` for persistent data in read-only access
+  - `/vol/cache` for cache data (persistent data that may be destroyed at any time without any consequence)
+  - `/vol/log` for the logs
+
+These paths are expected to be mounted as external volumes, either separately (typical for a production deployment) or with a single mount at `/vol` (typical for a development environment). The owner of external volumes must be the same as the uid used for the app inside the container.
+
+### Admin scripts
+
+Each container may contain a set of scripts for admin purpose (especially for managing the content of external volumes)
+
+  - `/dk/container_init` initialise the content of the external volumes (eg: create and seed a database, write a default config, ...)
+  - `/dk/image_upgrade` apply security upgrades to the image. This command is expected to exit with 0 if successful and to output something on stdout/stderr when something was upgraded an nothing if nothing was upgraded (thus if the output is empty, it is not necessary to commit a new image). 
+
+
+
+Development environment
+-----------------------
+
+The development environment is managed with docker-compose. There are 3 important files:
+
+ - `docker-compose.yml` the docker-compose configuration
+ - `prepare.sh` a shell script which is meant to be sourced in the shell before
+   working with the environment
+ - `bootstrap` the bootstrap script
+
+It provides 7 containers:
+
+ - `dev-docker`
+ - `dev-mysql`
+ - `dev-nginx`
+ - `dev-rails`
+ - `dev-registry`
+ - `dev-smtpsink`
+ - `dev-ssh`
+
+All external volumes are stored in `/data/dev/` (the path is absolute because
+it is tricky to use a relative path with the allgo/docker image).
+
+For convenience, all containers not running as root (rails, mysql, registry)
+have their user overridden to the UID:GID of the developer running
+docker-compose. This is managed with the `DOCKER_USER` environment variable set
+by `prepare.sh`.
+
+For convenience (again), there is an extra external volumes for `dev-rails`,
+`dev-docker` and `dev-ssh` so that the source directory of the app is mounted
+inside `/opt/` (in fact it overrides the actual application files provided by
+the docker image). The purpose is to avoid rebuilding a new docker image for
+each development iteration. 
+
+
+### Getting started
+
+The sources are located in two repositories:
+
+ - *rails-allgo*: the rails application repository
+ - *allgo*: the deployment repository
+
+To set up the development environment, run: 
+
+ 1. get the sources
+ 	<pre>
+	git clone git@gitlab.irisa.fr:charly.maupetit/rails-allgo.git
+	git clone git@gitlab.irisa.fr:sebastien.campion/allgo.git
+	cd allgo
+	</pre>
+	**Note:** the *allgo* repository has a symbolic link to the
+	*rails-allgo* repository. They should be cloned at the same place (or
+	the link will be broken).
+
+ 2. *(as root)* create `/data/dev` and make it owned by the developer
+	<pre>
+	sudo mkdir -p    /data/dev
+	sudo chown USER: /data/dev
+	</pre>
+
+ 3. source the `prepare.sh` script (to be run every time before you start working)
+ 	<pre>
+	. prepare.sh
+	</pre>
+
+ 4. bootstrap the environment
+ 	<pre>
+	./bootstrap
+	</pre>
+	This command will run the `/dk/init_container` in every container that
+	needs it, then start the container.
+
+	The first run takes a very long time because all images are built from
+	scratch (especially the rails image which builds ruby source).
+	You have enough time for a coffee break.
+
+	**Note** by default `bootstrap` works on all containers. It is possible
+	to give an explicit list of containers instead. Example:
+	<pre>
+		./bootstrap dev-mysql dev-rails
+	</pre>
+
+ 5. for convenience, you may want to alias `docker-compose` as `fig` (because
+     `fig` is much faster to type than `docker-compose` and you will have to
+     type it a lot). Somewhere in your `.bashrc` you should add:
+	<pre>
+	alias fig=docker-compose
+	</pre>
+
+
+### Common commands
+
+The official doc for docker-compose is available at: [https://docs.docker.com/compose/
+https://docs.docker.com/compose/]()
+
+ - start all containers (in the background)
+	<pre>
+	fig up -d
+	</pre>
+
+ - start all containers (in the foreground, i.e interactively, when you hit Ctrl-C all containers are stop)
+	<pre>
+	fig up -d
+	</pre>
+
+ - soft cleanup (stop and remove all containers)
+	<pre>
+	fig down
+	</pre>
+
+ - hard cleanup (remove images too)
+ 	<pre>
+	fig down --rmi local
+	</pre>
+
+ - restart a container
+	<pre>
+	fig restart dev-rails
+	</pre>
+
+ - restart a container using a new docker image (if the image has been rebuilt since the last start)
+ 	<pre>
+	fig up dev-rails
+	</pre>
+
+ - rebuild an image
+ 	<pre>
+	fig build dev-railf
+	</pre> 
+
+ - **Note:** most commands work on every container by default (eg: up down
+   start stop restart ...) they can be use on an individual container too:
+   	<pre>
+	fig restart dev-docker dev-rails
+	</pre>
+
+ - run a container with an arbitrary command (eg: to have access to the rails console)
+ 	<pre>
+	fig run dev-rails bash
+	</pre>
+
+	**Note:** containers created by `fig run` have the same parameters as
+	the referenced containers but their name is different (eg:
+	*allgo_dev-ssh_run_1*), which means that this container is not
+	reachable by the others (this may be an issue for example if you want
+	to run the mysqld server manually: `fig run dev-mysql mysqld` -> this
+	container won't be reachable by the ssh and rails containers)
+
+ - follow the output of all containers:
+ 	<pre>
+	fig logs --tail=1 --follow
+	</pre>
+
+Production environment
+----------------------
+
+- TODO unicorn/nginx integration
+- TODO use capistrano too ?
+
+
+Images design
+-------------
+
+
+## registry
+
+Hosts docker registry with a nginx configured as a reverse proxy. It listens to 4 ports:
+
+ - :8000 (*production*) access limited to `/v2/allgo/prod/`
+ - :8001 (*qualification*) access limited to `/v2/allgo/qualif/`
+ - :8002 (*development*)  access limited to `/v2/allgo/dev`
+ - :5000 (*legacy production*) access limited to `/v2/allgo`(which is mapped to `/v2/allgo/prod/`)
+
+## mysql
+
+Hosts a mysql server listening on port 3306 with two databases: `allgo` and
+`allgo_test` and two users: `allgo` and `ssh`.
+
+ - `allgo` has read/write access to both databases
+ - `ssh` has read only access to `allgo`
+
+
+## rails
+
+Hosts three daemons for running allgo:
+
+ - the rails server
+ - the sidekiq queue manager
+ - the redis db server
+
+(TODO add unicorn+nginx for production)
+
+## ssh
+
+Hosts the ssh front server for accessing the sandboxes (`ssh
+WEBAPP@sid.allgo.irisa.fr`). Each allgo webapp is mapped to a system user
+(using Glibc NSS) starting at uid 2000.
+
+- `/etc/passwd` and `/etc/group` are overriden so as to contain only the two users (*root* and *sshd*) and one group (*nogroup*) required to run the ssh server
+- Extra users are obtained from the mysql database (using libnss-mysql-bg) and mapped as follows:
+	<pre>
+	name  = webapps.docker_name
+	uid   = webapps.id
+	gid   = 65534 (nogroup)
+	gecos = webapps.name
+	shell = /bin/allgo-shell
+  	</pre>
+- The ssh server is configured to accept key-based authentication only. The
+  list of public keys is obtained from the (using an AuthorizedKeysCommand).
+
+- The forced shell (`allgo-shell`) connects to the webapp sandbox (if running).
+
+- The connection to the sandbox is made though a unix socket and a set of pipes
+  in the filesystem.
+
+## docker
+
+Hosts the *docker-allgo-proxy* which manages all docker operations (run, stop,
+rm, commit, pull, push, ...) on behalf of the rails container.
+
+Technically speaking this container had root privileges since it has access to
+the docker socket.
+
+The proxy script enforces restrictions (according to the current environment: eg prod/qualif/dev) on:
+
+ - the registry (for pulling/pushing)
+ - the paths of external volumes 
+ - the container names (*ENV*-user-*XXXX*)
+
+## nginx
+
+Hosts the frontal nginx server, its purpose is to:
+- give access to one or more allgo instances
+- manage TLS encryption
+
+## smtpsink
+
+Hosts a SMTP server (port 25) and an IMAP server (port 143) for
+development/qualification
+
+Its purpose is to channel all outgoing mail (received on port 25) into a single
+mailbox.
+
+The mailbox is accessible with IMAP as user *sink* (password *sink*).
+

base-debian/.dockerignore

+Dockerfile*
+.git
+.*.swo
+.*.swp

base-debian/Dockerfile

+FROM debian:jessie
+
+ADD 	apt-getq /usr/local/bin/
+RUN	sed -i 's/httpredir.debian.org/miroir.irisa.fr/; s/main *$/main contrib non-free/' /etc/apt/sources.list &&\
+	apt-getq update				&&\
+	apt-getq dist-upgrade			&&\
+	apt-getq install vim-tiny locales patch realpath logrotate python3
+
+ENV	LANG en_US.UTF-8
+
+COPY	. /tmp/context
+RUN	["sh", "/tmp/context/setup.sh"]

base-debian/apply-patches

+#!/bin/sh
+
+set -e
+
+for p in "$@"
+do
+	echo "`basename $p`"
+	patch -p0 <"$p"
+done
+

base-debian/apt-getq

+#!/bin/sh
+
+export DEBIAN_FRONTEND=noninteractive
+
+exec apt-get -qq -y --no-install-recommends "$@"

base-debian/diversions

+#!/bin/bash
+
+ETC="/etc/local-diversions"
+
+print_help()
+{
+	cmd="`basename $0`"
+	cat >&2 <<EOF
+usage:	$cmd show                 list existing diversions
+	$cmd add ORIG DEST        add a diversion (WARNING: erases ORIG)
+	$cmd rm  ORIG             remove a diversion
+EOF
+	exit 1
+}
+
+die()
+{
+	echo "error: $*" >&2
+	exit 1
+}
+
+load()
+{
+	# load the current diversions
+	if [ -e "$ETC" ] ; then
+		grep -q "^declare -g -A div=" "$ETC" || die "invalid config file $ETC"
+		. "$ETC"
+	else
+		declare -g -A div=()
+	fi
+}
+
+save()
+{
+	declare -p div > "$ETC.tmp"
+	sed -i "s/^declare/declare -g/" "$ETC.tmp"
+	mv "$ETC.tmp" "$ETC"
+}
+
+get()
+{
+	local value="${div["$1"]}"
+	[ -n "$value" ] || die "unknown key: $1"
+	echo "$value"
+}
+
+validate_key()
+{
+	[[ "$1" =~ ^/ ]] || die "path must be absolute: $1"
+
+	touch "$1"	# create the file in case it does not exist yet
+
+	local rp="`realpath "$1"`"
+
+	[ "$1" = "$rp" ] || [ "$1" = "$rp/" ] || die "path is not a real path: $1 (resolves to $rp)"
+
+	[ "$rp" != "/" ] || die "you should never try to divert /"
+	echo "$rp"
+}
+validate_value()
+{
+	[[ "$1" =~ ^/ ]] || die "path must be absolute: $1"
+	echo "$1"
+}
+
+
+
+enable_diversion()
+{
+	local key="$1"
+	local value="`get "$1"`"
+	
+	rm -rf "$key"
+	ln -s "$value" "$key"
+}
+
+disable_diversion()
+{
+	local key="$1"
+	
+	if [ -L "$key" ] ; then
+		rm "$key"
+	else
+		[ ! -e "$key" ] || die "unable to disable diversion: $key is not a symbolic link"
+	fi
+}
+
+
+
+set -e
+
+
+load
+
+CMD="$1"
+shift
+
+case "$CMD" in
+show)
+	[ $# = 0 ] || print_help
+
+	printf "%-32s %s\n" "PATH" "DIVERTED TO"
+	for k in ${!div[@]}
+	do
+		printf "%-32s %s\n" "$k" "${div["$k"]}"
+	done
+	;;
+
+add)
+	[ $# = 2 ] || print_help
+	KEY="`validate_key "$1"`"
+	VALUE="`validate_value "$2"`"
+
+	previous="${div[$KEY]}"
+	[ -z "$previous" ] || die "$KEY is already diverted to $previous"
+
+	div["$KEY"]="$VALUE"
+	enable_diversion "$KEY"
+	save
+	;;
+
+rm)
+	[ $# = 1 ] || print_help
+	KEY="$1"
+	VALUE="`get "$KEY"`"
+
+	unset div["$KEY"]
+	disable_diversion "$KEY"
+	save
+	;;
+
+enable|disable)
+	[ $# = 0 ] || print_help
+
+	for k in ${!div[@]}
+	do
+		"$CMD"_diversion "$k"
+	done
+	;;
+*)
+	print_help
+	;;
+esac

base-debian/dk/container_cron_daily_root.logrotate

+#!/bin/sh
+
+set -e
+
+STATEFILE="/vol/log/.logrotate.status"
+
+for CANDIDATE in  /vol/ro/logrotate.d  /dk/logrotate.d
+do
+	if [ -d "$CANDIDATE" ]
+	then
+		TMPFILE="`tempfile`"
+		cat > "$TMPFILE" <<EOF
+# see "man logrotate" for details
+# rotate log files weekly
+weekly
+
+# keep 4 weeks worth of backlogs
+rotate 4
+
+# create new (empty) log files after rotating old ones
+create
+
+# uncomment this if you want your log files compressed
+#compress
+
+# packages drop log rotation information into this directory
+include $CANDIDATE
+EOF
+		logrotate "$TMPFILE" -s "$STATEFILE"
+		rm "$TMPFILE"
+		exit 0
+	fi
+done

base-debian/dk/image_upgrade_root

+#!/bin/sh
+#
+# container upgrade script
+#
+# The output must be empty when no upgrade is needed.
+#
+
+set -e
+
+diversions disable
+
+apt-getq update
+apt-getq upgrade -u
+
+# restore our symbolic links in case they are overwritten by a package
+diversions enable
+

base-debian/dk_migration.py

+#!/usr/bin/python3
+
+import atexit, collections, os, subprocess, sys, traceback
+
+def die(fmt, *k):
+    sys.stderr.write("error:%s\n" % (fmt % k))
+    sys.exit(1)
+
+class Failure(Exception):
+    pass
+
+class migration:
+    __instances = collections.defaultdict(lambda: {})
+
+    def __init__(self, previous, next):
+        self.previous = previous
+        self.next     = next
+        self.func     = None
+
+        assert next not in self.__instances[previous], ("duplicated migration: %s -> %s" % key)
+
+        self.__instances[previous][next] = self
+
+    def __call__(self, func):
+        assert self.func is None, "migration function already set"
+        self.func = func
+
+    def __str__(self):
+        return "migration %r -> %r" % (self.previous, self.next)
+
+    @classmethod
+    def run(cls, from_version, to_version):
+        # find the shortest migration path for each version
+
+        # version -> (migration, ...)
+        routes = {from_version: ()}
+
+        queue = collections.deque([from_version])
+        while queue:
+            version = queue.popleft()
+            for migration in cls.__instances[version].values():
+                assert migration.previous == version
+                if migration.next not in routes:
+                    routes[migration.next] = routes[migration.previous] + (migration,)
+                    queue.append(migration.next)
+                else:
+                    assert len(routes[migration.next]) <= len(routes[migration.previous]) + 1
+
+        fmt_route = lambda r: from_version + "".join(" -> %s" % m.next for m in r)
+
+        if routes:
+            print ("possible migrations from version %s" % from_version)
+            for version in sorted(routes):
+                print("    %-8s    %s" % (version, fmt_route(routes[version])))
+            print()
+
+        route = routes.get(to_version)
+        if not route:
+            die("no migration path from version %s to %s", from_version, to_version)
+
+        print("migration path: %s" % fmt_route(route))
+
+        for migration in route:
+            print("\n"
+                  "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n"
+                  "┃ applying %-33s ┃\n"
+                  "┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛" % migration) 
+            try:
+                # flush stdout/stderr because we may have some interleaving
+                sys.stdout.flush()
+                sys.stderr.flush()
+                migration.func()
+            except subprocess.CalledProcessError as e:
+                sys.stderr.write("error: %s\n" % e)
+                die("migration %s failed", migration)
+            except SystemExit as e:
+                if e.code:
+                    die("migration %s failed", migration)
+            except Exception as e:
+                traceback.print_exc()
+                die("unexpected error in migration %s", migration)
+
+def run():
+    atexit.unregister(_guard)
+    if len(sys.argv) != 3:
+        die("usage: %s SRC_VERSION  DST_VERSION", sys.argv[0])
+    migration.run(*sys.argv[1:3])
+
+@atexit.register
+def _guard():
+    die("dk.migration.run() was not called")