README.md 11.7 KB
Newer Older
BAIRE Anthony's avatar
BAIRE Anthony committed
1 2 3 4 5 6 7 8 9 10
ALLGO containers
================

Overview
--------

A minimal deployment of allgo consists of 4 docker images:

- **allgo/rails**: the rails application server 
- **allgo/mysql**: the mysql database server
BAIRE Anthony's avatar
BAIRE Anthony committed
11
- **allgo/controller**: the manager for user docker containers
BAIRE Anthony's avatar
BAIRE Anthony committed
12
- **allgo/ssh**: the ssh frontend (giving access to the sandboxes)
13 14
- **allgo/toolbox**: an image containing a set of commands (scp, vi, nano,
  less, ...) to be mounted in the user sandboxes
BAIRE Anthony's avatar
BAIRE Anthony committed
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69

These images may be deployed multiple times to implement multiple independent
environments (eg: production, qualification, ...).

Additionally there are two images that are meant to be deployed only once (they
may serve multiple environments)

- **allgo/registry**: the docker registry
- **allgo/nginx**: the frontal http server

There is an extra image used only in development:

- **allgo/smtpsink**: a SMTP server that catches and stores all incoming messages into a single mailbox

Each environment has its own docker network. The nginx container is connected
to all these networks to that it can connect to the rails servers.


Conventions
-----------

All docker images use the following conventions.

### External volumes

They data is stored in:

  - `/vol/rw` for persistent data
  - `/vol/ro` for persistent data in read-only access
  - `/vol/cache` for cache data (persistent data that may be destroyed at any time without any consequence)
  - `/vol/log` for the logs

These paths are expected to be mounted as external volumes, either separately (typical for a production deployment) or with a single mount at `/vol` (typical for a development environment). The owner of external volumes must be the same as the uid used for the app inside the container.

### Admin scripts

Each container may contain a set of scripts for admin purpose (especially for managing the content of external volumes)

  - `/dk/container_init` initialise the content of the external volumes (eg: create and seed a database, write a default config, ...)
  - `/dk/image_upgrade` apply security upgrades to the image. This command is expected to exit with 0 if successful and to output something on stdout/stderr when something was upgraded an nothing if nothing was upgraded (thus if the output is empty, it is not necessary to commit a new image). 



Development environment
-----------------------

The development environment is managed with docker-compose. There are 3 important files:

 - `docker-compose.yml` the docker-compose configuration
 - `prepare.sh` a shell script which is meant to be sourced in the shell before
   working with the environment
 - `bootstrap` the bootstrap script

It provides 7 containers:

BAIRE Anthony's avatar
BAIRE Anthony committed
70
 - `dev-controller`
BAIRE Anthony's avatar
BAIRE Anthony committed
71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
 - `dev-mysql`
 - `dev-nginx`
 - `dev-rails`
 - `dev-registry`
 - `dev-smtpsink`
 - `dev-ssh`

All external volumes are stored in `/data/dev/` (the path is absolute because
it is tricky to use a relative path with the allgo/docker image).

For convenience, all containers not running as root (rails, mysql, registry)
have their user overridden to the UID:GID of the developer running
docker-compose. This is managed with the `DOCKER_USER` environment variable set
by `prepare.sh`.

For convenience (again), there is an extra external volumes for `dev-rails`,
BAIRE Anthony's avatar
BAIRE Anthony committed
87
`dev-controller` and `dev-ssh` so that the source directory of the app is mounted
BAIRE Anthony's avatar
BAIRE Anthony committed
88 89 90 91
inside `/opt/` (in fact it overrides the actual application files provided by
the docker image). The purpose is to avoid rebuilding a new docker image for
each development iteration. 

92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154
**Nota Bene ; installation on fedora Os versions , GROUP EDITION**

The docker demon is coded in such a way that if he doesn't find a docker group in the list of groups , 

he automatically sets the socket's permission as root .

Now if the group exists , the docker demon automatically gives the docker group read/write access to the socket , 

then users belonging to this group inherit the same rights .

So in order to install the developpement environment on fedora as non-sudo user you must edit the list of groups , 

adding a docker group with your username on it .

To do so you must run the following commands :

<pre>sudo groupadd docker

newgrp docker</pre>

Now if the new group docker command doesn't work then you have another solution to validate the group creation changes .

You can edit the changes using the vigr command .

<pre>sudo vigr</pre>

Beware though , while using vigr , you must put ":" after the group name before entering your username .

**Nota Bene ; installation on fedora Os versions , dev-mysql installation as non-sudo**

In the following "getting started" chapter , step 2 is a VERY important step , as a non-sudo user , you MUST set the permissions of these directories .

If you do not , the bootstrap script won't be able to write down all the data needed in the /data/dev directories at its execution .

Typically the error logs you will get will be :

mkdir: cannot create directory ‘/vol/rw’: Permission denied

mkdir: cannot create directory ‘/vol/log’: Permission denied

If somehow you skipped this step , the fix is as follows :

 <pre>id USER</pre>
 
 ==> uid=657078(USER) gid=29005(ascii) groups=29005(ascii),971(vboxusers),1000(docker)
 
 <pre>sudo chown -R USER:ascii /data/dev</pre>
 
 Then you must erase all the content of the /data/dev directories , to do so :
 
 <pre>sudo rm -rf /data/dev/mysql/*</pre>
 
 If you don't erase it's content , the typical error log you will get each time you try to execute the ./bootstrap script will be :

 error: db is already initialised

**Besides** : dev-nginx won't be able to start too at the ./bootsrap execution if he can't access the database in dev-mysql , 

so you would get the following error logs if dev-mysql hasn't started properly :

ERROR: for dev-nginx  Cannot create container for service dev-nginx: Device is Busy

ERROR: Encountered errors while bringing up the project.
BAIRE Anthony's avatar
BAIRE Anthony committed
155 156 157 158 159 160 161 162

### Getting started

The sources are located in two repositories:

 - *rails-allgo*: the rails application repository
 - *allgo*: the deployment repository

163

BAIRE Anthony's avatar
BAIRE Anthony committed
164 165 166 167
To set up the development environment, run: 

 1. get the sources
 	<pre>
BAIRE Anthony's avatar
BAIRE Anthony committed
168
	git clone git@gitlab.inria.fr:allgo/allgo.git
BAIRE Anthony's avatar
BAIRE Anthony committed
169 170 171 172 173 174
	cd allgo
	</pre>

 2. *(as root)* create `/data/dev` and make it owned by the developer
	<pre>
	sudo mkdir -p    /data/dev
Charles Deltel's avatar
Charles Deltel committed
175
	sudo chown USER: /data/dev
BAIRE Anthony's avatar
BAIRE Anthony committed
176 177 178 179
	</pre>

 3. source the `prepare.sh` script (to be run every time before you start working)
 	<pre>
Charles Deltel's avatar
Charles Deltel committed
180
	. ./prepare.sh
BAIRE Anthony's avatar
BAIRE Anthony committed
181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250
	</pre>

 4. bootstrap the environment
 	<pre>
	./bootstrap
	</pre>
	This command will run the `/dk/init_container` in every container that
	needs it, then start the container.

	The first run takes a very long time because all images are built from
	scratch (especially the rails image which builds ruby source).
	You have enough time for a coffee break.

	**Note** by default `bootstrap` works on all containers. It is possible
	to give an explicit list of containers instead. Example:
	<pre>
		./bootstrap dev-mysql dev-rails
	</pre>

 5. for convenience, you may want to alias `docker-compose` as `fig` (because
     `fig` is much faster to type than `docker-compose` and you will have to
     type it a lot). Somewhere in your `.bashrc` you should add:
	<pre>
	alias fig=docker-compose
	</pre>


### Common commands

The official doc for docker-compose is available at: [https://docs.docker.com/compose/
https://docs.docker.com/compose/]()

 - start all containers (in the background)
	<pre>
	fig up -d
	</pre>

 - start all containers (in the foreground, i.e interactively, when you hit Ctrl-C all containers are stop)
	<pre>
	fig up -d
	</pre>

 - soft cleanup (stop and remove all containers)
	<pre>
	fig down
	</pre>

 - hard cleanup (remove images too)
 	<pre>
	fig down --rmi local
	</pre>

 - restart a container
	<pre>
	fig restart dev-rails
	</pre>

 - restart a container using a new docker image (if the image has been rebuilt since the last start)
 	<pre>
	fig up dev-rails
	</pre>

 - rebuild an image
 	<pre>
	fig build dev-railf
	</pre> 

 - **Note:** most commands work on every container by default (eg: up down
   start stop restart ...) they can be use on an individual container too:
   	<pre>
BAIRE Anthony's avatar
BAIRE Anthony committed
251
	fig restart dev-controller dev-rails
BAIRE Anthony's avatar
BAIRE Anthony committed
252 253 254 255
	</pre>

 - run a container with an arbitrary command (eg: to have access to the rails console)
 	<pre>
BAIRE Anthony's avatar
BAIRE Anthony committed
256
	fig run --rm dev-rails bash
BAIRE Anthony's avatar
BAIRE Anthony committed
257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301
	</pre>

	**Note:** containers created by `fig run` have the same parameters as
	the referenced containers but their name is different (eg:
	*allgo_dev-ssh_run_1*), which means that this container is not
	reachable by the others (this may be an issue for example if you want
	to run the mysqld server manually: `fig run dev-mysql mysqld` -> this
	container won't be reachable by the ssh and rails containers)

 - follow the output of all containers:
 	<pre>
	fig logs --tail=1 --follow
	</pre>

Production environment
----------------------

- TODO unicorn/nginx integration
- TODO use capistrano too ?


Images design
-------------


## registry

Hosts docker registry with a nginx configured as a reverse proxy. It listens to 4 ports:

 - :8000 (*production*) access limited to `/v2/allgo/prod/`
 - :8001 (*qualification*) access limited to `/v2/allgo/qualif/`
 - :8002 (*development*)  access limited to `/v2/allgo/dev`
 - :5000 (*legacy production*) access limited to `/v2/allgo`(which is mapped to `/v2/allgo/prod/`)

## mysql

Hosts a mysql server listening on port 3306 with two databases: `allgo` and
`allgo_test` and two users: `allgo` and `ssh`.

 - `allgo` has read/write access to both databases
 - `ssh` has read only access to `allgo`


## rails

BAIRE Anthony's avatar
BAIRE Anthony committed
302
Hosts four daemons for running allgo:
BAIRE Anthony's avatar
BAIRE Anthony committed
303

BAIRE Anthony's avatar
BAIRE Anthony committed
304
 - the unicorn server (runnning the rails application)
BAIRE Anthony's avatar
BAIRE Anthony committed
305 306
 - the sidekiq queue manager
 - the redis db server
BAIRE Anthony's avatar
BAIRE Anthony committed
307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325
 - a nginx frontend for buffering the HTTP requests/responses

This container is managed with supervisor, the `supervisorctl` command allows
starting/stopping the daemons individually.

### Running the rails server manually


 - run the `dev-rails` container and open a shell:
	<pre>
	fig up -d
	docker exec -t -i  dev-rails bash
	</pre>

 - then, inside the container, run:
	<pre>
	supervisorctl stop rails
	rails server
	</pre>
BAIRE Anthony's avatar
BAIRE Anthony committed
326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380


## ssh

Hosts the ssh front server for accessing the sandboxes (`ssh
WEBAPP@sid.allgo.irisa.fr`). Each allgo webapp is mapped to a system user
(using Glibc NSS) starting at uid 2000.

- `/etc/passwd` and `/etc/group` are overriden so as to contain only the two users (*root* and *sshd*) and one group (*nogroup*) required to run the ssh server
- Extra users are obtained from the mysql database (using libnss-mysql-bg) and mapped as follows:
	<pre>
	name  = webapps.docker_name
	uid   = webapps.id
	gid   = 65534 (nogroup)
	gecos = webapps.name
	shell = /bin/allgo-shell
  	</pre>
- The ssh server is configured to accept key-based authentication only. The
  list of public keys is obtained from the (using an AuthorizedKeysCommand).

- The forced shell (`allgo-shell`) connects to the webapp sandbox (if running).

- The connection to the sandbox is made though a unix socket and a set of pipes
  in the filesystem.

## docker

Hosts the *docker-allgo-proxy* which manages all docker operations (run, stop,
rm, commit, pull, push, ...) on behalf of the rails container.

Technically speaking this container had root privileges since it has access to
the docker socket.

The proxy script enforces restrictions (according to the current environment: eg prod/qualif/dev) on:

 - the registry (for pulling/pushing)
 - the paths of external volumes 
 - the container names (*ENV*-user-*XXXX*)

## nginx

Hosts the frontal nginx server, its purpose is to:
- give access to one or more allgo instances
- manage TLS encryption

## smtpsink

Hosts a SMTP server (port 25) and an IMAP server (port 143) for
development/qualification

Its purpose is to channel all outgoing mail (received on port 25) into a single
mailbox.

The mailbox is accessible with IMAP as user *sink* (password *sink*).