README.md 11.3 KB
Newer Older
BAIRE Anthony's avatar
BAIRE Anthony committed
1
2
3
4
5
6
ALLGO containers
================

Overview
--------

7
A minimal deployment of allgo consists of 6 docker images:
BAIRE Anthony's avatar
BAIRE Anthony committed
8

9
10
11
- **allgo/redis**     : the redis application server 
- **allgo/django**    : the django application server 
- **allgo/mysql**     : the mysql database server
BAIRE Anthony's avatar
BAIRE Anthony committed
12
- **allgo/controller**: the manager for user docker containers
13
14
- **allgo/ssh**       : the ssh frontend (giving access to the sandboxes)
- **allgo/toolbox**   : an image containing a set of commands (scp, vi, nano,
15
  less, ...) to be mounted in the user sandboxes
BAIRE Anthony's avatar
BAIRE Anthony committed
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

These images may be deployed multiple times to implement multiple independent
environments (eg: production, qualification, ...).

Additionally there are two images that are meant to be deployed only once (they
may serve multiple environments)

- **allgo/registry**: the docker registry
- **allgo/nginx**: the frontal http server

There is an extra image used only in development:

- **allgo/smtpsink**: a SMTP server that catches and stores all incoming messages into a single mailbox

Each environment has its own docker network. The nginx container is connected
31
to all these networks.
BAIRE Anthony's avatar
BAIRE Anthony committed
32
33


CAMPION Sebastien's avatar
CAMPION Sebastien committed
34
License 
CAMPION Sebastien's avatar
CAMPION Sebastien committed
35
36
37
38
39
40
-------

GNU AFFERO GENERAL PUBLIC LICENSE
https://www.gnu.org/licenses/agpl-3.0.html


BAIRE Anthony's avatar
BAIRE Anthony committed
41
42
43
44
45
46
47
Conventions
-----------

All docker images use the following conventions.

### External volumes

LETORT Sebastien's avatar
LETORT Sebastien committed
48
Their data are stored in:
BAIRE Anthony's avatar
BAIRE Anthony committed
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

  - `/vol/rw` for persistent data
  - `/vol/ro` for persistent data in read-only access
  - `/vol/cache` for cache data (persistent data that may be destroyed at any time without any consequence)
  - `/vol/log` for the logs

These paths are expected to be mounted as external volumes, either separately (typical for a production deployment) or with a single mount at `/vol` (typical for a development environment). The owner of external volumes must be the same as the uid used for the app inside the container.

### Admin scripts

Each container may contain a set of scripts for admin purpose (especially for managing the content of external volumes)

  - `/dk/container_init` initialise the content of the external volumes (eg: create and seed a database, write a default config, ...)
  - `/dk/image_upgrade` apply security upgrades to the image. This command is expected to exit with 0 if successful and to output something on stdout/stderr when something was upgraded an nothing if nothing was upgraded (thus if the output is empty, it is not necessary to commit a new image). 


65
66
67
Constraints
-----------

68
* The deployment repository has to be named *allgo*, because docker-compose names its network with <dir_name> prefix, and we used this network name in some parts.
69

BAIRE Anthony's avatar
BAIRE Anthony committed
70
71
72
73

Development environment
-----------------------

BAIRE Anthony's avatar
BAIRE Anthony committed
74
The development environment is managed with docker-compose. There are 2 important files:
BAIRE Anthony's avatar
BAIRE Anthony committed
75
76
77
78

 - `docker-compose.yml` the docker-compose configuration
 - `bootstrap` the bootstrap script

BAIRE Anthony's avatar
BAIRE Anthony committed
79
It provides 8 containers:
BAIRE Anthony's avatar
BAIRE Anthony committed
80

BAIRE Anthony's avatar
BAIRE Anthony committed
81
 - `dev-controller`
BAIRE Anthony's avatar
BAIRE Anthony committed
82
83
 - `dev-mysql`
 - `dev-nginx`
BAIRE Anthony's avatar
BAIRE Anthony committed
84
85
 - `dev-docker`
 - `dev-reddit`
BAIRE Anthony's avatar
BAIRE Anthony committed
86
87
88
89
90
91
92
 - `dev-registry`
 - `dev-smtpsink`
 - `dev-ssh`

All external volumes are stored in `/data/dev/` (the path is absolute because
it is tricky to use a relative path with the allgo/docker image).

93
For convenience, all containers not running as root (django, mysql, registry)
BAIRE Anthony's avatar
BAIRE Anthony committed
94
have their user overridden to the UID:GID of the developer running
BAIRE Anthony's avatar
BAIRE Anthony committed
95
docker-compose. This is managed with the `DOCKERUSER` environment variable set
96
[in the `.env` file](https://docs.docker.com/compose/environment-variables/#the-env-file) by
97
`prepare.sh`.
BAIRE Anthony's avatar
BAIRE Anthony committed
98

BAIRE Anthony's avatar
typos    
BAIRE Anthony committed
99
For convenience (again), there is an extra external volume for `dev-django`,
BAIRE Anthony's avatar
BAIRE Anthony committed
100
`dev-controller` and `dev-ssh` so that the source directory of the app is mounted
BAIRE Anthony's avatar
BAIRE Anthony committed
101
102
103
104
105
106
107
inside `/opt/` (in fact it overrides the actual application files provided by
the docker image). The purpose is to avoid rebuilding a new docker image for
each development iteration. 


### Getting started

108
The sources are located in one repository:
BAIRE Anthony's avatar
BAIRE Anthony committed
109
110
111

 - *allgo*: the deployment repository

112

BAIRE Anthony's avatar
BAIRE Anthony committed
113
114
115
To set up the development environment, run: 

 1. get the sources
116
	<pre>
BAIRE Anthony's avatar
BAIRE Anthony committed
117
	git clone git@gitlab.inria.fr:allgo/allgo.git
BAIRE Anthony's avatar
BAIRE Anthony committed
118
119
120
121
122
123
	cd allgo
	</pre>

 2. *(as root)* create `/data/dev` and make it owned by the developer
	<pre>
	sudo mkdir -p    /data/dev
LETORT Sebastien's avatar
LETORT Sebastien committed
124
	sudo chown $USER: /data/dev
BAIRE Anthony's avatar
BAIRE Anthony committed
125
126
	</pre>

127
 3. bootstrap the environment
128
	<pre>
BAIRE Anthony's avatar
BAIRE Anthony committed
129
130
131
132
133
	./bootstrap
	</pre>
	This command will run the `/dk/init_container` in every container that
	needs it, then start the container.

134
135
136
	The first run takes a few minutes because all images are built from
	scratch.
	You may have enough time for a short coffee break.
BAIRE Anthony's avatar
BAIRE Anthony committed
137
138
139
140

	**Note** by default `bootstrap` works on all containers. It is possible
	to give an explicit list of containers instead. Example:
	<pre>
141
		./bootstrap dev-mysql dev-django
BAIRE Anthony's avatar
BAIRE Anthony committed
142
143
	</pre>

144
 4. for convenience, you may want to alias `docker-compose` as `fig` (because
BAIRE Anthony's avatar
BAIRE Anthony committed
145
146
147
148
149
150
     `fig` is much faster to type than `docker-compose` and you will have to
     type it a lot). Somewhere in your `.bashrc` you should add:
	<pre>
	alias fig=docker-compose
	</pre>

151
152
153
154
155
 5. after bootstrap, in development the db is initialised with three users (all with the password `allgo`):
    - `admin@localhost` which is superuser
    - `devel@localhost` which is the owner of a webapp named `sleep`
    - `guest@localhost`

BAIRE Anthony's avatar
BAIRE Anthony committed
156
157
158
159

### Common commands

The official doc for docker-compose is available at: [https://docs.docker.com/compose/
160
](https://docs.docker.com/compose/)
BAIRE Anthony's avatar
BAIRE Anthony committed
161
162
163
164
165
166
167
168

 - start all containers (in the background)
	<pre>
	fig up -d
	</pre>

 - start all containers (in the foreground, i.e interactively, when you hit Ctrl-C all containers are stop)
	<pre>
169
	fig up
BAIRE Anthony's avatar
BAIRE Anthony committed
170
171
172
173
174
175
176
177
	</pre>

 - soft cleanup (stop and remove all containers)
	<pre>
	fig down
	</pre>

 - hard cleanup (remove images too)
178
	<pre>
BAIRE Anthony's avatar
BAIRE Anthony committed
179
180
181
182
183
	fig down --rmi local
	</pre>

 - restart a container
	<pre>
184
	fig restart dev-django
BAIRE Anthony's avatar
BAIRE Anthony committed
185
186
187
	</pre>

 - restart a container using a new docker image (if the image has been rebuilt since the last start)
188
189
	<pre>
	fig up dev-django
BAIRE Anthony's avatar
BAIRE Anthony committed
190
191
192
	</pre>

 - rebuild an image
193
194
	<pre>
	fig build dev-django
BAIRE Anthony's avatar
BAIRE Anthony committed
195
196
197
198
	</pre> 

 - **Note:** most commands work on every container by default (eg: up down
   start stop restart ...) they can be use on an individual container too:
199
200
	<pre>
	fig restart dev-controller dev-django
BAIRE Anthony's avatar
BAIRE Anthony committed
201
202
	</pre>

203
204
205
 - run a container with an arbitrary command (eg: to have access to the django console)
	<pre>
	fig run --rm dev-django bash
BAIRE Anthony's avatar
BAIRE Anthony committed
206
207
208
209
210
211
212
	</pre>

	**Note:** containers created by `fig run` have the same parameters as
	the referenced containers but their name is different (eg:
	*allgo_dev-ssh_run_1*), which means that this container is not
	reachable by the others (this may be an issue for example if you want
	to run the mysqld server manually: `fig run dev-mysql mysqld` -> this
213
	container won't be reachable by the ssh and django containers)
BAIRE Anthony's avatar
BAIRE Anthony committed
214
215

 - follow the output of all containers:
216
	<pre>
BAIRE Anthony's avatar
BAIRE Anthony committed
217
218
219
	fig logs --tail=1 --follow
	</pre>

220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255

### Troubleshooting

#### Fedora OS: docker.sock usable by root user only

The docker socket `/run/docker.sock` for interacting with the docker daemon is
owned by `root:root` with `0600` permissions. Only the root user can use it

To use it as an ordinary user, create the `docker` group, then the docker
engine will automatically make its socket owned by `root:docker`. Users who are
members of the docker group will be able to use it. 

To do so you must run the following commands :

<pre>sudo groupadd docker</pre>

Edit the group to add the relevant users to the `docker` group (add them at the
end of the line starting with `docker:`, separated by commas)

<pre>sudo vigr</pre>

Re-open your session (to be effectively in the docker group)


#### **dev-mysql** initialisation fails

The `prepare.sh` script (also sourced by the `./bootstrap` script) creates the
`/data/dev/*/` with the ownership set to the calling user. **Be sure not to run
it as root**, otherwise it will be owned by root and you may have errors like:

	mkdir: cannot create directory ‘/vol/rw’: Permission denied
	mkdir: cannot create directory ‘/vol/log’: Permission denied

If somehow you skipped this step, you can reset the ownership to the current user:

	sudo chown    USER: /data/dev
256
	sudo chown -R USER: /data/dev/{registry,mysql,django}
257
258
259
260
261
262
263
264
265
266

If you are completely lost, you can just restart the initialisation from scratch:

	fig down
	sudo rm -rf /data/dev
	. prepare.sh
	./bootstrap
 


BAIRE Anthony's avatar
BAIRE Anthony committed
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
Production environment
----------------------

- TODO unicorn/nginx integration
- TODO use capistrano too ?


Images design
-------------


## registry

Hosts docker registry with a nginx configured as a reverse proxy. It listens to 4 ports:

 - :8000 (*production*) access limited to `/v2/allgo/prod/`
 - :8001 (*qualification*) access limited to `/v2/allgo/qualif/`
 - :8002 (*development*)  access limited to `/v2/allgo/dev`
 - :5000 (*legacy production*) access limited to `/v2/allgo`(which is mapped to `/v2/allgo/prod/`)

## mysql

Hosts a mysql server listening on port 3306 with two databases: `allgo` and
`allgo_test` and two users: `allgo` and `ssh`.

 - `allgo` has read/write access to both databases
 - `ssh` has read only access to `allgo`


296
## django
BAIRE Anthony's avatar
BAIRE Anthony committed
297

298
299
300
301
302
303
Hosts three daemons for running the allgo web server:
 
- a nginx frontend for buffering the HTTP requests/responses and routing them
  to the other daemons. It also serves static files directly
- the gunicorn server (running the django application)
- the allgo.aio server (serving the asynchronous requests)
BAIRE Anthony's avatar
BAIRE Anthony committed
304
305


306
### Running the django server manually
BAIRE Anthony's avatar
BAIRE Anthony committed
307

308
TODO ?
BAIRE Anthony's avatar
BAIRE Anthony committed
309

310
[comment]: # ( - run the `dev-rails` container and open a shell:
BAIRE Anthony's avatar
BAIRE Anthony committed
311
312
313
314
315
316
317
318
319
320
	<pre>
	fig up -d
	docker exec -t -i  dev-rails bash
	</pre>

 - then, inside the container, run:
	<pre>
	supervisorctl stop rails
	rails server
	</pre>
321
)
BAIRE Anthony's avatar
BAIRE Anthony committed
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336

## ssh

Hosts the ssh front server for accessing the sandboxes (`ssh
WEBAPP@sid.allgo.irisa.fr`). Each allgo webapp is mapped to a system user
(using Glibc NSS) starting at uid 2000.

- `/etc/passwd` and `/etc/group` are overriden so as to contain only the two users (*root* and *sshd*) and one group (*nogroup*) required to run the ssh server
- Extra users are obtained from the mysql database (using libnss-mysql-bg) and mapped as follows:
	<pre>
	name  = webapps.docker_name
	uid   = webapps.id
	gid   = 65534 (nogroup)
	gecos = webapps.name
	shell = /bin/allgo-shell
337
	</pre>
BAIRE Anthony's avatar
BAIRE Anthony committed
338
339
340
341
342
343
344
345
- The ssh server is configured to accept key-based authentication only. The
  list of public keys is obtained from the (using an AuthorizedKeysCommand).

- The forced shell (`allgo-shell`) connects to the webapp sandbox (if running).

- The connection to the sandbox is made though a unix socket and a set of pipes
  in the filesystem.

346
## controller
BAIRE Anthony's avatar
BAIRE Anthony committed
347

348
Hosts the *docker-controller* which manages all docker operations (run, stop,
349
rm, commit, pull, push, ...) on behalf of the django container.
BAIRE Anthony's avatar
BAIRE Anthony committed
350

BAIRE Anthony's avatar
typos    
BAIRE Anthony committed
351
Technically speaking this container has root privileges since it has access to
BAIRE Anthony's avatar
BAIRE Anthony committed
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
the docker socket.

The proxy script enforces restrictions (according to the current environment: eg prod/qualif/dev) on:

 - the registry (for pulling/pushing)
 - the paths of external volumes 
 - the container names (*ENV*-user-*XXXX*)

## nginx

Hosts the frontal nginx server, its purpose is to:
- give access to one or more allgo instances
- manage TLS encryption

## smtpsink

Hosts a SMTP server (port 25) and an IMAP server (port 143) for
development/qualification

Its purpose is to channel all outgoing mail (received on port 25) into a single
mailbox.

The mailbox is accessible with IMAP as user *sink* (password *sink*).

376
377
NOTE: in the development environment, django's default is to dump outgoing
e-mails to the console. Thus this container is only useful in the qualif setup.