GitLab

This way we get redis updates when a job is deleted and this also
prevents inserting a 'DONE' state when the job is not done

Note: we never update the redis job state key from the django server to
avoid race conditions

While this is not needed for the job_detail page (because we just reload
the page when the job is done), we will need this information for the
job_list page (because we do not want to reload the job_list each time a
job terminates)

The controller and django can both change the job state, especially when
it is in the WAITING state (django may delete the job and the controller
may start the job).

To prevent any inconsistency, we must ensure that these transitions are
made atomically.

because it is not a good idea to send our secret token
(for our private registry) to the official docker registry

This commit removes the old notification channel (socket listening
on port 4567), and uses the redis channel 'notify:controller' instead.

The django job creation views are updated accordingly.

This commit makes several changes.

In the controller:

- duplicates the logs produced by the jobs. Initially they were only
  stored into allgo.log, now they are also forwarded to the container
  output (using the 'tee' command) so that the controller can read
  them

- add a log_task that reads the logs from docker and feeds them into
  the redis db key "log:job:<ID>" (this is implemented with aiohttp
  in order to be fully asynchronous)

- store the job state in a new redis key "state:job:<ID>"

- send notification to the redis pubsub 'notify:aio' channel when
  the job state has changed or when new logs are available

In the allgo.aio frontend:

- implement the /aio/jobs/<ID>/events endpoints which streams all
  job events & logs to the client (using json formatted messages)

In django:

- refactor the JobDetail view and template to update the page
  dynamically for job updates (state/logs)
    - allgo.log is read only when the job is already terminated.
      Otherwise the page uses the /aio/jobs/<ID>/events channel
      to stream the logs
    - the state icon is patched on the page when the state changes,
      except for the DONE state which triggers a full page reload
      (because there are other parts to be updated)

now located in the 'django' container and full path is
"{DATASTORE}/{JOB_ID}"

Fix #185

- webapps are now located directly at the root of the registry
  (not in the /webapp subdir)

- factories are no longer stored in our registry, we directly reference
  images on the official docker registry

because it is now based on tokens (instead of using TLS client certificates)

this variant is closer to the actual meaning
(the fact that the limit was reached does not automatically imply
 that the process is starving, we cannot decide how much memory
 a process needs without doing some profiling)

fix #96

fix #55

fix #121

fix #125

- to have SIGTERM forwarded to the process
- to propagate the exit code of the process

to let the task implementations detect that they are being rescheduled

(disable_future_warning)

- add SwarmAbstractionClient: a class that extends docker.Client and
  hides the API differences between the docker remote API and the
  swarm API. Thus a single docker engine can be used like a swarm

- add SharedSwarmClient: a class that extends SwarmAbstractionClient
  and monitors the swarm health and its resource (cpu/mem) and manages
  the resource allocation.
  - resources are partitioned in groups (to allow reserving resources
    for higher priority jobs)
  - two SharedSwarmClient can work together over TCP in a master/slave
    configuration (to allow the production and qualification platforms
    to use the same swarm without any interference)

- the controller is modified to:
  - use SharedSwarmClient to:
    - wait for the end of a job (in place of DockerWatcher)
    - manage resource reservation (LONG_APPS vs. BIGMEM_APPS vs normal
      apps) and monitor swarm health (fix #124)
    - NOTE: resources of the swarm and sandbox are now managed
      separately (2 instances of SharedSwarmClient), whereas it was
      global before (this was suboptimal)
  - rely on SwarmAbstractionClient to compute the cpu quotas
  - store the container_id of jobs into the DB (fix #128), this is a
    prerequisite to permit renaming apps in the future
  - store the class of the job (normal vs. long app) in the container
    name (for the resource management with SharedSwarmClient)
  - read the configuration from a yaml file (/vol/ro/config.yml) for:
    - cpu/mem quotas
    - swarm resources allocation policy
    - master/slave configuration

with swarm, we had:

2017-Apr-24 18:09:13 DEBUG    docker event b'{"from":"
node:worker1","Type":"network","Action":"disconnect","Actor":{"ID":"83192c2a09bee6e50fba8cc506b1870fb6d96438afe0a821541e00b43a7229be","Attributes":{"contain
er":"d89ab93f01cbde3e78ca2bfcc688656b15a3a664060ea85733f1497ea7531d58","name":"bridge","node.addr":"192.168.21.100:2375","node.id":"V24K:QBXP:VO3N:K2EV:M2MH:IRI4:FV7Z:LOTZ:GTTD:PDBS:SSO2:KHIG","node.ip":"192.168
.21.100","node.name":"worker1","type":"bridge"}},"time":1493050153,"timeNano":1493050153228713620,"node":{"Name":"worker1","Id":"V24K:QBXP:VO3N:K2EV:M2MH:IRI4:FV7Z:LOTZ:GTTD:PDBS:SSO2:KHIG","Addr":"192.168.21.10
0:2375","Ip":"192.168.21.100"}}'
2017-Apr-24 18:09:13 ERROR    docker watcher exception
Traceback (most recent call last):
  File "/opt/allgo-docker/controller.py", line 950, in _thread_func
      if event["status"] == "die":
      KeyError: 'status'

changing workdir /tmp -> /work is deferred to 0.6

fix #103

- sshd server installed in the toolbox
- ssh keys & config stored in ssh:/vol/cache and mounted as
  /.sandbox inside the sandbox
- toolbox mounted as /.toolbox inside the sandbox
- ssh agent & X11 forwarding are now working
- the toolbox commands available by default in every sandboxes
  (vim, less, nc, scp, ...)
- sandboxes now attached to a separate network (named
  'allgo_sandboxes' by default)

fix #88