1. 27 Jun, 2018 13 commits
    • BAIRE Anthony's avatar
      Stream job logs and job state updates to the user · 1bb4acf4
      BAIRE Anthony authored
      This commit makes several changes.
      In the controller:
      - duplicates the logs produced by the jobs. Initially they were only
        stored into allgo.log, now they are also forwarded to the container
        output (using the 'tee' command) so that the controller can read
      - add a log_task that reads the logs from docker and feeds them into
        the redis db key "log:job:<ID>" (this is implemented with aiohttp
        in order to be fully asynchronous)
      - store the job state in a new redis key "state:job:<ID>"
      - send notification to the redis pubsub 'notify:aio' channel when
        the job state has changed or when new logs are available
      In the allgo.aio frontend:
      - implement the /aio/jobs/<ID>/events endpoints which streams all
        job events & logs to the client (using json formatted messages)
      In django:
      - refactor the JobDetail view and template to update the page
        dynamically for job updates (state/logs)
          - allgo.log is read only when the job is already terminated.
            Otherwise the page uses the /aio/jobs/<ID>/events channel
            to stream the logs
          - the state icon is patched on the page when the state changes,
            except for the DONE state which triggers a full page reload
            (because there are other parts to be updated)
    • BAIRE Anthony's avatar
    • BAIRE Anthony's avatar
    • BAIRE Anthony's avatar
      update the location of the job files · c5f93183
      BAIRE Anthony authored
      now located in the 'django' container and full path is
    • BAIRE Anthony's avatar
      add a redis client in the controller · 553fee62
      BAIRE Anthony authored
    • BAIRE Anthony's avatar
      remove the factories · 52a9e3ec
      BAIRE Anthony authored
      Fix #185
      - webapps are now located directly at the root of the registry
        (not in the /webapp subdir)
      - factories are no longer stored in our registry, we directly reference
        images on the official docker registry
    • BAIRE Anthony's avatar
      introduce a special token for the controller · b0807974
      BAIRE Anthony authored
      Initially the authentication with the registry was performed with a TLS
      client certificate installed on the controller.
      The registry is now configured to use token-based authentication (to
      give access to the users), but unfortunately it cannot be configured to
      support multiple auth methods. So we have to provide a token for the
      This is a 'God' token, it gives total access (pull and push) on all
    • BAIRE Anthony's avatar
      support registry authentication in the controller · 5a5a06d2
      BAIRE Anthony authored
      because it is now based on tokens (instead of using TLS client certificates)
    • BAIRE Anthony's avatar
      rename tables ad dj_* · a0ecda20
      BAIRE Anthony authored
    • BAIRE Anthony's avatar
    • BAIRE Anthony's avatar
      Add a reddis client & reddis listener task to the aio server · 7fa7818d
      BAIRE Anthony authored
      listens for notifications on channel 'notify:aio'
      format of the notifications messages  "TYPE:ID"
      (eg: "job:127" for notifying that there is an update on the
       job with id 127)
    • BAIRE Anthony's avatar
      Add an auxiliary HTTP server (allgo.aio) for serving asynchronous requests · c5cd2bc1
      BAIRE Anthony authored
      There are two purposes:
      - implement server push (using long-lived HTTP requests) for:
          - sending status updates for the jobs and sandboxes
          - live-streaming of the job logs
      - have a really async implementation for pushing image manifests into
        the registry (the preliminary implementation in
        5451a6df was blocking)
      It is implemented with aiohttp.web (a lighweight HTTP framework,
      similar to what we can do with flask but asynchronously).
      The alternative would have been to use the django channels plugin, but:
      - it went through a major refactoring (v2) recently
      - it requires replacing unicorn with an ASGI server (daphne)
      - django-channels and daphne are not yet debian, and for the HTTP server
        i would prefer an implementation for which we have stable security
      (anyway this can be ported to django-channels later)
      The nginx config redirects the /aio/ path to this server (and the image
      manifests pushes too).
      The allgo.aio server interacts only with the nginx, reddis and django
      containers. It relies totally on the django server for authenticating
      the user and for accessing the mysql db (so there is no ORM).
      NOTE: in this design the django server has to trust blindly the requests
      coming from the allgo.aio server (for some endpoints). To prevent
      security issues, the nginx configuration is modified to set the header
      'X-Origin: nginx'. Thus django knowns who he can trust.
      This commits implements only the image pushs. Job updated and logs
      streaming will come in a later pull request.
    • BAIRE Anthony's avatar
      fix param name in job detail view · 880e1e2a
      BAIRE Anthony authored
  2. 26 Jun, 2018 6 commits
  3. 25 Jun, 2018 3 commits
  4. 22 Jun, 2018 6 commits
  5. 21 Jun, 2018 2 commits
  6. 20 Jun, 2018 10 commits