1. 14 Apr, 2020 1 commit
  2. 02 Apr, 2020 2 commits
  3. 14 Jan, 2019 1 commit
  4. 27 Sep, 2018 1 commit
  5. 17 Sep, 2018 2 commits
  6. 07 Aug, 2018 5 commits
  7. 05 Jul, 2018 1 commit
  8. 28 Jun, 2018 2 commits
  9. 27 Jun, 2018 4 commits
    • BAIRE Anthony's avatar
      Stream job logs and job state updates to the user · 1bb4acf4
      BAIRE Anthony authored
      This commit makes several changes.
      
      In the controller:
      
      - duplicates the logs produced by the jobs. Initially they were only
        stored into allgo.log, now they are also forwarded to the container
        output (using the 'tee' command) so that the controller can read
        them
      
      - add a log_task that reads the logs from docker and feeds them into
        the redis db key "log:job:<ID>" (this is implemented with aiohttp
        in order to be fully asynchronous)
      
      - store the job state in a new redis key "state:job:<ID>"
      
      - send notification to the redis pubsub 'notify:aio' channel when
        the job state has changed or when new logs are available
      
      In the allgo.aio frontend:
      
      - implement the /aio/jobs/<ID>/events endpoints which streams all
        job events & logs to the client (using json formatted messages)
      
      In django:
      
      - refactor the JobDetail view and template to update the page
        dynamically for job updates (state/logs)
          - allgo.log is read only when the job is already terminated.
            Otherwise the page uses the /aio/jobs/<ID>/events channel
            to stream the logs
          - the state icon is patched on the page when the state changes,
            except for the DONE state which triggers a full page reload
            (because there are other parts to be updated)
      1bb4acf4
    • BAIRE Anthony's avatar
    • BAIRE Anthony's avatar
      Add a reddis client & reddis listener task to the aio server · 7fa7818d
      BAIRE Anthony authored
      listens for notifications on channel 'notify:aio'
      
      format of the notifications messages  "TYPE:ID"
      (eg: "job:127" for notifying that there is an update on the
       job with id 127)
      7fa7818d
    • BAIRE Anthony's avatar
      Add an auxiliary HTTP server (allgo.aio) for serving asynchronous requests · c5cd2bc1
      BAIRE Anthony authored
      There are two purposes:
      - implement server push (using long-lived HTTP requests) for:
          - sending status updates for the jobs and sandboxes
          - live-streaming of the job logs
      - have a really async implementation for pushing image manifests into
        the registry (the preliminary implementation in
        5451a6df was blocking)
      
      It is implemented with aiohttp.web (a lighweight HTTP framework,
      similar to what we can do with flask but asynchronously).
      
      The alternative would have been to use the django channels plugin, but:
      - it went through a major refactoring (v2) recently
      - it requires replacing unicorn with an ASGI server (daphne)
      - django-channels and daphne are not yet debian, and for the HTTP server
        i would prefer an implementation for which we have stable security
        updates
      
      (anyway this can be ported to django-channels later)
      
      The nginx config redirects the /aio/ path to this server (and the image
      manifests pushes too).
      
      The allgo.aio server interacts only with the nginx, reddis and django
      containers. It relies totally on the django server for authenticating
      the user and for accessing the mysql db (so there is no ORM).
      
      NOTE: in this design the django server has to trust blindly the requests
      coming from the allgo.aio server (for some endpoints). To prevent
      security issues, the nginx configuration is modified to set the header
      'X-Origin: nginx'. Thus django knowns who he can trust.
      
      This commits implements only the image pushs. Job updated and logs
      streaming will come in a later pull request.
      c5cd2bc1