1. 05 Jul, 2018 1 commit
  2. 04 Jul, 2018 5 commits
  3. 03 Jul, 2018 12 commits
  4. 29 Jun, 2018 2 commits
  5. 28 Jun, 2018 6 commits
  6. 27 Jun, 2018 10 commits
    • BAIRE Anthony's avatar
      Use the redis db to trigger controller actions · 01dd48e6
      BAIRE Anthony authored
      This commit removes the old notification channel (socket listening
      on port 4567), and uses the redis channel 'notify:controller' instead.
      
      The django job creation views are updated accordingly.
      01dd48e6
    • BAIRE Anthony's avatar
      Stream job logs and job state updates to the user · 1bb4acf4
      BAIRE Anthony authored
      This commit makes several changes.
      
      In the controller:
      
      - duplicates the logs produced by the jobs. Initially they were only
        stored into allgo.log, now they are also forwarded to the container
        output (using the 'tee' command) so that the controller can read
        them
      
      - add a log_task that reads the logs from docker and feeds them into
        the redis db key "log:job:<ID>" (this is implemented with aiohttp
        in order to be fully asynchronous)
      
      - store the job state in a new redis key "state:job:<ID>"
      
      - send notification to the redis pubsub 'notify:aio' channel when
        the job state has changed or when new logs are available
      
      In the allgo.aio frontend:
      
      - implement the /aio/jobs/<ID>/events endpoints which streams all
        job events & logs to the client (using json formatted messages)
      
      In django:
      
      - refactor the JobDetail view and template to update the page
        dynamically for job updates (state/logs)
          - allgo.log is read only when the job is already terminated.
            Otherwise the page uses the /aio/jobs/<ID>/events channel
            to stream the logs
          - the state icon is patched on the page when the state changes,
            except for the DONE state which triggers a full page reload
            (because there are other parts to be updated)
      1bb4acf4
    • BAIRE Anthony's avatar
      introduce a special token for the controller · b0807974
      BAIRE Anthony authored
      Initially the authentication with the registry was performed with a TLS
      client certificate installed on the controller.
      
      The registry is now configured to use token-based authentication (to
      give access to the users), but unfortunately it cannot be configured to
      support multiple auth methods. So we have to provide a token for the
      controller.
      
      This is a 'God' token, it gives total access (pull and push) on all
      images.
      b0807974
    • BAIRE Anthony's avatar
    • BAIRE Anthony's avatar
      Add a reddis client & reddis listener task to the aio server · 7fa7818d
      BAIRE Anthony authored
      listens for notifications on channel 'notify:aio'
      
      format of the notifications messages  "TYPE:ID"
      (eg: "job:127" for notifying that there is an update on the
       job with id 127)
      7fa7818d
    • BAIRE Anthony's avatar
      Add an auxiliary HTTP server (allgo.aio) for serving asynchronous requests · c5cd2bc1
      BAIRE Anthony authored
      There are two purposes:
      - implement server push (using long-lived HTTP requests) for:
          - sending status updates for the jobs and sandboxes
          - live-streaming of the job logs
      - have a really async implementation for pushing image manifests into
        the registry (the preliminary implementation in
        5451a6df was blocking)
      
      It is implemented with aiohttp.web (a lighweight HTTP framework,
      similar to what we can do with flask but asynchronously).
      
      The alternative would have been to use the django channels plugin, but:
      - it went through a major refactoring (v2) recently
      - it requires replacing unicorn with an ASGI server (daphne)
      - django-channels and daphne are not yet debian, and for the HTTP server
        i would prefer an implementation for which we have stable security
        updates
      
      (anyway this can be ported to django-channels later)
      
      The nginx config redirects the /aio/ path to this server (and the image
      manifests pushes too).
      
      The allgo.aio server interacts only with the nginx, reddis and django
      containers. It relies totally on the django server for authenticating
      the user and for accessing the mysql db (so there is no ORM).
      
      NOTE: in this design the django server has to trust blindly the requests
      coming from the allgo.aio server (for some endpoints). To prevent
      security issues, the nginx configuration is modified to set the header
      'X-Origin: nginx'. Thus django knowns who he can trust.
      
      This commits implements only the image pushs. Job updated and logs
      streaming will come in a later pull request.
      c5cd2bc1
    • BAIRE Anthony's avatar
      1cf27475
    • BAIRE Anthony's avatar
      reshape the job logs window · b1d9d3aa
      BAIRE Anthony authored
      - make it scrollable
      - use 'console' instead of 'json' as language name because json
        highlighting is not appropriate ('console' is not known by prism
        so it does not do any highlighting at all)
      
      Note: one thing that could be useful to highlight is the VT100 color
      codes
      b1d9d3aa
    • BAIRE Anthony's avatar
      fix param name in job detail view · 880e1e2a
      BAIRE Anthony authored
      880e1e2a
    • BERJON Matthieu's avatar
      Update of the home view · fac1174e
      BERJON Matthieu authored
      
      
      I rewrote the home view using the CBV approach given by Django. I
      updated the `url.py` file accordingly.
      Signed-off-by: BERJON Matthieu's avatarMatthieu Berjon <matthieu.berjon@inria.fr>
      fac1174e
  7. 26 Jun, 2018 4 commits
    • BAIRE Anthony's avatar
      render the job status with a template filter · f87d9f4c
      BAIRE Anthony authored
      This makes much less code (and possibly less bugs!)
      
      Also I did two other changes:
      
      - removed the "result" field from the template and use the 'status'
        field instead. Actually the status is what we display to the user (the
        'state' and 'result' fields are internal to allgo).
      
        The status is the textual representation of:
        - the 'result' field if the job is terminated (success, error, timeout
          or aborted)
        - the 'state' field otherwise (new, waiting, running, aborting)
      
      - changed the colors of the waiting and timeout state
        - waiting:  orange -> yellow
        - timeout:  yellow -> orange
        The rationale is that 'waiting' is a more normal state than
        'timeout' (thus the timeout color should be closer to red)
      
      - introduce a separate icon for the 'aborting' state (orange fa-play)
      f87d9f4c
    • BAIRE Anthony's avatar
    • BAIRE Anthony's avatar
      fix the type of Job.container_id · 7982242d
      BAIRE Anthony authored
      (this is a docker container id, not a foreign key)
      7982242d
    • BAIRE Anthony's avatar
      mandate minimum token size · 06807c69
      BAIRE Anthony authored
      06807c69