1. 08 Mar, 2018 7 commits
    • BERJON Matthieu's avatar
      Add of the Supervisor configuration · c49cdc57
      BERJON Matthieu authored
      
      
      I created the Supervisor configuration file. At the moment, the
      configuration hasn't been tested because of the incompatibilities of
      Supervisor with Python 3. It still needs to be resolved.
      
      The configuration has been copied from Rails and adapted to run a Django
      application and later on a Redis cache system (commented at the moment).
      Signed-off-by: BERJON Matthieu's avatarMatthieu Berjon <matthieu.berjon@inria.fr>
      c49cdc57
    • BERJON Matthieu's avatar
      Add of the app running script · 28e2b935
      BERJON Matthieu authored
      
      
      In order to launch the app, I create this script. It just launch the
      Django internal HTTP server on the port 4000 (later on, it should
      handled by Gunicorn or any WSGI server).
      Then I launch the nginx server.
      Signed-off-by: BERJON Matthieu's avatarMatthieu Berjon <matthieu.berjon@inria.fr>
      28e2b935
    • BERJON Matthieu's avatar
      Add Nginx configuration file · e456fcb8
      BERJON Matthieu authored
      
      
      In order to mimic the Rails container using Django, I installed a Nginx
      reverse proxy that communicate directly with the Django application
      (through Gunicorn or directly).
      
      The current configuration is a copy of the Rails configuration except
      the listening port.
      Signed-off-by: BERJON Matthieu's avatarMatthieu Berjon <matthieu.berjon@inria.fr>
      e456fcb8
    • BERJON Matthieu's avatar
      Update of the Nginx configuration · 2925d93e
      BERJON Matthieu authored
      
      
      I updated the Nginx configuration that is generated in the Python
      script. At the moment the configuration is hard-coded and need to be
      improved. It's working as such at the moment.
      Signed-off-by: BERJON Matthieu's avatarMatthieu Berjon <matthieu.berjon@inria.fr>
      2925d93e
    • BERJON Matthieu's avatar
      Update of the initialisation process · 574022a0
      BERJON Matthieu authored
      
      
      I update the initialisation process (that was empty before).
      I create the different folders to host the logs, cache and others.
      
      I create a symbolic link for the following:
      
      - a run-allgo script that instanciate the Django app and Nginx
      - supervisor configuration (at the moment, commented)
      - nginx configuration
      
      I then execute the run-algo script. At the moment, it seems that the
      execution of this script either doesn't occur or create an error with
      docker compose at launch (the dev-django container doesn't run). So at
      the moment, I run by hand the script once the container is up.
      Signed-off-by: BERJON Matthieu's avatarMatthieu Berjon <matthieu.berjon@inria.fr>
      574022a0
    • BERJON Matthieu's avatar
      Update of the packages installation and container initialisation · 1ff796b1
      BERJON Matthieu authored
      
      
      I updated the packages installed in order to:
      
      - remove some unecessary packages related mainly to Rails
      - add packages relation to python 3 and Django
      
      I updated the container initialisation that seems to fit the current
      practice with other container initialisation process.
      
      One thing to mention. Supervisor can't work with Python 3. Anthony
      suggested to install Python 2 just to make Supervisor happy and run
      Gunicorn/Python 3 as such.
      Signed-off-by: BERJON Matthieu's avatarMatthieu Berjon <matthieu.berjon@inria.fr>
      1ff796b1
    • BERJON Matthieu's avatar
      Delete of some commented commands · 571c0f4b
      BERJON Matthieu authored
      
      
      I deleted some commands that were commented because they were specific
      to rails and not relevant in the Django context.
      Signed-off-by: BERJON Matthieu's avatarMatthieu Berjon <matthieu.berjon@inria.fr>
      571c0f4b
  2. 05 Mar, 2018 2 commits
  3. 02 Mar, 2018 4 commits
    • BERJON Matthieu's avatar
      Add of the container initialization · 6825071b
      BERJON Matthieu authored
      
      
      I added a container_init file as it's required by the stack. It will be
      used later on anyway to setup the Django app.
      Signed-off-by: BERJON Matthieu's avatarMatthieu Berjon <matthieu.berjon@inria.fr>
      6825071b
    • BERJON Matthieu's avatar
      Copy of the container_init · fa87f2a5
      BERJON Matthieu authored
      
      
      Thanks to Sébastien, I updated the Dockerfile in order to resolve issue
      realted to unknown container_init. In the end, I copy this file into the
      init file and give the right permission to it.
      
      I removed as welle the symbolic link to Python3 as it seems to be the
      standard on this Debian image.
      
      I removed the command "run-allgo" as it was coming from the rails
      Dockerfile which is not relevant at this stage of the image.
      Signed-off-by: BERJON Matthieu's avatarMatthieu Berjon <matthieu.berjon@inria.fr>
      fa87f2a5
    • BERJON Matthieu's avatar
      Add of the Django container · 9bc9baa5
      BERJON Matthieu authored
      
      
      I modified the main file to build the new django image called
      dev-django. I added the folder in the bootstrap and prepare.sh file.
      I added the necessary configuration in the docker-compose.yaml based on
      the rails example.
      
      At the moment, the container by itself doesn't do anything except having
      the python stack installed.
      Signed-off-by: BERJON Matthieu's avatarMatthieu Berjon <matthieu.berjon@inria.fr>
      9bc9baa5
    • BERJON Matthieu's avatar
      Add Dockerfile for the Django app · e69a28a2
      BERJON Matthieu authored
      
      
      I added the Dockerfile for the future Django application. It's based on
      the rails Dockerfile with the following tweaks:
      
      - Installation of Python 3 and pip
      - Edit of the Python symbolic link to force the use of Python 3 by
      default
      
      I decided to store the app in the same folder as the rails app was as
      they are not meant to work side by side. It will also facilitate the
      debugging for developpers who were working on the rails app. If they
      need to look into it for debbugging reasons, the path where to find it
      is still the same.
      Signed-off-by: BERJON Matthieu's avatarMatthieu Berjon <matthieu.berjon@inria.fr>
      e69a28a2
  4. 15 Feb, 2018 1 commit
  5. 29 Jan, 2018 2 commits
  6. 08 Jan, 2018 3 commits
  7. 21 Nov, 2017 2 commits
    • BAIRE Anthony's avatar
      always display the memory limit · 0aa7a6aa
      BAIRE Anthony authored
      but disable the widget for non-admins
      0aa7a6aa
    • BAIRE Anthony's avatar
      fix out of memory message · 06420260
      BAIRE Anthony authored
      this variant is closer to the actual meaning
      (the fact that the limit was reached does not automatically imply
       that the process is starving, we cannot decide how much memory
       a process needs without doing some profiling)
      06420260
  8. 20 Nov, 2017 11 commits
  9. 16 Nov, 2017 1 commit
  10. 14 Nov, 2017 7 commits
    • BAIRE Anthony's avatar
      update the job container command · 2cfd30b8
      BAIRE Anthony authored
      - to have SIGTERM forwarded to the process
      - to propagate the exit code of the process
      2cfd30b8
    • BAIRE Anthony's avatar
      order jobs by id in the job list · 9a4b2818
      BAIRE Anthony authored
      (should be less confusing)
      9a4b2818
    • BAIRE Anthony's avatar
      add the 'rescheduled' future · 7d5df09e
      BAIRE Anthony authored
      to let the task implementations detect that they are being rescheduled
      7d5df09e
    • BAIRE Anthony's avatar
      extend shutdown case (in test_manager_futures) · 9f37db99
      BAIRE Anthony authored
      - add a pending task
      - add a timeout
      9f37db99
    • BAIRE Anthony's avatar
      factorisation · 5173f5e1
      BAIRE Anthony authored
      (disable_future_warning)
      5173f5e1
    • BAIRE Anthony's avatar
      add --verbose · 6a2a886f
      BAIRE Anthony authored
      console log level:
       (default)    ->  WARNING
       -v/--verbose ->  INFO
       -d/--debug   ->  DEBUG
      
      log files:
        /vol/log/controller.log   -> INFO
        /vol/log/debug.log        -> DEBUG  (disabled unless -d/--debug or
                                     unless env var DEBUG is set)
      6a2a886f
    • BAIRE Anthony's avatar
      refactor the management of swarm/sandbox resources · 0e301e74
      BAIRE Anthony authored
      - add SwarmAbstractionClient: a class that extends docker.Client and
        hides the API differences between the docker remote API and the
        swarm API. Thus a single docker engine can be used like a swarm
      
      - add SharedSwarmClient: a class that extends SwarmAbstractionClient
        and monitors the swarm health and its resource (cpu/mem) and manages
        the resource allocation.
        - resources are partitioned in groups (to allow reserving resources
          for higher priority jobs)
        - two SharedSwarmClient can work together over TCP in a master/slave
          configuration (to allow the production and qualification platforms
          to use the same swarm without any interference)
      
      - the controller is modified to:
        - use SharedSwarmClient to:
          - wait for the end of a job (in place of DockerWatcher)
          - manage resource reservation (LONG_APPS vs. BIGMEM_APPS vs normal
            apps) and monitor swarm health (fix #124)
          - NOTE: resources of the swarm and sandbox are now managed
            separately (2 instances of SharedSwarmClient), whereas it was
            global before (this was suboptimal)
        - rely on SwarmAbstractionClient to compute the cpu quotas
        - store the container_id of jobs into the DB (fix #128), this is a
          prerequisite to permit renaming apps in the future
        - store the class of the job (normal vs. long app) in the container
          name (for the resource management with SharedSwarmClient)
        - read the configuration from a yaml file (/vol/ro/config.yml) for:
          - cpu/mem quotas
          - swarm resources allocation policy
          - master/slave configuration
      0e301e74