GitLab

authorization is done by django route /auth if user is logged or by token

This way all controller notifications will be implemented in a single
place and the notifications will also be sent when updating the models
through the /admin UI.

(until now it was not the case)

I updated the breadcrumbs in order to start at the same place as the
logo.
Signed-off-by: Matthieu Berjon <matthieu.berjon@inria.fr>

I updated the commands for both the SSH and docker commands.
Signed-off-by: Matthieu Berjon <matthieu.berjon@inria.fr>

Now the form is displayed only on the SSH panel.
Signed-off-by: Matthieu Berjon <matthieu.berjon@inria.fr>

the redis notification could be received before the db entry is
written

(so that the code is not duplicated and so that the security checks
 are implemented in the API too)

'file_obj' is misleading, because it has a meaning in python
https://docs.python.org/3/glossary.html#term-file-object

also i fixed the doc, because the actual param is not a dict

(because the controller requires it to store the results)

(note: i used the name 'filepath' instead of 'endpoint' because it
 is less misleading: the variable containes the path of the file
 in the local filesystem whereas 'endpoint' would rather refer to
 the path in the http request)

This is to fix a vulnerability. The user submitting the job
**must not** be able to write a file outside the job dir
(for example by submitting a file named "../../something")

Two points
- it is not guaranteed to work properly in production because the
  gunicorn server is multithreaded (the umask may not be correctly
  restored if two jobs are created in the same time)
- it is actually not needed. In production, the datastore is a nfs
  volume exported with the 'all_squash' option (which makes all file
  accesses from the jobs made with the same uid as django).

Rationale: in development the ip address of the local machine is not
easily predictible because when docker creates virtual networks it
assigns the ip prefixes/addres dynamically by default (and i do not want
to configure static addresses because of it may interefere and cause
nasty side-effects if using docker for other projects on the same
machine)

Now in development we allow admin actions from 0.0.0.0/0 (which means
all ip addresses)

Note: I used the IPy package (whose purpose is to handle ranges of
IP addresses)

I did several changes to improve the readibility and look. This goes
from improving the explanation of the SSH setup to few modifications on
the margins. The form has been modified to put together the field
version and changelog.

Resolves: #208

Signed-off-by: Matthieu Berjon <matthieu.berjon@inria.fr>

The redirection of the webapp creation now redirects to the webapp
sandbox creation page instead of the user webapp list.
Signed-off-by: Matthieu Berjon <matthieu.berjon@inria.fr>

This fixes the bug related to the active pane. This is a small hook. It
shows that some parts of the code are duplicated across both templates
and should be refactored at some point.

Resolves: #204

Signed-off-by: Matthieu Berjon <matthieu.berjon@inria.fr>

I added a clearfix on the template to avoid the form to overflow on the
footer.
Signed-off-by: Matthieu Berjon <matthieu.berjon@inria.fr>

I added a a fixed height to the footer in order to prevent an overflow
of the footer on the data. This bug fix doesn't work on small viewport
and need the help of a media query to change the behaviour in this case.
Signed-off-by: Matthieu Berjon <matthieu.berjon@inria.fr>

hostname may be null

(because it's kind of confusing to have it stand there
 with nothing inside)

`rep.drain() `is potentially blocking, thus it should not be called
while the condition is locked (within the `async with cond`).

The starvation could happen if two clients are listening to the same
job and one of them has a low bandwidth and the job is producing lots
of logs (yeah that makes many ands, i know ;-)

- use `CMD ["run-allgo"]` so that the run-allgo command is executed
  directly (whereas `CMD run-allgo` launches it inside a shell
  which would receive the termination signal in place of django)

- use `SIGINT` instead of `SIGTERM` for stopping the container
  (because the django server ignores SIGTERM)

With these changes, stopping or restarting the container
(eg: `fig restart dev-django`) will be immediate

I redesigned the webapp template by removing the left hand menu. I
replaced it with a small right hand menu aligned with the app name.

I display either the readme file or the description according to what
exists.
Signed-off-by: Matthieu Berjon <matthieu.berjon@inria.fr>

The version of misaka used in development and in the docker container
weren't the same, bringing a type error. I fixed by using the right
syntax. The drawback of using an older version of misaka markdown
renderer is the lack of extesion support for some common features such
as footnotes and new syntax for links.
Signed-off-by: Matthieu Berjon <matthieu.berjon@inria.fr>

This commit removes the old notification channel (socket listening
on port 4567), and uses the redis channel 'notify:controller' instead.

The django job creation views are updated accordingly.

This commit makes several changes.

In the controller:

- duplicates the logs produced by the jobs. Initially they were only
  stored into allgo.log, now they are also forwarded to the container
  output (using the 'tee' command) so that the controller can read
  them

- add a log_task that reads the logs from docker and feeds them into
  the redis db key "log:job:<ID>" (this is implemented with aiohttp
  in order to be fully asynchronous)

- store the job state in a new redis key "state:job:<ID>"

- send notification to the redis pubsub 'notify:aio' channel when
  the job state has changed or when new logs are available

In the allgo.aio frontend:

- implement the /aio/jobs/<ID>/events endpoints which streams all
  job events & logs to the client (using json formatted messages)

In django:

- refactor the JobDetail view and template to update the page
  dynamically for job updates (state/logs)
    - allgo.log is read only when the job is already terminated.
      Otherwise the page uses the /aio/jobs/<ID>/events channel
      to stream the logs
    - the state icon is patched on the page when the state changes,
      except for the DONE state which triggers a full page reload
      (because there are other parts to be updated)

Initially the authentication with the registry was performed with a TLS
client certificate installed on the controller.

The registry is now configured to use token-based authentication (to
give access to the users), but unfortunately it cannot be configured to
support multiple auth methods. So we have to provide a token for the
controller.

This is a 'God' token, it gives total access (pull and push) on all
images.

listens for notifications on channel 'notify:aio'

format of the notifications messages  "TYPE:ID"
(eg: "job:127" for notifying that there is an update on the
 job with id 127)

There are two purposes:
- implement server push (using long-lived HTTP requests) for:
    - sending status updates for the jobs and sandboxes
    - live-streaming of the job logs
- have a really async implementation for pushing image manifests into
  the registry (the preliminary implementation in
  5451a6df was blocking)

It is implemented with aiohttp.web (a lighweight HTTP framework,
similar to what we can do with flask but asynchronously).

The alternative would have been to use the django channels plugin, but:
- it went through a major refactoring (v2) recently
- it requires replacing unicorn with an ASGI server (daphne)
- django-channels and daphne are not yet debian, and for the HTTP server
  i would prefer an implementation for which we have stable security
  updates

(anyway this can be ported to django-channels later)

The nginx config redirects the /aio/ path to this server (and the image
manifests pushes too).

The allgo.aio server interacts only with the nginx, reddis and django
containers. It relies totally on the django server for authenticating
the user and for accessing the mysql db (so there is no ORM).

NOTE: in this design the django server has to trust blindly the requests
coming from the allgo.aio server (for some endpoints). To prevent
security issues, the nginx configuration is modified to set the header
'X-Origin: nginx'. Thus django knowns who he can trust.

This commits implements only the image pushs. Job updated and logs
streaming will come in a later pull request.

- make it scrollable
- use 'console' instead of 'json' as language name because json
  highlighting is not appropriate ('console' is not known by prism
  so it does not do any highlighting at all)

Note: one thing that could be useful to highlight is the VT100 color
codes

I rewrote the home view using the CBV approach given by Django. I
updated the `url.py` file accordingly.
Signed-off-by: Matthieu Berjon <matthieu.berjon@inria.fr>

This makes much less code (and possibly less bugs!)

Also I did two other changes:

- removed the "result" field from the template and use the 'status'
  field instead. Actually the status is what we display to the user (the
  'state' and 'result' fields are internal to allgo).

  The status is the textual representation of:
  - the 'result' field if the job is terminated (success, error, timeout
    or aborted)
  - the 'state' field otherwise (new, waiting, running, aborting)

- changed the colors of the waiting and timeout state
  - waiting:  orange -> yellow
  - timeout:  yellow -> orange
  The rationale is that 'waiting' is a more normal state than
  'timeout' (thus the timeout color should be closer to red)

- introduce a separate icon for the 'aborting' state (orange fa-play)

(this is a docker container id, not a foreign key)