Commit 979954ef authored by CAMPION Sebastien's avatar CAMPION Sebastien

Merge branch 'django' of gitlab.inria.fr:allgo/allgo into django

parents b8397d81 cb875ab4
Pipeline #89691 failed with stages
in 3 minutes and 1 second
...@@ -11,3 +11,4 @@ __pycache__ ...@@ -11,3 +11,4 @@ __pycache__
.coverage .coverage
/controller/htmlcov /controller/htmlcov
/.env /.env
metrics/*.stats
image: docker:latest
variables:
DOCKER_DRIVER: overlay2
stages: stages:
- build - build
- test
- cleanning
before_script:
- docker info
- docker-compose --version
- pwd
# ---------------------------
bootstrap: bootstrap:
tags:
- allgo
stage: build stage: build
script: script:
- docker info - rm -f .env
- apk update - sudo rm -rf data/*
- apk upgrade - ./bootstrap
- apk add python python-dev py-pip build-base bash openssl python3
- pip install docker-compose
- mkdir -p /data/dev
- rm -Rf /data/dev/*
- /bin/bash bootstrap dev-mysql dev-controller dev-ssh dev-django dev-nginx dev-smtpsink dev-registry
django_pylint:
stage: test
# only:
# - /django/
script:
- echo $PYLINTHOME
- docker exec -i dev-django pylint3 --rcfile=.pylintrc allgo
allow_failure: true
#~ django_test:
#~ stage: test
#~ # only:
#~ # - /django/
#~ script:
#~ - docker exec -i dev-django python3 manage.py test
nettoyage:
stage: cleanning
when: always
script:
- sudo rm -rf data/*
...@@ -4,13 +4,14 @@ ALLGO containers ...@@ -4,13 +4,14 @@ ALLGO containers
Overview Overview
-------- --------
A minimal deployment of allgo consists of 4 docker images: A minimal deployment of allgo consists of 6 docker images:
- **allgo/rails**: the rails application server - **allgo/redis** : the redis application server
- **allgo/mysql**: the mysql database server - **allgo/django** : the django application server
- **allgo/mysql** : the mysql database server
- **allgo/controller**: the manager for user docker containers - **allgo/controller**: the manager for user docker containers
- **allgo/ssh**: the ssh frontend (giving access to the sandboxes) - **allgo/ssh** : the ssh frontend (giving access to the sandboxes)
- **allgo/toolbox**: an image containing a set of commands (scp, vi, nano, - **allgo/toolbox** : an image containing a set of commands (scp, vi, nano,
less, ...) to be mounted in the user sandboxes less, ...) to be mounted in the user sandboxes
These images may be deployed multiple times to implement multiple independent These images may be deployed multiple times to implement multiple independent
...@@ -27,7 +28,7 @@ There is an extra image used only in development: ...@@ -27,7 +28,7 @@ There is an extra image used only in development:
- **allgo/smtpsink**: a SMTP server that catches and stores all incoming messages into a single mailbox - **allgo/smtpsink**: a SMTP server that catches and stores all incoming messages into a single mailbox
Each environment has its own docker network. The nginx container is connected Each environment has its own docker network. The nginx container is connected
to all these networks to that it can connect to the rails servers. to all these networks.
License License
...@@ -44,7 +45,7 @@ All docker images use the following conventions. ...@@ -44,7 +45,7 @@ All docker images use the following conventions.
### External volumes ### External volumes
They data is stored in: Their data are stored in:
- `/vol/rw` for persistent data - `/vol/rw` for persistent data
- `/vol/ro` for persistent data in read-only access - `/vol/ro` for persistent data in read-only access
...@@ -84,14 +85,13 @@ It provides 8 containers: ...@@ -84,14 +85,13 @@ It provides 8 containers:
All external volumes are stored in `/data/dev/` (the path is absolute because All external volumes are stored in `/data/dev/` (the path is absolute because
it is tricky to use a relative path with the allgo/docker image). it is tricky to use a relative path with the allgo/docker image).
For convenience, all containers not running as root (rails, mysql, registry) For convenience, all containers not running as root (django, mysql, registry)
have their user overridden to the UID:GID of the developer running have their user overridden to the UID:GID of the developer running
docker-compose. This is managed with the `DOCKERUSER` environment variable set docker-compose. This is managed with the `DOCKERUSER` environment variable set
[in the `.env` [in the `.env` file](https://docs.docker.com/compose/environment-variables/#the-env-file) by
file](https://docs.docker.com/compose/environment-variables/#the-env-file) by
`prepare.sh`. `prepare.sh`.
For convenience (again), there is an extra external volumes for `dev-rails`, For convenience (again), there is an extra external volume for `dev-django`,
`dev-controller` and `dev-ssh` so that the source directory of the app is mounted `dev-controller` and `dev-ssh` so that the source directory of the app is mounted
inside `/opt/` (in fact it overrides the actual application files provided by inside `/opt/` (in fact it overrides the actual application files provided by
the docker image). The purpose is to avoid rebuilding a new docker image for the docker image). The purpose is to avoid rebuilding a new docker image for
...@@ -100,16 +100,15 @@ each development iteration. ...@@ -100,16 +100,15 @@ each development iteration.
### Getting started ### Getting started
The sources are located in two repositories: The sources are located in one repository:
- *rails-allgo*: the rails application repository
- *allgo*: the deployment repository - *allgo*: the deployment repository
To set up the development environment, run: To set up the development environment, run:
1. get the sources 1. get the sources
<pre> <pre>
git clone git@gitlab.inria.fr:allgo/allgo.git git clone git@gitlab.inria.fr:allgo/allgo.git
cd allgo cd allgo
</pre> </pre>
...@@ -117,19 +116,19 @@ To set up the development environment, run: ...@@ -117,19 +116,19 @@ To set up the development environment, run:
2. *(as root)* create `/data/dev` and make it owned by the developer 2. *(as root)* create `/data/dev` and make it owned by the developer
<pre> <pre>
sudo mkdir -p /data/dev sudo mkdir -p /data/dev
sudo chown USER: /data/dev sudo chown $USER: /data/dev
</pre> </pre>
3. bootstrap the environment 3. bootstrap the environment
<pre> <pre>
./bootstrap ./bootstrap
</pre> </pre>
This command will run the `/dk/init_container` in every container that This command will run the `/dk/init_container` in every container that
needs it, then start the container. needs it, then start the container.
The first run takes a very long time because all images are built from The first run takes a few minutes because all images are built from
scratch (especially the rails image which builds ruby source). scratch.
You have enough time for a coffee break. You may have enough time for a short coffee break.
**Note** by default `bootstrap` works on all containers. It is possible **Note** by default `bootstrap` works on all containers. It is possible
to give an explicit list of containers instead. Example: to give an explicit list of containers instead. Example:
...@@ -171,34 +170,34 @@ The official doc for docker-compose is available at: [https://docs.docker.com/co ...@@ -171,34 +170,34 @@ The official doc for docker-compose is available at: [https://docs.docker.com/co
</pre> </pre>
- hard cleanup (remove images too) - hard cleanup (remove images too)
<pre> <pre>
fig down --rmi local fig down --rmi local
</pre> </pre>
- restart a container - restart a container
<pre> <pre>
fig restart dev-rails fig restart dev-django
</pre> </pre>
- restart a container using a new docker image (if the image has been rebuilt since the last start) - restart a container using a new docker image (if the image has been rebuilt since the last start)
<pre> <pre>
fig up dev-rails fig up dev-django
</pre> </pre>
- rebuild an image - rebuild an image
<pre> <pre>
fig build dev-railf fig build dev-django
</pre> </pre>
- **Note:** most commands work on every container by default (eg: up down - **Note:** most commands work on every container by default (eg: up down
start stop restart ...) they can be use on an individual container too: start stop restart ...) they can be use on an individual container too:
<pre> <pre>
fig restart dev-controller dev-rails fig restart dev-controller dev-django
</pre> </pre>
- run a container with an arbitrary command (eg: to have access to the rails console) - run a container with an arbitrary command (eg: to have access to the django console)
<pre> <pre>
fig run --rm dev-rails bash fig run --rm dev-django bash
</pre> </pre>
**Note:** containers created by `fig run` have the same parameters as **Note:** containers created by `fig run` have the same parameters as
...@@ -206,10 +205,10 @@ The official doc for docker-compose is available at: [https://docs.docker.com/co ...@@ -206,10 +205,10 @@ The official doc for docker-compose is available at: [https://docs.docker.com/co
*allgo_dev-ssh_run_1*), which means that this container is not *allgo_dev-ssh_run_1*), which means that this container is not
reachable by the others (this may be an issue for example if you want reachable by the others (this may be an issue for example if you want
to run the mysqld server manually: `fig run dev-mysql mysqld` -> this to run the mysqld server manually: `fig run dev-mysql mysqld` -> this
container won't be reachable by the ssh and rails containers) container won't be reachable by the ssh and django containers)
- follow the output of all containers: - follow the output of all containers:
<pre> <pre>
fig logs --tail=1 --follow fig logs --tail=1 --follow
</pre> </pre>
...@@ -249,7 +248,7 @@ it as root**, otherwise it will be owned by root and you may have errors like: ...@@ -249,7 +248,7 @@ it as root**, otherwise it will be owned by root and you may have errors like:
If somehow you skipped this step, you can reset the ownership to the current user: If somehow you skipped this step, you can reset the ownership to the current user:
sudo chown USER: /data/dev sudo chown USER: /data/dev
sudo chown -R USER: /data/dev/{registry,mysql,rails} sudo chown -R USER: /data/dev/{registry,mysql,django}
If you are completely lost, you can just restart the initialisation from scratch: If you are completely lost, you can just restart the initialisation from scratch:
...@@ -289,22 +288,21 @@ Hosts a mysql server listening on port 3306 with two databases: `allgo` and ...@@ -289,22 +288,21 @@ Hosts a mysql server listening on port 3306 with two databases: `allgo` and
- `ssh` has read only access to `allgo` - `ssh` has read only access to `allgo`
## rails ## django
Hosts four daemons for running allgo:
- the unicorn server (runnning the rails application) Hosts three daemons for running the allgo web server:
- the sidekiq queue manager
- the redis db server - a nginx frontend for buffering the HTTP requests/responses and routing them
- a nginx frontend for buffering the HTTP requests/responses to the other daemons. It also serves static files directly
- the gunicorn server (running the django application)
- the allgo.aio server (serving the asynchronous requests)
This container is managed with supervisor, the `supervisorctl` command allows
starting/stopping the daemons individually.
### Running the rails server manually ### Running the django server manually
TODO ?
- run the `dev-rails` container and open a shell: [comment]: # ( - run the `dev-rails` container and open a shell:
<pre> <pre>
fig up -d fig up -d
docker exec -t -i dev-rails bash docker exec -t -i dev-rails bash
...@@ -315,7 +313,7 @@ starting/stopping the daemons individually. ...@@ -315,7 +313,7 @@ starting/stopping the daemons individually.
supervisorctl stop rails supervisorctl stop rails
rails server rails server
</pre> </pre>
)
## ssh ## ssh
...@@ -331,7 +329,7 @@ WEBAPP@sid.allgo.irisa.fr`). Each allgo webapp is mapped to a system user ...@@ -331,7 +329,7 @@ WEBAPP@sid.allgo.irisa.fr`). Each allgo webapp is mapped to a system user
gid = 65534 (nogroup) gid = 65534 (nogroup)
gecos = webapps.name gecos = webapps.name
shell = /bin/allgo-shell shell = /bin/allgo-shell
</pre> </pre>
- The ssh server is configured to accept key-based authentication only. The - The ssh server is configured to accept key-based authentication only. The
list of public keys is obtained from the (using an AuthorizedKeysCommand). list of public keys is obtained from the (using an AuthorizedKeysCommand).
...@@ -340,12 +338,12 @@ WEBAPP@sid.allgo.irisa.fr`). Each allgo webapp is mapped to a system user ...@@ -340,12 +338,12 @@ WEBAPP@sid.allgo.irisa.fr`). Each allgo webapp is mapped to a system user
- The connection to the sandbox is made though a unix socket and a set of pipes - The connection to the sandbox is made though a unix socket and a set of pipes
in the filesystem. in the filesystem.
## docker ## controller
Hosts the *docker-allgo-proxy* which manages all docker operations (run, stop, Hosts the *docker-controller* which manages all docker operations (run, stop,
rm, commit, pull, push, ...) on behalf of the rails container. rm, commit, pull, push, ...) on behalf of the django container.
Technically speaking this container had root privileges since it has access to Technically speaking this container has root privileges since it has access to
the docker socket. the docker socket.
The proxy script enforces restrictions (according to the current environment: eg prod/qualif/dev) on: The proxy script enforces restrictions (according to the current environment: eg prod/qualif/dev) on:
...@@ -370,3 +368,5 @@ mailbox. ...@@ -370,3 +368,5 @@ mailbox.
The mailbox is accessible with IMAP as user *sink* (password *sink*). The mailbox is accessible with IMAP as user *sink* (password *sink*).
NOTE: in the development environment, django's default is to dump outgoing
e-mails to the console. Thus this container is only useful in the qualif setup.
#!/bin/bash #!/bin/bash
CONTAINERS="dev-redis dev-mysql dev-controller dev-ssh dev-django dev-smtpsink dev-registry dev-nginx" CONTAINERS="dev-redis dev-mysql dev-controller dev-ssh dev-django dev-smtpsink dev-registry dev-nginx dev-toolbox"
die() die()
......
...@@ -29,4 +29,7 @@ ENV ENV="" \ ...@@ -29,4 +29,7 @@ ENV ENV="" \
ALLGO_REDIS_HOST="{ENV}-redis" \ ALLGO_REDIS_HOST="{ENV}-redis" \
ALLGO_IMPORT_REGISTRY="cargo.irisa.fr:8003/allgo/prod/webapp" ALLGO_IMPORT_REGISTRY="cargo.irisa.fr:8003/allgo/prod/webapp"
# to prevent __pycache__generation, which is owned by root.
ENV PYTHONDONTWRITEBYTECODE 1
LABEL dk.migrate_always=1 LABEL dk.migrate_always=1
This diff is collapsed.
...@@ -10,7 +10,9 @@ RUN apt-getq update && apt-getq install \ ...@@ -10,7 +10,9 @@ RUN apt-getq update && apt-getq install \
python-mysqldb python3-crypto gunicorn3 python3-redis python-mysqldb \ python-mysqldb python3-crypto gunicorn3 python3-redis python-mysqldb \
python3-crypto python3-natsort python3-aiohttp python3-aioredis supervisor \ python3-crypto python3-natsort python3-aiohttp python3-aioredis supervisor \
python3-ipy python3-django-taggit python3-iso8601 python3-robot-detection \ python3-ipy python3-django-taggit python3-iso8601 python3-robot-detection \
python3-sqlparse python3-sqlparse \
python3-django-extensions python3-pydotplus \
python3-pylint-django
COPY requirements.txt /tmp/ COPY requirements.txt /tmp/
RUN cd /tmp && pip3 install -r requirements.txt && rm requirements.txt RUN cd /tmp && pip3 install -r requirements.txt && rm requirements.txt
...@@ -22,6 +24,7 @@ USER allgo ...@@ -22,6 +24,7 @@ USER allgo
WORKDIR /opt/allgo WORKDIR /opt/allgo
LABEL dk.migrate_always=1 LABEL dk.migrate_always=1
ENV PYTHONUNBUFFERED 1 ENV PYTHONUNBUFFERED 1
ENV PYLINTHOME /opt/allgo_metrics
# NOTE: we use SIGINT instead of SIGTERM because the django server does not # NOTE: we use SIGINT instead of SIGTERM because the django server does not
# catch SIGTERM (while gunicorn catches both SIGTERM & SIGINT) # catch SIGTERM (while gunicorn catches both SIGTERM & SIGINT)
......
...@@ -7,5 +7,5 @@ app_name = 'api' ...@@ -7,5 +7,5 @@ app_name = 'api'
urlpatterns = [ urlpatterns = [
url(r'^jobs$', views.jobs, name='jobs'), url(r'^jobs$', views.jobs, name='jobs'),
url(r'^jobs/(?P<pk>\d+)', views.APIJobView.as_view(), name='job'), url(r'^jobs/(?P<pk>\d+)', views.APIJobView.as_view(), name='job'),
url(r'^datastore/(?P<pk>\d+)/(.*)/(.*)', views.APIDownloadView, name='download'), url(r'^datastore/(?P<pk>\d+)/(.*)', views.APIDownloadView.as_view(), name='download'),
] ]
...@@ -5,6 +5,7 @@ import config.env ...@@ -5,6 +5,7 @@ import config.env
from django.core.validators import ValidationError from django.core.validators import ValidationError
from django.http import JsonResponse from django.http import JsonResponse
from django.shortcuts import redirect from django.shortcuts import redirect
from django.urls import reverse
from django.views.decorators.csrf import csrf_exempt from django.views.decorators.csrf import csrf_exempt
from django.views.generic import View from django.views.generic import View
from main.helpers import upload_data, get_base_url, lookup_job_file, get_request_user from main.helpers import upload_data, get_base_url, lookup_job_file, get_request_user
...@@ -21,29 +22,21 @@ def get_link(jobid, dir, filename, request): ...@@ -21,29 +22,21 @@ def get_link(jobid, dir, filename, request):
return '/'.join((get_base_url(request), "datastore", str(jobid), filename)) return '/'.join((get_base_url(request), "datastore", str(jobid), filename))
def job_response(job, request):
status = {0: 'new',
1: 'waiting',
2: 'running',
3: 'done',
4: 'archived',
5: 'deleted',
6: 'aborting'}
job_dir = job.data_dir
files = {f: get_link(job.id, job_dir, f, request) for f in os.listdir(job_dir)
if lookup_job_file(job.id, f)}
return {str(job.id): {'status': status[job.state],
'files': files
}
}
class APIJobView(JobAuthMixin, View): class APIJobView(JobAuthMixin, View):
def get(self, request, pk): def get(self, request, pk):
try: try:
job = Job.objects.get(id=pk) job = Job.objects.get(id=pk)
return JsonResponse(job_response(job, request)) files = {}
for f in os.listdir(job.data_dir):
if lookup_job_file(job.id, f):
files[f] = get_link(job.id, job.data_dir, f, request)
response = {
job.id: files,
"status": job.get_state_display().lower(),
}
return JsonResponse(response)
except Job.DoesNotExist as e: except Job.DoesNotExist as e:
log.error("Job not found %s", str(e)) log.error("Job not found %s", str(e))
return JsonResponse({'error': 'Job not found'}, status=404) return JsonResponse({'error': 'Job not found'}, status=404)
...@@ -65,6 +58,10 @@ def jobs(request): ...@@ -65,6 +58,10 @@ def jobs(request):
if not app: if not app:
return JsonResponse({'error': 'Application not found'}, status=404) return JsonResponse({'error': 'Application not found'}, status=404)
if app.get_webapp_version() is None:
log.debug('No usable versions')
return JsonResponse({'error': "This app is not yet published"}, status=404)