Commit 6d2cef61 authored by BAIRE Anthony's avatar BAIRE Anthony

Merge remote-tracking branch 'origin/django' into django-fix

parents fb08a468 1fb233f4
*.sql
.DS_Store
*.pyc
.idea
.*.sw[op]
.stamp.*
.deps.*
......
......@@ -4,6 +4,12 @@ set -e
. prepare.sh
mkdir -p certs
if [ ! -f cert/server.key ] & [ ! -f certs/server.crt ]; then
openssl req -subj '/CN=localhost/O=Registry Demo/C=US' -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout certs/server.key -out certs/server.crt
fi
# build base image (if not present)
(set -x ; make base-debian)
......
......@@ -12,14 +12,15 @@ RUN apt-get update && apt-get -t stretch-backports install -y \
python3-django python3-django-allauth
RUN apt-get install -y mysql-server default-libmysqlclient-dev \
nginx-light zip python3-dev python3-pip python3-mysqldb python-mysqldb \
nginx-light zip python3-dev python3-pip python3-mysqldb python-mysqldb python3-crypto \
supervisor && pip3 install gunicorn
COPY . /opt/allgo
COPY .env /opt/allgo
COPY init /dk/container_init
RUN pip3 install -r /opt/allgo/requirements.txt
RUN patch /etc/nginx/nginx.conf < /opt/allgo/setup/dk/nginx.patch
RUN rm /etc/nginx/sites-enabled/default && \
rm /etc/supervisor/supervisord.conf && \
......@@ -30,4 +31,5 @@ RUN rm /etc/nginx/sites-enabled/default && \
WORKDIR /opt/allgo
LABEL dk.migrate_always=1
ENV PYTHONUNBUFFERED 1
CMD run-allgo
=====
Allgo
=====
Django front-end for the Allgo (A||go) platform.
`Allgo`_ is a platform for building deploying apps that analyze massive data in
Linux containers, it has been specifically designed for use in scientific
applications. This documentation is related to its front-end. Please see the
:ref:`installation` to get started.
* License: None
* Documentation: please refer to the README file
Features
--------
Installation
============
The current version aims to reproduce the basic features offered by the rails
front-end such as:
Warning, because we use a legacy database, Django can't deal with it because
it doesn't have the history of the previous migrations. This process must be
applied each time that you create a new app calling a legacy database. The
following process must be applied:
- authentication
- documentation of the API
- creation of an app
- launch of a processing job
.. bash
# load the database
mysql -u root -h localhost < dump.sql
Installation
------------
# Tell Django the to start the migration at this stage of the database
python manage.py makemigrations <app_label>
Please refer to :ref:`installation` in the `docs` directory for a detailed
explaination of the setup.
# Create the migrations
python manage.py migrate --fake-initial
License
-------
This work is under a proprietary license.
From that point, you can add the any required fields and process to the
migrations as usual.
.. _Allgo: https://allgo.inria.fr/
......@@ -2,6 +2,8 @@ from __future__ import unicode_literals
from django.contrib.auth.models import User
from django.db import models
from django.db.models.signals import post_save
from django.dispatch import receiver
class TimeStampModel(models.Model):
......@@ -28,11 +30,15 @@ class AllgoUser(models.Model):
# Relationships
user = models.OneToOneField(
User, on_delete=models.CASCADE, related_name="allgouser")
User, on_delete=models.CASCADE, related_name="allgouser")
class Meta:
db_table = 'dj_users'
def getApp(self):
return [a.name for a in Webapp.objects.filter(user_id=self.user.id)]
class DockerOs(TimeStampModel):
"""
......@@ -105,9 +111,9 @@ class Webapp(TimeStampModel):
# A webapp has one docker os type
docker_os = models.OneToOneField(DockerOs, related_name="webappdockeros")
user = models.ForeignKey(
User, on_delete=models.CASCADE, related_name="webappuser")
User, on_delete=models.CASCADE, related_name="webappuser")
job_queue = models.OneToOneField(
JobQueue, on_delete=models.CASCADE, related_name="webappjobqueue")
JobQueue, on_delete=models.CASCADE, related_name="webappjobqueue")
class Meta:
db_table = 'dj_webapps'
......@@ -142,9 +148,10 @@ class WebappVersion(TimeStampModel):
docker_image_size = models.FloatField(blank=True, null=True)
state = models.IntegerField(null=True)
published = models.IntegerField()
url = models.TextField()
# Relationships
dj_webapp = models.ForeignKey(Webapp, related_name="webappversion")
webapp = models.ForeignKey(Webapp, related_name="webappversion")
class Meta:
db_table = 'dj_webapp_versions'
......@@ -163,9 +170,9 @@ class Quota(TimeStampModel):
# Relationships
dj_user = models.ForeignKey(
User, on_delete=models.CASCADE, related_name="quotauser")
User, on_delete=models.CASCADE, related_name="quotauser")
dj_webapp = models.ForeignKey(
Webapp, on_delete=models.CASCADE, related_name="quotawebapp")
Webapp, on_delete=models.CASCADE, related_name="quotawebapp")
class Meta:
db_table = 'dj_quotas'
......@@ -207,10 +214,21 @@ class JobUploads(TimeStampModel):
job_file_content_type = models.CharField(max_length=255, blank=True)
job_file_file_size = models.IntegerField(blank=True, null=True)
job_file_updated_at = models.DateTimeField(
blank=True, auto_now_add=True, null=True)
blank=True, auto_now_add=True, null=True)
# Relationships
dj_job = models.ForeignKey(Job, related_name="jobuploadjob")
class Meta:
db_table = 'dj_job_uploads'
@receiver(post_save, sender=User)
def create_user_profile(sender, instance, created, **kwargs):
if created:
AllgoUser.objects.create(user=instance)
@receiver(post_save, sender=User)
def save_user_profile(sender, instance, **kwargs):
instance.allgouser.save()
import os
import time
import hashlib
import base64
import subprocess
from Crypto.PublicKey import RSA
from jose import jwt
SIGNING_KEY_PATH = os.environ.get('SIGNING_KEY_PATH')
SIGNING_KEY_TYPE = os.environ.get('SIGNING_KEY_TYPE')
SIGNING_KEY_ALG = os.environ.get('SIGNING_KEY_ALG')
SIGNING_KEY = open(SIGNING_KEY_PATH).read()
ISSUER = os.environ.get('ISSUER')
TOKEN_EXPIRATION = os.environ.get('TOKEN_EXPIRATION')
TOKEN_TYPE = os.environ.get('TOKEN_TYPE')
def run_command(command):
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
return process.communicate()
def key_id_encode(the_bytes):
source = base64.b32encode(the_bytes)
result = []
for i in range(0, len(source), 4):
start = i
end = start+4
result.append(str(source[start:end], 'utf-8'))
return ":".join(result)
def kid_from_crypto_key(private_key_path, key_type):
"""
python implementation of
https://github.com/jlhawn/libtrust/blob/master/util.go#L192
returns a distinct identifier which is unique to
the public key derived from this private key.
The format generated by this library is a base32 encoding of a 240 bit
hash of the public key data divided into 12 groups like so:
ABCD:EFGH:IJKL:MNOP:QRST:UVWX:YZ23:4567:ABCD:EFGH:IJKL:MNOP
"""
assert key_type == 'RSA', "Only RSA key type supported"
algorithm = hashlib.sha256()
key = RSA.importKey(open(private_key_path).read())
der = key.publickey().exportKey("DER")
algorithm.update(der)
return key_id_encode(algorithm.digest()[:30])
class Token(object):
def __init__(self, service, access_type="", access_name="",
access_actions=None, subject=''):
if access_actions is None:
access_actions = []
self.issuer = ISSUER
self.signing_key = SIGNING_KEY
self.signing_key_path = SIGNING_KEY_PATH
self.signing_key_type = SIGNING_KEY_TYPE
self.signing_key_alg = SIGNING_KEY_ALG
self.token_expiration = TOKEN_EXPIRATION
self.token_type = TOKEN_TYPE
self.header = {
'typ': self.token_type,
'alg': self.signing_key_alg,
'kid': kid_from_crypto_key(self.signing_key_path, self.signing_key_type)
}
self.claim = {
'iss': self.issuer,
'sub': subject,
'aud': service,
'exp': int(time.time()) + int(self.token_expiration),
'nbf': int(time.time()) - 30,
'iat': int(time.time()),
'access': [
{
'type': access_type,
'name': access_name,
'actions': access_actions
}
]
}
def set_header(self, header):
self.header = header
def get_header(self):
return self.header
def set_claim(self, claim):
self.claim = claim
def get_claim(self):
return self.claim
def encode_token(self):
token = jwt.encode(self.claim, self.signing_key,
algorithm=self.signing_key_alg,
headers=self.header)
return token
......@@ -5,15 +5,10 @@ app_name = 'main'
urlpatterns = [
url(r'^$', views.index, name="home"),
url(r'^tokens$', views.tokens, name="tokens"),
url(r'^registryhook', views.registryhook, name="registryhook"),
url(r'^apps/$', views.WebappList.as_view(), name='webapp_list'),
url(r'^app/(?P<docker_name>[\w-]+)/$',
views.WebappDetail.as_view(),
name='webapp_detail'),
url(r'^app/(?P<docker_name>[\w-]+)/$', views.WebappDetail.as_view(), name='webapp_detail'),
url(r'^jobs/$', views.JobList.as_view(), name='job_list'),
# Settings
url(
r'^settings/$',
views.UserUpdateView.as_view(),
name='user_settings'),
url(r'^settings/$', views.UserUpdateView.as_view(), name='user_settings'),
]
import base64
import json
import socket
import logging
import os
from django.contrib.auth.mixins import LoginRequiredMixin
from django.views.decorators.csrf import csrf_exempt
from django.contrib.auth.models import User
from django.shortcuts import render, get_object_or_404
from django.http import JsonResponse, HttpResponse
from django.urls import reverse
from django.views.generic import (
ListView,
DetailView,
UpdateView,
)
ListView,
DetailView,
UpdateView,
)
from .tokens import Token
from .models import Webapp, Job, AllgoUser, WebappVersion
log = logging.getLogger('allgo')
def get_allowed_actions(user, scope, actions):
allgouser = AllgoUser(user=user)
resource_type, resource_name, resource_actions = scope.split(":")
if resource_type == "repository" and resource_name.rstrip('-incoming') in allgouser.getApp():
return actions
else:
return []
from .models import Webapp, Job, AllgoUser
def update_webapp_metadata(repository, url):
log.info("New webapp version received %s %s", repository, url)
webapp = Webapp.objects.filter(name=repository)[0]
WebappVersion(url=url, state=0, published=0, webapp=webapp).save()
notify_controler()
def notify_controler():
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(5)
sock.connect((os.environ.get('ALLGO_CONTROLLER_HOST'), int(os.environ.get('ALLGO_CONTROLLER_PORT'))))
sock.settimeout(None)
sock.makefile('rb', 0)
sock.shutdown(socket.SHUT_RDWR)
sock.close()
log.info("Controller notified")
def index(request):
return render(request, "home.html")
@csrf_exempt
def registryhook(request):
body_unicode = request.body.decode('utf-8')
body = json.loads(body_unicode)
for event in body['events']:
if event['action'] == "push":
update_webapp_metadata(event['target']['repository'], event['target']['url'])
return HttpResponse(status=200)
def tokens(request):
auth_header = request.META.get('HTTP_AUTHORIZATION', '')
if not auth_header:
log.info("Token request without http authorisation %s %s %s", request.META['HTTP_USER_AGENT'],
request.META['REMOTE_ADDR'], request.META['QUERY_STRING'])
return HttpResponse(status=401)
token_type, credentials = auth_header.split(' ')
username, password = base64.b64decode(credentials).decode('utf-8').split(':')
try:
user = User.objects.get(email=username)
except User.DoesNotExist:
log.warning("Token request but user doest not exist")
return HttpResponse(status=401)
password_valid = user.check_password(password)
if token_type != 'Basic' or not password_valid:
log.info("Token request but user password mismatch")
return HttpResponse(status=401)
service = request.GET['service']
scope = request.GET['scope'] if 'scope' in request.GET.keys() else None
if not scope:
typ = ''
name = ''
actions = []
else:
params = scope.split(':')
if len(params) != 3:
return JsonResponse({'error': 'Invalid scope parameter'}, status=400)
typ = params[0]
name = params[1]
actions = params[2].split(',')
authorized_actions = get_allowed_actions(user, scope, actions)
token = Token(service, typ, name, authorized_actions)
encoded_token = token.encode_token()
return JsonResponse({'token': encoded_token})
class WebappList(ListView):
"""
Display the paginated list of available webapps.
......@@ -69,8 +158,8 @@ class UserUpdateView(LoginRequiredMixin, UpdateView):
def get_success_url(self):
return reverse(
'main:user_settings',
kwargs={'username': self.request.user.username})
'main:user_settings',
kwargs={'username': self.request.user.username})
def get_object(self):
# Only get the User record for the user making the request
......
import os
import sys
ROOT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
APPS_DIR = os.path.join(ROOT_DIR, 'allgo')
......@@ -69,8 +70,7 @@ LOCAL_APPS = [
]
if "ALLGO_ADDITIONAL_APPS" in os.environ:
INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS + \
os.environ['ALLGO_ADDITIONAL_APPS'].split(",")
INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS + os.environ['ALLGO_ADDITIONAL_APPS'].split(",")
else:
INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS
......@@ -211,3 +211,32 @@ ACCOUNT_CONFIRM_EMAIL_ON_GET = True
ACCOUNT_ADAPTER = 'allgo.main.adapter.AccountAdapter'
ACCOUNT_PRESERVE_USERNAME_CASING = False # force lowercase on username
ACCOUNT_USERNAME_VALIDATORS = 'allgo.main.adapter.custom_username_validators'
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
'style': '{',
},
'simple': {
'format': '{levelname} {message}',
'style': '{',
},
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'formatter': 'simple'
},
},
'loggers': {
'allgo': {
'handlers': ['console'],
'level': os.getenv('DJANGO_LOG_LEVEL', 'INFO'),
},
},
}
\ No newline at end of file
......@@ -19,9 +19,10 @@
#
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
from django.conf import settings
import allgo
sys.path.insert(0, os.path.abspath('..'))
settings.configure()
# -- General configuration ------------------------------------------------
......@@ -33,9 +34,12 @@ import allgo
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode',
'sphinx.ext.todo',
'sphinx.ext.mathjax']
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.viewcode',
'sphinx.ext.todo',
'sphinx.ext.mathjax'
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
......@@ -59,9 +63,9 @@ author = 'Matthieu Berjon'
# built documents.
#
# The short X.Y version.
version = allgo.__version__
version = '0.1.0'
# The full version, including alpha/beta/rc tags.
release = allgo.__version__
release = '0.1.0'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
......
......@@ -7,7 +7,7 @@ applications. This documentation is related to its front-end. Please see the
:ref:`installation` to get started.
.. toctree::
:maxdepth: 2
:maxdepth: 1
:caption: Contents:
readme
......@@ -19,7 +19,7 @@ Additional Notes
Design notes, legal information and changelog are here for the interested.
.. toctree::
:maxdepth: 2
:maxdepth: 1
changelog
license
......
......@@ -3,81 +3,177 @@
Installation
============
Python Version
--------------
This Django application can be run locally or through a django container. The
application has one set of settings that can be overriden
(:ref:`environment-variable-label`) and satisfies a production type
configuration (:ref:`production-environment-label`) .
We recommend using the latest version of Python 3. Allgo supports Python 3.4
and newer.
Dependencies
------------
Mandatory
^^^^^^^^^
- python 3.4 or newer
- Django 1.11
- Django-allauth 0.35.0
- mysql v15.1 or newer
- python3-mysql
Development
-----------
Optional
^^^^^^^^
+-------------------------+-----------------------------------------------+
| VARIABLE | DEFAULT |
+=========================+===============================================+
| ALLGO_SECRET_KEY | None |
+-------------------------+-----------------------------------------------+
| ALLGO_DEBUG | False |
+-------------------------+-----------------------------------------------+
| ALLGO_ALLOWED_HOSTS | 'localhost' |
+-------------------------+-----------------------------------------------+
| ALLGO_DATABASE_ENGINE | 'django.db.backends.mysql' |
+-------------------------+-----------------------------------------------+
| ALLGO_DATABASE_NAME | 'allgo' |
+-------------------------+-----------------------------------------------+
| ALLGO_DATABASE_USER | 'allgo' |
+-------------------------+-----------------------------------------------+
| ALLGO_DATABASE_PASSWORD | None |
+-------------------------+-----------------------------------------------+
| ALLGO_ADDITIONAL_APPS | None |
+-------------------------+-----------------------------------------------+
| ALLGO_EMAIL_BACKEND | 'django.core.mail.backends.smtp.EmailBackend' |
+-------------------------+-----------------------------------------------+
For development purposes you can install several optional dependencies such as
Sphinx, etc. The complete list is available in the `requirements_dev.txt` and
can be installed through pip.
.. warning::
.. code-block:: bash
pip install -r requirements_dev.txt
Development environment
-----------------------
By default, the `config/settings.py` is setup for a production config and
requires to setup at minimum two environment variables:
- ALLGO_SECRET_KEY
- ALLGO_DATABASE_PASSWORD
These variables must be written into a `.env` file located at the root of the
django docker container (same level as the `manage.py` file. You can override
over variables to alter the behaviour of the application.
For a detailled list of all environment variables, please refer to
:ref:`environment-variable-label`.
In production, you must at least set up a `.env` file containing both the
environment variables ALLGO_DATABASE_PASSWORD and ALLGO_SECRET_KEY.
.. code-block:: bash
# run the application
python manage.py runserver
# The application can be reached at http://localhost:8000
.. todo::
ensure the use of https protocol by generating or using appropriate
certificates. More information at
https://docs.djangoproject.com/fr/1.11/topics/security/
Docker
^^^^^^
When the application is launched within the docker container, it can be reached
at https://localhost/django.
.. warning::
The environement variables can be declared wherever you feel the best and
can be managed at the docker-compose level for example.
Set up of the Mysql database. Ensure that there is an existing user called
allgo with the right privileges on the database allgopy
Database
^^^^^^^^
.. warning::
because we depend at the moment on a legacy database we have a specific
setup for Django. We have chose to recreate the database into Django and by
setup for Django. We have chose to recreate the database into Django and by