Commit 5775c7bd authored by LETORT Sebastien's avatar LETORT Sebastien

Merge branch 'purge_rails' into 'django'

Purge rails

See merge request !173
parents 8877a9c4 e39d5843
Pipeline #77944 passed with stages
in 7 minutes and 16 seconds
......@@ -4,13 +4,14 @@ ALLGO containers
Overview
--------
A minimal deployment of allgo consists of 4 docker images:
A minimal deployment of allgo consists of 6 docker images:
- **allgo/rails**: the rails application server
- **allgo/mysql**: the mysql database server
- **allgo/redis** : the redis application server
- **allgo/django** : the django application server
- **allgo/mysql** : the mysql database server
- **allgo/controller**: the manager for user docker containers
- **allgo/ssh**: the ssh frontend (giving access to the sandboxes)
- **allgo/toolbox**: an image containing a set of commands (scp, vi, nano,
- **allgo/ssh** : the ssh frontend (giving access to the sandboxes)
- **allgo/toolbox** : an image containing a set of commands (scp, vi, nano,
less, ...) to be mounted in the user sandboxes
These images may be deployed multiple times to implement multiple independent
......@@ -27,7 +28,7 @@ There is an extra image used only in development:
- **allgo/smtpsink**: a SMTP server that catches and stores all incoming messages into a single mailbox
Each environment has its own docker network. The nginx container is connected
to all these networks to that it can connect to the rails servers.
to all these networks.
Conventions
......@@ -37,7 +38,7 @@ All docker images use the following conventions.
### External volumes
They data is stored in:
Their data are stored in:
- `/vol/rw` for persistent data
- `/vol/ro` for persistent data in read-only access
......@@ -77,14 +78,13 @@ It provides 8 containers:
All external volumes are stored in `/data/dev/` (the path is absolute because
it is tricky to use a relative path with the allgo/docker image).
For convenience, all containers not running as root (rails, mysql, registry)
For convenience, all containers not running as root (django, mysql, registry)
have their user overridden to the UID:GID of the developer running
docker-compose. This is managed with the `DOCKERUSER` environment variable set
[in the `.env`
file](https://docs.docker.com/compose/environment-variables/#the-env-file) by
[in the `.env` file](https://docs.docker.com/compose/environment-variables/#the-env-file) by
`prepare.sh`.
For convenience (again), there is an extra external volumes for `dev-rails`,
For convenience (again), there is an extra external volume for `dev-django`,
`dev-controller` and `dev-ssh` so that the source directory of the app is mounted
inside `/opt/` (in fact it overrides the actual application files provided by
the docker image). The purpose is to avoid rebuilding a new docker image for
......@@ -93,16 +93,15 @@ each development iteration.
### Getting started
The sources are located in two repositories:
The sources are located in one repository:
- *rails-allgo*: the rails application repository
- *allgo*: the deployment repository
To set up the development environment, run:
1. get the sources
<pre>
<pre>
git clone git@gitlab.inria.fr:allgo/allgo.git
cd allgo
</pre>
......@@ -110,19 +109,19 @@ To set up the development environment, run:
2. *(as root)* create `/data/dev` and make it owned by the developer
<pre>
sudo mkdir -p /data/dev
sudo chown USER: /data/dev
sudo chown $USER: /data/dev
</pre>
3. bootstrap the environment
<pre>
<pre>
./bootstrap
</pre>
This command will run the `/dk/init_container` in every container that
needs it, then start the container.
The first run takes a very long time because all images are built from
scratch (especially the rails image which builds ruby source).
You have enough time for a coffee break.
The first run takes a few minutes because all images are built from
scratch.
You may have enough time for a short coffee break.
**Note** by default `bootstrap` works on all containers. It is possible
to give an explicit list of containers instead. Example:
......@@ -164,34 +163,34 @@ The official doc for docker-compose is available at: [https://docs.docker.com/co
</pre>
- hard cleanup (remove images too)
<pre>
<pre>
fig down --rmi local
</pre>
- restart a container
<pre>
fig restart dev-rails
fig restart dev-django
</pre>
- restart a container using a new docker image (if the image has been rebuilt since the last start)
<pre>
fig up dev-rails
<pre>
fig up dev-django
</pre>
- rebuild an image
<pre>
fig build dev-railf
<pre>
fig build dev-django
</pre>
- **Note:** most commands work on every container by default (eg: up down
start stop restart ...) they can be use on an individual container too:
<pre>
fig restart dev-controller dev-rails
<pre>
fig restart dev-controller dev-django
</pre>
- run a container with an arbitrary command (eg: to have access to the rails console)
<pre>
fig run --rm dev-rails bash
- run a container with an arbitrary command (eg: to have access to the django console)
<pre>
fig run --rm dev-django bash
</pre>
**Note:** containers created by `fig run` have the same parameters as
......@@ -199,10 +198,10 @@ The official doc for docker-compose is available at: [https://docs.docker.com/co
*allgo_dev-ssh_run_1*), which means that this container is not
reachable by the others (this may be an issue for example if you want
to run the mysqld server manually: `fig run dev-mysql mysqld` -> this
container won't be reachable by the ssh and rails containers)
container won't be reachable by the ssh and django containers)
- follow the output of all containers:
<pre>
<pre>
fig logs --tail=1 --follow
</pre>
......@@ -242,7 +241,7 @@ it as root**, otherwise it will be owned by root and you may have errors like:
If somehow you skipped this step, you can reset the ownership to the current user:
sudo chown USER: /data/dev
sudo chown -R USER: /data/dev/{registry,mysql,rails}
sudo chown -R USER: /data/dev/{registry,mysql,django}
If you are completely lost, you can just restart the initialisation from scratch:
......@@ -282,22 +281,21 @@ Hosts a mysql server listening on port 3306 with two databases: `allgo` and
- `ssh` has read only access to `allgo`
## rails
Hosts four daemons for running allgo:
## django
- the unicorn server (runnning the rails application)
- the sidekiq queue manager
- the redis db server
- a nginx frontend for buffering the HTTP requests/responses
Hosts three daemons for running the allgo web server:
- a nginx frontend for buffering the HTTP requests/responses and routing them
to the other daemons. It also serves static files directly
- the gunicorn server (running the django application)
- the allgo.aio server (serving the asynchronous requests)
This container is managed with supervisor, the `supervisorctl` command allows
starting/stopping the daemons individually.
### Running the rails server manually
### Running the django server manually
TODO ?
- run the `dev-rails` container and open a shell:
[comment]: # ( - run the `dev-rails` container and open a shell:
<pre>
fig up -d
docker exec -t -i dev-rails bash
......@@ -308,7 +306,7 @@ starting/stopping the daemons individually.
supervisorctl stop rails
rails server
</pre>
)
## ssh
......@@ -324,7 +322,7 @@ WEBAPP@sid.allgo.irisa.fr`). Each allgo webapp is mapped to a system user
gid = 65534 (nogroup)
gecos = webapps.name
shell = /bin/allgo-shell
</pre>
</pre>
- The ssh server is configured to accept key-based authentication only. The
list of public keys is obtained from the (using an AuthorizedKeysCommand).
......@@ -333,12 +331,12 @@ WEBAPP@sid.allgo.irisa.fr`). Each allgo webapp is mapped to a system user
- The connection to the sandbox is made though a unix socket and a set of pipes
in the filesystem.
## docker
## controller
Hosts the *docker-allgo-proxy* which manages all docker operations (run, stop,
rm, commit, pull, push, ...) on behalf of the rails container.
Hosts the *docker-controller* which manages all docker operations (run, stop,
rm, commit, pull, push, ...) on behalf of the django container.
Technically speaking this container had root privileges since it has access to
Technically speaking this container has root privileges since it has access to
the docker socket.
The proxy script enforces restrictions (according to the current environment: eg prod/qualif/dev) on:
......@@ -363,3 +361,5 @@ mailbox.
The mailbox is accessible with IMAP as user *sink* (password *sink*).
NOTE: in the development environment, django's default is to dump outgoing
e-mails to the console. Thus this container is only useful in the qualif setup.
......@@ -79,9 +79,6 @@ Database
integrating the database constraints (that are managed by Ruby on Rails and
not the SGDB).
It has been decided to use the same database for both rails and django but with
a different naming.
At the moment the django docker container take care of the migration by calling
the migration script. The migration process consist of two files located int
`tools` folder:
......@@ -129,7 +126,7 @@ The different configuration file for the docker file such as the nginx
configuration in the `setup/dk` directory. This includes:
- `allgo.conf`: nginx configuration for the django docker
- `container_init`: initialisation of the container (imports the rails database)
- `container_init`: initialisation of the container
- `nginx.patch`: main nginx configuration
- `run-allgo`: bash script creating the necessary directories and running the
different services necessary for the application
......
......@@ -120,24 +120,6 @@ services:
networks: [dev, sandboxes]
# RAILS
######################################################################################################################
dev-rails:
container_name: dev-rails
build: rails
user: "$DOCKERUSER"
ports:
- "127.0.0.1:3000:8080"
volumes:
- "/data/dev/rails:/vol"
- "./rails:/opt/allgo"
environment:
RAILS_ENV: development
networks: [dev]
tty: true
stdin_open: true
# SMTP
######################################################################################################################
......
{
"directory": "/opt/bower_components"
}
Dockerfile*
.git
.*.swo
.*.swp
.DS_Store
.vagrant
*.rbc
*.sassc
.sass-cache
capybara-*.html
.rspec
.rvmrc
.bundle
vendor/bundle
log
log/*
tmp/
tmp/*
db/*.sqlite3
public/system/*
coverage/
spec/tmp/*
**.orig
rerun.txt
pickle-email-*.html
.project
dump.rdb
app/assets/dockers
app/assets/dockers/*
app/assets/stylesheets/theme/
public/app/*
db/backup/*
public/datastore
.settings/
.vagrant/
deploy/.vagrant/
deploy/atom
deploy/debian-jessie
.keep
*.keep
doing.txt
active_admin.rb.old
vendor/assets/components/*
deploy/nginxconf
public/assets
.*.sw[po]
FROM allgo/base-debian
# configure the node reporisory
# http://linuxbsdos.com/2017/06/26/how-to-install-node-js-lts-on-debian-9-stretch/
RUN apt-getq install curl gnupg ca-certificates &&\
curl -sL https://deb.nodesource.com/setup_8.x | bash -
# install system packages + bower
RUN apt-getq install mariadb-client libmariadb-client-lgpl-dev-compat \
redis-server curl imagemagick git ca-certificates \
gcc g++ make libc6-dev file libffi-dev libgdbm-dev libgmp-dev \
libncurses5-dev libncursesw5-dev libreadline6-dev libssl-dev \
libyaml-dev openssl procps systemtap-sdt-dev zlib1g-dev \
nodejs npm supervisor nginx-light zip \
&& ln -s /usr/bin/nodejs /usr/local/bin/node \
&& npm install -g bower
# build & install ruby
ENV PATH="/opt/ruby/bin:$PATH"
COPY deploy/docker/ruby /tmp/ruby
RUN apply-patches /tmp/ruby/*.diff &&\
useradd -m -s /bin/bash allgo &&\
chown allgo: /opt &&\
su allgo -c 'sh /tmp/ruby/install.sh'
# install ruby gems
COPY Gemfile* /tmp/
RUN cd /tmp && chown allgo: /tmp/Gemfile* && su allgo -c "bundle install"
# install bower modules under /opt/bower_components
# (so that they are located in the docker image, even when /opt/allgo is mounted from an external volume)
COPY bower.json .bowerrc /tmp/
RUN mkdir /opt/bower_components &&\
chown allgo: /opt/bower_components &&\
cd /tmp && su allgo -c "bower install"
# install allgo sources + run setup script
COPY . /opt/allgo
RUN sh /opt/allgo/deploy/docker/setup/setup.sh
USER allgo
WORKDIR /opt/allgo
CMD ["run-allgo"]
LABEL dk.migrate_always=1
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
_
_ | | _____ ____
/\ | | | | / ____|/ __ \
/ \ | | | | | | __| | | |
/ /\ \ | | | | | | |_ | | | |
/ ____ \| |____ | | | |__| | |__| |
/_/ \_\______| | | \_____|\____/
|_|
#What is Allgo? (A||Go)
A || Go is a web infrastructure to put online quickly and easily possible software or scientific prototypes. The objective is to reduce the cost of transferring
the computer researcher to end users (team member, community
Scientific Startup SMEs, the general public).
Application deployment, and especially their uses are found in simplified. One can execute via the web interface or an HTTP API.
Technically, A||Go is based on the Ruby on Rails framework to provide the web interface and HTTP services REST. Use of Docker for managing application images.
Docker provides Linux containers to manage / store / apps we use host.
A job is the action to run an application, with one file or more, which will return a result.
---
If you want to contribute:
You can fork the project, create a branch and make a pull-request.
You can refeer to the gitlab issues.
Our main need is someone who know ruby on rails testing.
To deploy the test environment, you will need Vagrant.
Go in /deploy folder and run vagrant up, you don't need any particular software on your computer.
For more information on the infrastructure, you can refeer to the /doc folder.
# Add your own tasks in files placed in lib/tasks ending in .rake,
# for example lib/tasks/capistrano.rake, and they will automatically be available to Rake.
require File.expand_path('../config/application', __FILE__)
Allgo::Application.load_tasks
// JS and CSS bundles, creates for sass 6 and polymerjs
//
//= link_directory ../javascripts .js
//= link_directory ../stylesheets .css
// link_directory ../../../vendor/assets/stylesheets .css
// link_directory ../../../vendor/assets/javascripts .js
// Images and fonts so that views can link to them
//
//= link_tree ../fonts
//= link_tree ../images
// This is a manifest file that'll be compiled into application.js, which will include all the files
// listed below.
//
// Any JavaScript/Coffee file within this directory, lib/assets/javascripts, vendor/assets/javascripts,
// or vendor/assets/javascripts of plugins, if any, can be referenced here using a relative path.
//
// It's not advisable to add code directly here, but if you do, it'll appear at the bottom of the
// compiled file.
//
// Read Sprockets README (https://github.com/sstephenson/sprockets#sprockets-directives) for details
// about supported directives.
//
//= require jquery
//= require jquery-ujs
//= require bootstrap
// require jquery-ui/widget
//= require jquery-fileupload/basic
//= require jquery-color/jquery.color
//= require d3
//= require nvd3
//= require turbolinks
//= require_tree
// NOTE rails.validations must be included *after* turbolinks
//= require rails.validations
// initialise the tooltips
$(function () {
$('[data-toggle="tooltip"]').tooltip()
})
# Action Cable provides the framework to deal with WebSockets in Rails.
# You can generate new channels where WebSocket features live using the rails generate channel command.
#
# Turn on the cable connection by removing the comments after the require statements (and ensure it's also on in config/routes.rb).
#
#= require action_cable
#= require_self
#= require_tree ./channels
#
# @App ||= {}
# App.cable = ActionCable.createConsumer()
// Place all the behaviors and hooks related to the matching controller here.
// All this logic will automatically be available in application.js.
# Place all the behaviors and hooks related to the matching controller here.
# All this logic will automatically be available in application.js.
# You can use CoffeeScript in this file: http://coffeescript.org/
'use strict';
var job; // Store the job's id once created.
/*
Fired when user clicks on "Remove".
It calls the jobs' controller aynschronously to remove the file and update the table.
*/
function destroyFile(filename) {
$.get( '../job/' + job + '/destroy_file', {'filename': filename} ).done(function( data ) {
$.getJSON( '../job/' + job + '/attached_files', function(response) {
displayAttachedFiles(response.files);
});
});
}
/*
After the user's added a file, update the table showing uploaded files.
*/
function displayAttachedFiles(files) {
if (files.length > 0) {
var text = '<table class=\"table\"><thead><tr><th>Files uploaded</th><th></th></tr></thead>';
for (var i in files) {
text += '<tr><td>' + files[i] + '</td><td><a class="remove-file" id="' + files[i] + '" style="cursor: pointer;">Remove</a></td></tr>';
}
text += '</table>';
$('#filesUploaded').html(text);
$('.remove-file').on('click', function() {
destroyFile(this.id);
});
} else {
$('#filesUploaded').html('');
}
$('#progress .progress-bar').remove(); // We cannot set the width to 0 because of the animations thus we remove and create a new bar.
$('#progress').append('<div class=\"progress-bar progress-bar-success\"></div>');
}
/*
We're using FileUpload to upload files to the server. We're using AJAX to create job before sending a file.
*/
function enableOneUpload(webapp_id) {
$('#fileupload').fileupload({
//maxChunkSize: 20000000, //20MB
//maxFileSize: 30000000 * 1000, //3Go
url: '/upload',
dataType: 'json',
add: function (e, data) {
//console.log("this file is :"+data.files[0].size);
if (data.files[0].size > 1000000) //show it will take time
{
//$('#progress .progress-bar').next.text("it will take time");
}
if (job == undefined) { // We don't have a job yet, we create one.
$.getJSON( '../job/inside_creation', {'webapp_id': webapp_id}, function(response) {
job = response.job_id;
$('#job_id').val(job);
data.formData = {'id': job};
data.submit();
});
} else {