Commit 230285f4 authored by Mathieu Giraud's avatar Mathieu Giraud
Browse files

doc/: merge production-management.org into server.org

parent eb717275
* Introduction
When manipulating a production environment it is important to take certain
precautionnary mesures, in order to ensure production can either be rolled
back to a previous version or simply that any encurred loss of data can be
retrieved.
Web2py and Vidjil are no exception to this rule.
* Making backups
Performing an Analysis in Vidjil is time-consuming, therefore should the
data be lost, valuable man-hours are also lost.
In order to prevent this we make regular incremental (?) backups of the
data stored on the vidjil servers.
This not only applies to the fiels uploaded and created by vidjil, but also
to the database.
* Autodelete and Permissions
Web2py has a handy feature called AutoDelete which allows the administrator
to state that file reference deletions should be cascaded if no other
references to the file exist.
When deploying to production one needs to make sure AutoDelete is
deactivated.
As a second precaution it is also wise to temporarily restrict web2py's
access to referenced files.
Taking two mesures to prevent file loss might seem like overkill, but
securing data is more important than the small amount of extra time spent
putting these mesures into place.
* Deploying the server
Currently deploying changes to production is analogous to merging into the
rbx branch and pulling from the server.
Once this has been done, it is important that any database migrations have
been applied.
This can be verified by refreshing the server (calling a controller) and
then looking at the database.
* Step by Step
- Set AutoDelete to False
- Check permissions on the uploads folder (set to 100)
- you can also check the amount of files present at this point for future
reference
- Backup database: Archive old backup.csv and then from admin page: backup
db
- pull rbx (if already merged dev)
- Check the database (for missing data or to ensure mmigrations have been
applied)
- Check files to ensure no files are missing
- Reset the folder permissions on uploads (755 seems to be the minimum
requirement for web2py)
- Run unit tests (Simply a precaution: Continuous Integration renders this
step redundant but it's better to be sure)
- Check site functionnality
#+TITLE: Vidjil -- Server Manual
#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="../css/org-mode.css" />
This is the preliminary help of the upcomping Vidjil server.
This is the preliminary help of the Vidjil server.
This help is intended for server administrators.
Users of the should consult the browser help.
Users should consult the web application manual.
* Requirements
** Hardware
*** Minimal
The minimal requirement for running Vidjil is quite low. For a single run,
a single core processor with 2GB of RAM will suffice (Vidjil Typically uses
approx. 1.2GB of RAM to run on a 1GB fastq and will take approx. 5 minutes+).
Therefore in order to maintain a stable service of the server with a single
worker for running Vidjil, any standard multi-core processor will do.
*** Recommended
When choosing hardware for your server it is important to know the scale
of usage you require.
If you have many users that use the app on a daily basis, you will need to
have multiple cores to ensure the worker queues don't build up.
One worker will occupy one core completely when running Vidjil (Vidjil is
currently single-threaded).
For reference, here is the current Vidjil setup we use on our public
testing server:
- Processor: Quad core Intel 2.4MHz
- RAM: 16GB
This configuration currently serves 15 regular users (~40 total).
Given that the CPU is quad-core, we have 3 workers for executing Vidjil.
This way the webserver can always have a dedicated cpu core and
running the workers at full capacity has no adverse effects on the
web server's response times.
** Storage
The main limitation of Vidjil is the amount of data involved. Files can
sometimes weigh several GB, so a full installation's total storage should
be be serveral hundred GB, or even several TB.
We recommend a RAID setup of at least 2x2TB to allow for user files and at
least one backup.
* Installing and running the vidjil server
......@@ -210,37 +251,62 @@ These installation instruction are for Ubuntu server 14.04
** Restarting web2py
Just touch the file =/etc/uwsgi/web2py.ini=.
* Hardware
** Minimal
The minimal requirement for running Vidjil is quite low. For a single run,
a single core processor with 2GB of RAM will suffice (Vidjil Typically uses
approx. 1.2GB of RAM to run on a 1GB fastq and will take approx. 5 minutes+).
Therefore in order to maintain a stable service of the server with a single
worker for running Vidjil, any standard multi-core processor will do.
** Recommended
When choosing hardware for your server it is important to know the scale
of usage you require.
If you have many users that use the app on a daily basis, you will need to
have multiple cores to ensure the worker queues don't build up.
One worker will occupy one core completely when running Vidjil (Vidjil is
currently single-threaded).
For reference, here is the current Vidjil setup we use on our public
testing server:
- Processor: Quad core Intel 2.4MHz
- RAM: 16GB
This configuration currently serves 15 regular users (~40 total).
Given that the CPU is quad-core, we have 3 workers for executing Vidjil.
This way the webserver can always have a dedicated cpu core and
running the workers at full capacity has no adverse effects on the
web server's response times.
* Storage
The main limitation of Vidjil is the amount of data involved. Files can
sometimes weigh several GB, so a full installation's total storage should
be be serveral hundred GB, or even several TB.
We recommend a RAID setup of at least 2x2TB to allow for user files and at
least one backup.
* Running the server in a production environment
** Introduction
When manipulating a production environment it is important to take certain
precautionnary mesures, in order to ensure production can either be rolled
back to a previous version or simply that any encurred loss of data can be
retrieved.
Web2py and Vidjil are no exception to this rule.
** Making backups
Performing an Analysis in Vidjil is time-consuming, therefore should the
data be lost, valuable man-hours are also lost.
In order to prevent this we make regular incremental (?) backups of the
data stored on the vidjil servers.
This not only applies to the fiels uploaded and created by vidjil, but also
to the database.
** Autodelete and Permissions
Web2py has a handy feature called AutoDelete which allows the administrator
to state that file reference deletions should be cascaded if no other
references to the file exist.
When deploying to production one needs to make sure AutoDelete is
deactivated.
As a second precaution it is also wise to temporarily restrict web2py's
access to referenced files.
Taking two mesures to prevent file loss might seem like overkill, but
securing data is more important than the small amount of extra time spent
putting these mesures into place.
** Deploying the server
Currently deploying changes to production is analogous to merging into the
rbx branch and pulling from the server.
Once this has been done, it is important that any database migrations have
been applied.
This can be verified by refreshing the server (calling a controller) and
then looking at the database.
** Step by Step
- Set AutoDelete to False
- Check permissions on the uploads folder (set to 100)
- you can also check the amount of files present at this point for future
reference
- Backup database: Archive old backup.csv and then from admin page: backup
db
- pull rbx (if already merged dev)
- Check the database (for missing data or to ensure mmigrations have been
applied)
- Check files to ensure no files are missing
- Reset the folder permissions on uploads (755 seems to be the minimum
requirement for web2py)
- Run unit tests (Simply a precaution: Continuous Integration renders this
step redundant but it's better to be sure)
- Check site functionnality
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment