Commit 73de1af9 authored by Mathieu Giraud's avatar Mathieu Giraud

doc/: merge several files into dev.org

Some information was scattered in the previous organization.
This file is now the central place to store *developer* information.
We may one day re-split this documentation into several files.
parent 230285f4
#+TITLE: Vidjil Browser -- development documentation
#+AUTHOR: The Vidjil team (Marc, Mathieu, Mikaël and Tatiana)
** Architecture
The Vidjil browser is a set of /views/ linked to a same /model/.
The model keeps the views in sync on some global properties,
most notably dealing with the selection of clones, with the clone filtering,
as well with the locus selection.
- The model (=js/model.js=) is the main object of the Vidjil browser.
It loads and saves =.vidjil= json data (either directly from data, or from a local file, or from some url).
It provides function to access and edit information on the clones and on the global parameters
It keeps all the views in sync.
- Each of the views (=Graph=, =ScatterPlot=, =List=, =Segment=) is rendered inside one or several =<div>= elements,
and kept sync with the model. All the views are optional, and several views of the same type can be added.
See =js/main.js= for the invovcation
- The link with the patient database/server is done with the =Database= object (=js/database.js=)
- Other objects: =Report=, =Shortcut=
Extends functionalities but requires elements from the full =index.html=.
** Integrating the browser
*** HTML and CSS
- The =index.html= contains the =<div>= for all views and the menus
- The CSS (=css/light.css=) is generated by =less= from =css/vidjil.less=
- The =small_example.html= is a minimal example embedding basic HTML, CSS, as well as some data.
As the menus are not embedded in this file, functionalities should be provided by direct calls to the models and the views.
*** Javascript
- The wonderful library =require.js= is used, so there is only one file to include
<script data-main="js/app.js" src="js/lib/require.js"></script>
- =js/main.js= creates the different views and binds them to the model.
Another option is to directly define a function named =main()=, as in =small_example.html=.
*** JSON .vidjil data
Clone lists can be passed to the model through several ways:
- directly by the user (import/export)
- from a patient database (needs a database)
- trough the API (see below)
- or by directly providing data through Javascript (as in =small_example.html=)
The first three solutions needs some further elements from the full =index.html=.
**** Browser API
The browser can be opened on a data file specified from a =data= attribute,
and optionally on an analysis file specified from a =analysis= attribute,
as in the following URLs on our test server:
- http://rbx.vidjil.org/browser/?data=test.vidjil
- http://rbx.vidjil.org/browser/?data=test.vidjil&analysis=test.analysis
- http://rbx.vidjil.org/browser/?data=http://rbx.vidjil.org/browser/test.vidjil
Both GET and POST requests are accepted.
Note that the =browser/index.html= file and the =.vidjil/.analysis= files should be hosted on the same server.
Otherwise, the server hosting the =.vidjil/.analysis= files must accept cross-domain queries.
Here is a procedure for installing watir and launching the browser tests
* Install rvm
#+BEGIN_SRC sh
\curl -sSL https://get.rvm.io | bash
#+END_SRC
Afterwards you may need to launch:
#+BEGIN_SRC sh
source /etc/profile.d/rvm.sh
#+END_SRC
* Install ruby 2.1.1
#+BEGIN_SRC sh
rvm install 2.1.1
#+END_SRC
* Switch to ruby 2.1.1
#+BEGIN_SRC sh
rvm use 2.1.1
#+END_SRC
* Install necessary gems
#+BEGIN_SRC sh
gem install minitest
gem install minitest-ci
gem install watir-webdriver
gem install test-unit
#+END_SRC
* Launch browser tests
#+BEGIN_SRC sh
make functional
#+END_SRC
** Headless mode
On servers without a X server the browser tests can be launched in headless
mode.
For this sake one needs to install a few more dependencies:
#+BEGIN_SRC sh
gem install headless
#+END_SRC
The virtual framebuffer X server (=xvfb=) must also be installed. Depending
on the operating system the command will be different:
#+BEGIN_SRC sh
# On Debian/Ubuntu
apt-get install xvfb
# On Fedora/CentOS
yum install xvfb
#+END_SRC
Then the browser tests can be launched in headless mode with:
#+BEGIN_SRC sh
make headless
#+END_SRC
* Database schema
** Patients and samples
A patient has several samples. A sample is an uploaded .fasta/.fastq file.
** Launching and results
A software/pipeline (as for example Vidjil or Brno pipeline)
process .fasta/.fastq file and produce a "data file".
The options of the software/pipeline are stored in a "config".
** "Fusing" results on different samples
The result "data files" of a same patient (and processed within a same "config")
are "fused" to a unique "fused file".
** Analysis
All changes done by the user (on a specific "fused file") are recorded
into a ".analysis" file and stored into the database.
** Authentification
There is a mecanism to authenticate users/groups and to grant them some accesses on the patients:
- read
- add samples (to an existing patient, uploading .fasta/.fastq files)
- add patients
- save an analysis (with some clones merged, labeled or renamed)
- delete samples and patients
#+TITLE: Vidjil -- Developer Documentation
#+AUTHOR: The Vidjil team (Florian, Marc, Mathieu, Mikaël, Ryan and Tatiana)
#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="../css/org-mode.css" />
# This manual can be browsed online:
# http://www.vidjil.org/doc/dev.html (last stable release)
# http://git.vidjil.org/blob/master/doc/dev.org (development version)
# Vidjil -- High-throughput Analysis of V(D)J Immune Repertoire -- [[http://www.vidjil.org]]
# Copyright (C) 2011, 2012, 2013, 2014, 2015, 2016 by Bonsai bioinformatics
# at CRIStAL (UMR CNRS 9189, Université Lille) and Inria Lille
# contact@vidjil.org
This is the main developer documentation.
It is not as stable as the user documentation.
* Algorithm
** Code organisation
The algorithm follows roughly those steps:
......@@ -45,6 +61,73 @@ able to open =.vidjil= files with the =import/export= menu.
To work with actual data, the easiest way is to copy =js/conf.js.sample= to =js/conf.js=.
This will unlock the =patients= menu and allow your local browser
to access the public server at http://rbx.vidjil.org/.
** Browser API
The browser can be opened on a data file specified from a =data= attribute,
and optionally on an analysis file specified from a =analysis= attribute,
as in the following URLs on our test server:
- http://rbx.vidjil.org/browser/?data=test.vidjil
- http://rbx.vidjil.org/browser/?data=test.vidjil&analysis=test.analysis
- http://rbx.vidjil.org/browser/?data=http://rbx.vidjil.org/browser/test.vidjil
Both GET and POST requests are accepted.
Note that the =browser/index.html= file and the =.vidjil/.analysis= files should be hosted on the same server.
Otherwise, the server hosting the =.vidjil/.analysis= files must accept cross-domain queries.
** Architecture
The Vidjil browser is a set of /views/ linked to a same /model/.
The model keeps the views in sync on some global properties,
most notably dealing with the selection of clones, with the clone filtering,
as well with the locus selection.
- The model (=js/model.js=) is the main object of the Vidjil browser.
It loads and saves =.vidjil= json data (either directly from data, or from a local file, or from some url).
It provides function to access and edit information on the clones and on the global parameters
It keeps all the views in sync.
- Each of the views (=Graph=, =ScatterPlot=, =List=, =Segment=) is rendered inside one or several =<div>= elements,
and kept sync with the model. All the views are optional, and several views of the same type can be added.
See =js/main.js= for the invovcation
- The link with the patient database/server is done with the =Database= object (=js/database.js=)
- Other objects: =Report=, =Shortcut=
Extends functionalities but requires elements from the full =index.html=.
** Integrating the browser
*** HTML and CSS
- The =index.html= contains the =<div>= for all views and the menus
- The CSS (=css/light.css=) is generated by =less= from =css/vidjil.less=
- The =small_example.html= is a minimal example embedding basic HTML, CSS, as well as some data.
As the menus are not embedded in this file, functionalities should be provided by direct calls to the models and the views.
*** Javascript
- The wonderful library =require.js= is used, so there is only one file to include
<script data-main="js/app.js" src="js/lib/require.js"></script>
- =js/main.js= creates the different views and binds them to the model.
Another option is to directly define a function named =main()=, as in =small_example.html=.
*** JSON .vidjil data
Clone lists can be passed to the model through several ways:
- directly by the user (import/export)
- from a patient database (needs a database)
- trough the API (see below)
- or by directly providing data through Javascript (as in =small_example.html=)
The first three solutions needs some further elements from the full =index.html=.
** Notifications
*** Priority
#<<browser:priority>>
......@@ -91,8 +174,25 @@ to access the public server at http://rbx.vidjil.org/.
fields under the =seg= field are displayed as soon as they have a =start= and
=stop=. Some of them can be explicitly not displayed by filling the
=exclude_seg_info= array in =getHtmlInfo=.
* Server
** Notifications
The news system is a means of propagating messages to the users of a vidjil server installation.
Messages are propagated in near-realtime for users interacting directly with the server and at a slightly slower rate for users simply using the browser but for which the server is configured.
*** Message Retrieval
The browser by default periodically queries the server to retrieve any new messages and are displayed on a per user basis. This means that any message having already been viewed by the user is not displayed in the browser.
Older messages can be viewed from the index of news items.
*** Caching
News items are kept in cache in order to relieve the database from a potentially large amount of queries.
The cache is stored for each user and is updated only when a change occurs (message read, message created or message edited).
*** Formatting
Messages can be formatted by using the Markdown syntax. Syntax details are
available here: http://commonmark.org/help/
*** Priority
The priority determines how the notification is shown (see [[browser:priority][here for more
details]]). From the server we have two ways of modifiying the priority.
......@@ -168,12 +268,10 @@ to access the public server at http://rbx.vidjil.org/.
reasons. Under Chromium/Chrome this should work fine by just opening the
webpage.
*** Functional
All the browser functional testing is done in the directory
=browser/tests=.
Several stuff must be installed to launch the functional browser
tests. Please see the doc in browser-tests.org.
**** Architecture
The browser functional testing is done in the directory
=browser/tests=, with Watir.
The functional tests are built using two base files:
- vidjil_browser.rb :: abstracts the vidjil browser (avoid using IDs or
class names that could change in the test). The tests must rely as
......@@ -186,3 +284,120 @@ to access the public server at http://rbx.vidjil.org/.
tests are launched by the script in =launch_functional_tests= which launch
all the files matching the previous pattern. It also backs up the test
reports as =ci_reporter= removes them before each file is run.
**** Installation
***** Install rvm
#+BEGIN_SRC sh
\curl -sSL https://get.rvm.io | bash
#+END_SRC
Afterwards you may need to launch:
#+BEGIN_SRC sh
source /etc/profile.d/rvm.sh
#+END_SRC
***** Install ruby 2.1.1
#+BEGIN_SRC sh
rvm install 2.1.1
#+END_SRC
***** Switch to ruby 2.1.1
#+BEGIN_SRC sh
rvm use 2.1.1
#+END_SRC
***** Install necessary gems
#+BEGIN_SRC sh
gem install minitest
gem install minitest-ci
gem install watir-webdriver
gem install test-unit
#+END_SRC
**** Launch browser tests
#+BEGIN_SRC sh
make functional
#+END_SRC
**** Headless mode
On servers without a X server the browser tests can be launched in headless
mode.
For this sake one needs to install a few more dependencies:
#+BEGIN_SRC sh
gem install headless
#+END_SRC
The virtual framebuffer X server (=xvfb=) must also be installed. Depending
on the operating system the command will be different:
#+BEGIN_SRC sh
# On Debian/Ubuntu
apt-get install xvfb
# On Fedora/CentOS
yum install xvfb
#+END_SRC
Then the browser tests can be launched in headless mode with:
#+BEGIN_SRC sh
make headless
#+END_SRC
* Packaging
** Packaging Vidjil into a Debian Package
*** Requirements
- The release version of Vidjil you wish to package
- Knowledge of Debian packaging
In this documentation we will not go over all the specifics of creating a
debian package. You can find the required information here:
https://wiki.debian.org/HowToPackageForDebian
and https://wiki.debian.org/Packaging/Intro?action=show&redirect=IntroDebianPackaging
*** Creating the orig archive
Debian packages use an orig archive to map a changelog. In order to create
this archive for vidjil you must:
Extract the release archive you wish to package:
$ tar -xf vidjil-<version>.tgz
$ cd vidjil-<version>/
Download the necessary germlines and data files:
$ mkdir browser # if it doesn't exist
$ make germline
$ make data
Archive the folder as follows:
Note that debian uses a different version annotation than vidjil, so it is
likely that you will need to determine the correct annotation to use.
$ cd ..
$ tar -cvzf vidjil_<debian-version-annotation>.orig.tar.gz
Make a standard compliant debian folder for the package (see documentation
in Requirements section) and add the 'missing-sources' file (this is a
workaround to the issue that Lintian cannot find the sources for some of the
binaries included) with the following content:
$ /browser/cgi/align.cgi
$ /browser/cgi/similarity.cgi
$ /algo/tools/similarity
Commit the source changes (for some reason the dep.mk files need to be
committed):
$ dpkg-source --commit
Build the package just like any debian package:
$ debuild -us -uc
You're done! You can now install the debian package with:
$ sudo ``dpkg -i path/to/package
* News
The news system is a means of propagating messages to the users of a vidjil server installation.
Messages are propagated in near-realtime for users interacting directly with the server and at a slightly slower rate for users simply using the browser but for which the server is configured.
** Message Retrieval
The browser by default periodically queries the server to retrieve any new messages and are displayed on a per user basis. This means that any message having already been viewed by the user is not displayed in the browser.
Older messages can be viewed from the index of news items.
** Caching
News items are kept in cache in order to relieve the database from a potentially large amount of queries.
The cache is stored for each user and is updated only when a change occurs (message read, message created or message edited).
** Formatting
Messages can be formatted by using the Markdown syntax. Syntax details are
available here: http://commonmark.org/help/
*Packaging Vidjil into a Debian Package
**Requirements
- The release version of Vidjil you wish to package
- Knowledge of debian packaging
In this documentation we will not go over all the specifics of creating a
debian package. You can find the required information here:
https://wiki.debian.org/HowToPackageForDebian
and https://wiki.debian.org/Packaging/Intro?action=show&redirect=IntroDebianPackaging
** Creating the orig archive
Debian packages use an orig archive to map a changelog. In order to create
this archive for vidjil you must:
Extract the release archive you wish to package:
$ tar -xf vidjil-<version>.tgz
$ cd vidjil-<version>/
Download the necessary germlines and data files:
$ mkdir browser # if it doesn't exist
$ make germline
$ make data
Archive the folder as follows:
Note that debian uses a different version annotation than vidjil, so it is
likely that you will need to determine the correct annotation to use.
$ cd ..
$ tar -cvzf vidjil_<debian-version-annotation>.orig.tar.gz
Make a standard compliant debian folder for the package (see documentation
in Requirements section) and add the 'missing-sources' file (this is a
workaround to the issue that Lintian cannot find the sources for some of the
binaries included) with the following content:
$ /browser/cgi/align.cgi
$ /browser/cgi/similarity.cgi
$ /algo/tools/similarity
Commit the source changes (for some reason the dep.mk files need to be
committed):
$ dpkg-source --commit
Build the package just like any debian package:
$ debuild -us -uc
You're done! You can now install the debian package with:
$ sudo ``dpkg -i path/to/package
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment