Matlab and SPM 12 stability

Getting started

Requirements

1. You have read or are reading the paper XXXXXX

2. Minimal knowledge of git, Python, Jupyter, Docker and Linux command line

3. You will need python3, pip and jupyter installed on your platform

N.B We strongly suggest the use of virtualenv

4. Clone this repository in your workspace.

5. Go and do what's asked on https://github.com/Inria-Empenn/narps_open_pipelines Make sure to have this repository alongside this very repository, not within it.

   N.B We advise to get the whole dataset if possible. Beware, you will need at least 2TB of storage. Also note that running experiments on a HDD will lead to a huge bottlenecks as Nipype processes are very IO intensive. A single run can take nearly 24h on a HDD vs 3-4 hours on a SSD. Also bare in mind that pipelines can reduce the lifespan of SSDs due to the very intensive IO activity (200-400GB per run).

6. If not done yet, open a terminal, cd to your workspace and start jupyter :

jupyter lab

It should open your browser and display Jupyter notebook.

Experiment process

1. Docker_gen.ipynb generates configurable Dockerfiles. Make sure you run Jupyter with a user that has the rights to run Docker. If everything runs smoothly, a docker image will be built and the notebook will display the currently available images after the cell containing client.images.list() Otherwise use the Dockerfile that the notebook generates to build an image from command line.

2. Run a pipeline using the following command line in your terminal

   for i in {1..N}; do docker run -it -v [WORKSPACE]/narps_open_pipelines/:/work/ -v [RUNS_DIR]:/home/run/reproduced/ [DOCKER IMAGE] /neurodocker/job.sh [PL] [NB_SUB]; mv [RUNS_DIR]/NARPS-[PL]-reproduced [RUNS_DIR]/NARPS-[PL]-reproduced_MLAB_VERSION?_[NB_SUB]_run$i; done

   Where N is the number of time you will run a single experiment (batch size), [NB_SUB] the number of subjects to consider in the experiment and [PL] a SPM based pipeline ID (2T6S, U26C, ...) c.f

Example for Matlab variant

for i in {1..4}; do docker run -it -v /extsd/RIPOST/NARPS/narps_open_pipelines/:/work/ -v /extsd/RIPOST/NARPS/narps_open_pipelines/run/reproduced/:/home/run/reproduced/ diverse/matlab_r2018a /neurodocker/job.sh U26C 100; mv run/reproduced/NARPS-U26C-reproduced run/reproduced/NARPS-U26C-reproduced_r2018a_100_run$i; done  

Example for SPM12 variant

for i in {1..4}; do docker run -it -v /extsd/RIPOST/NARPS/narps_open_pipelines/:/work/ -v /extsd/RIPOST/NARPS/narps_open_pipelines/run/reproduced/:/home/run/reproduced/ diverse/narps_spm_r7487 /neurodocker/job.sh U26C 100; mv run/reproduced/NARPS-U26C-reproduced run/reproduced/NARPS-U26C-reproduced_2010a_r7487_100_run$i; don

To avoid modifying the folders names parsing, we highly suggest to use this scheme of naming : NARPS-TEAMID-reproduced_MATLAB_VERSION_NBSUJECTS_runX for matlab variations and NARPS-TEAMID-reproduced_MATLAB_VERSION_SPMRELEASE_NBSUJECTS_runX for SPM 12 releases variations.

3. Once done, you can analyze the batch data with Analyze_batch_single_team.ipynb This notebook computes similarities of all images of batches (Hypotheses files + other files), pairwise. It uses Dice-Sorrenson algorithm

4. Repeat from step 1 with other configurations, let it be SPM12, matlab or even hardware variation.

   N.B: ARM hardware is not available through docker as long as it is only available on Apple Sillicon with matlab 2023b and spm12 r7771

5. Use Analyze_2_runs_single_team.ipynb to compare results between two runs of the same team based on different configurations (resp spm12, matlab, hw)

6. Use Comparisons_niftis.ipynb to compare Hypotheses outputs between two runs. It generates :

   • Statiscal map images for each hypotheses of each run as imgs/[TEAM]/raw_hyps/[HYPOTHESIS]/NARPS*.png
   • Statistical map of the difference between two hypotheses when relevant as imgs/[TEAM]/diff_hyps/[HYPOTHESIS]/diff*.png
   • An emphasized difference statistical map when relevant (a diverging pixel is set to 1E2 to emphasize it) as imgs/[TEAM]/mask_hyps/[HYPOTHESIS]/mask*.png
   • A 3D representation including 4 views in 4 orientations of the difference mask with dimensional colorization (red in image center, green in outer bounds) as imgs/[TEAM]/3D_hyps/[HYPOTHESIS]/mask*.png. Note that those 3D representation come with a .txt files listing all the experimentations that
   • A .csv of the dice coefficient of the hypotheses of both runs (semantics) as csv/[TEAM]/dice/[HYPOTHESIS]/dice*.csv
   • A .csv of the statiscial difference of the 3 types of cuts (sagittal, horizontal, frontal) when there is a structural difference between hypotheses as csv/[TEAM]/stats/[HYPOTHESIS]/stats*.csv. for each slice in the 3 dimensions, it contains mean, std_dev, min, max, nb_of_points_in_slice, nb_of_points_diverging

7. Use CSV_Analysis.ipynb to analyse the csv that were generated previously and observe how important are the differences. It groups Dice values by hypothesis and displays mean, std dev, min and max for each of them. It also summarizes statiscal data for each hypothesis and each cut orientation (sagital, horizontal, frontal).