15.7 KB
Newer Older
A HPC finite element library developed for the implementation of the models of M3DISIM Inria team.

3 4
<!-- toc -->

5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
- [Installation](#installation)
  * [Requirements](#requirements)
    + [Compilers](#compilers)
      - [Installing compilers on macOS](#installing-compilers-on-macos)
      - [Installing compilers on Ubuntu](#installing-compilers-on-ubuntu)
      - [Installing compilers on Fedora](#installing-compilers-on-fedora)
    + [Third-party libraries](#third-party-libraries)
  * [MoReFEM compilation](#morefem-compilation)
    + [XCode](#xcode)
    + [CMake](#cmake)
  * [Testing MoReFEM](#testing-morefem)
- [Using MoReFEM](#using-morefem)
  * [Levels of users](#levels-of-users)
  * [Structure of the source code](#structure-of-the-source-code)
- [How to contribute](#how-to-contribute)
- [Continuous integration](#continuous-integration)
21 22 23
  * [Basic use: on each push](#basic-use-on-each-push)
  * [More thorough jobs](#more-thorough-jobs)
  * [Setting up CI for your fork](#setting-up-ci-for-your-fork)
24 25 26
  * [Doxygen](#doxygen)
  * [Introduction talks](#introduction-talks)
  * [Lexicon](#lexicon)
  * [ModelTutorial](#modeltutorial)
28 29 30
  * [Coding standards](#coding-standards)
- [Getting help](#getting-help)

<!-- tocstop -->
32 33 34 35 36

# Installation

## Requirements

37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
### Compilers

MoReFEM relies on modern C++ and required features shipped with new standard C++ 17. Its has been successfully tested with gcc 7.1, AppleClang 8.0 and LLVM Clang 7.0. gfortran is required by third party libraries (below).

I will detail a bit further below for three OS, but I should stress a fairly recent OS is required to be able to install a recent clang or gcc compiler!

#### Installing compilers on macOS

- Apple clang is shipped with XCode; just install XCode from the AppleStore and open it; additional components will be installed at this moment (including development tools such as git command line).
- gfortran may be taken from [this website](, you may either pick gcc-7.1-bin.tar.gz (the most recent version at the time of this writing) to install all GNU compilers or just gfortran-7.1-bin.tar.gz to install just gfortran.

Assuming it was in your Download file, than run:

cd /
sudo tar xzf ~/Download/gfortran-7.1-bin.tar.gz

and then check it is the correctly found by typing in a terminal:

which gfortran

which should return the proper version number (7.1 here).

#### Installing compilers on Ubuntu

Ubuntu distribution is rather conservative in its compiler version and your Ubuntu is probably shipped with a gcc 5.

To install a more recent one, type:

sudo apt-get install --no-install-recommends -y software-properties-common 
72 73
sudo add-apt-repository ppa:ubuntu-toolchain-r/test && sudo apt-get update \
     && sudo apt-get install -y gcc-7 g++-7 gfortran-7     
74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 100 --slave /usr/bin/g++ g++ /usr/bin/g++-7 --slave /usr/bin/gfortran gfortran /usr/bin/gfortran-7


and check with

which g++

the correct version is returned.

Running again 

sudo update-alternatives --config gcc

allows you to switch back to another version of gcc if need be.

#### Installing compilers on Fedora

Contrary to Ubuntu, Fedora is shipped with fairly recent versions of compilers; check it but on recent Fedora gcc is already a recent one.

### Third-party libraries
101 102 103 104

MoReFEM relies on several third party libraries:

- An optimized Blas library. On macos you may use the Accelerate framework; Openblas has also been successfully tested.
- Lua 5.3, which is required by Ops library (more on that one below).
- Boost C++ library; Boost is present only while waiting for full support of filesystem module in libstdc++ and libc++ STL.
- Openmpi 4
- Parmetis 4.0.3 used to partition the meshes.
- Petsc 3.11: The linear algebra library in charge of the large matrices and vectors. Most of the mpi heavy-lifting is in fact handled by this library. MUMPS, ScaLAPACK and SuperLU\_dist are installed through their Petsc embedded version.
- [TClap](, a header-only library to handle command line arguments.
- [Xtensor](, a linear algebra C++ library used to hande small matrices and vectors in MoReFEM, and its dependencies / extensions ([Xtl](, [Xsimd]( and [Xtensor-blas](
112 113 114
- [Libmesh](, a mesh utility.

The [ThirdPartyCompilationFactory]( project on gitlab provides smooth ways to install these dependencies: either by running directly a Python script or by using one of the given Docker images.
116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157

## MoReFEM compilation

There are currently three different ways to compile MoReFEM library: XCode (for macOS users), SCons (soon-to-be deprecated) and CMake (introduced in early 2018).

### XCode

XCode is widely used among developers in M3DISIM team, so a XCode project is provided with MoReFEM library.

For it to work directly without having to tinker with it, the local hierarchy of projects should be:

- `/Users/${USER}/Codes/MoReFEM/CoreLibrary` which should include the repository of the main library,
- `/Users/${USER}/Codes/MoReFEM/Models` which should include the repositories of the models not embedded within the main library, typically found on [this gitlab folder](

where `USER` is your login on your computer.

The commands to create this structure are:

mkdir -p Codes/MoReFEM/Models
cd Codes/MoReFEM
git clone CoreLibrary

and if you want to install one Model (say AcousticWave for the example):

cd ~/Codes/MoReFEM/Models
git clone 

One you're there, you may run one of the embedded model left there for play here; to change the input file used you just have to edit a scheme in Product > Scheme.

### CMake

To compile the code:

- Go in cmake/PreCache directory and have a look at the predefined files. You may either copy one and modify what you need in the copy directly, or use the script presented below.
- Then go to the place into which you want to build your code (say a build directory at the root of MoReFEM library) and run the command (here for a single static MoReFEM library built for macOS with AppleClang compiler in debug mode):
160 161

python ../cmake/Scripts/ --cache_file=../cmake/PreCache/macos_apple_clang.cmake --cmake_args="-G Ninja"  --install_directory=/Volumes/Data/opt/clang_debug 
163 164

165 166 167
This command calls the adequate CMake command which will configure your project; it will especially create a PreCache file which will keep your actual settings and install it qalong with the library.

To compile the project, proceed with your chosen generator, e.g.:
168 169

ninja -j 8
171 172 173
ninja install

174 175 176 177 178 179 180 181 182 183

make -j 8
make install

__WARNING__: The IDE build do not seem to work satisfactorily at the moment (at least XCode version which is very slow to open).

Once done, you should first check the tests run correctly.

You may also try one of the embedded model with one of the Lua files; to do so you may have to define (unless you're on a Mac and follows the convention location used within M3DISIM team) the following environment variable:
186 187 188 189

- MOREFEM\_ROOT, which should points to the directory in which you should find this very on your disk.
- MOREFEM\_RESULT\_DIR, which shouls points to wherever you want to write your outputs.

The default Lua files use these macro for some paths; you may also create a new Lua file and put in hard the path to your directories.

192 193 194 195 196 197 198 199 200 201 202
## Testing MoReFEM

From v18.16 onward, a CTest integration has been introduced. So once your code is compiled, you should run:

ninja test

and both unit tests and sanity checks for the basic models should be run. Those tests are designed to run in a couple of minutes (with of course strong dependency of the compilation mode chosen).

203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247
# Using MoReFEM

## Levels of users

We have identified 4 levels of users:

- Model user: someone that takes an already implemented Model (e.g. the ones defined in Sources/ModelInstances, or some in [dedicated Model group]( and provides their own Lua file to compute on their own data.
- Model developer: someone able to implement its own model, using for that purpose existent parameters and operators. All the Models defined in Sources/ModelInstances are at this level of expertise.
- Advanced model developer: Same as Model developer, but who also uses up more advanced features (that are in _Advanced_ namespace). Typically, an advanced model user might develop new operators tailored for its own purpose.
- Library developer: someone that writes code involved in low level operations of the library (typically in namespace _Internal_). 

## Structure of the source code

The hierarchy of the source code is the following:

- Utilities: a lot of handy tools that might as well be used in completely different projects (e.g. displaying content of a container, creating safely a file on a filesystem, etc...)
- ThirdParty: stuff related to third party libraries: actual sources for some of them, wrappers to provide a nice C++ interface within the code, facility to avoid getting compiler warnings from header files, etc...

Both those directories are compiled together when the compilation is set to generate MoReFEM in several libraries, as they are strongly intertwined together.

- Core: This module encompass some generic stuff that may be used at every level of MoReFEM; contrary to Utilities ones however they are already specific to the library.
- Geometry: This module encompass purely geometric objects of the code, such as the mesh, geometric elements, domains and so forth.
- FiniteElement: Large module to encompass stuff closely related to finite elements: nodes and dofs, boundary conditions, finite element spaces, unknowns and so forth...
- Parameters: This module encompass the API of so-called Parameters, which are objects for which values at quadrature points might be queried.
- Operators: This module encompass operators the API for several type of operators: variational operators, interpolators, etc...
- ParameterInstances: This module provides implementation of several parameters; it is separated from Parameters module as knowledge of stuff defined in Operators is required.
- OperatorInstances: This module provides implementation of several operators. As for ParameterInstances, it is separated from Operators module as it requires stuff from Parameters and ParameterInstances.
- FormulationSolver: module that provides the base class used to define a variational formulation.
- Model: module around the Model class, which drives the whole MoReFEM program.

The modules above form the core of MoReFEM library, which may be used to create a model of your own.

Besides them, three additional modules are provided:

- PostProcessing: a module which uses up the output data to provide a usable output. For instance default output might be used to provide input for Ensight visualisation software.
- ModelInstances: few models let here to show how the library might be used to define a Model. 
- Test: several integration tests that are run before each tag. They should soon be adapted on Inria continuous integration platform.

# How to contribute

As explained [earlier](#levels-of-users), most of the time you shouldn't have to contribute directly to the library and should be able to define locally in your own model what you need. However, if you have somes fixes to provide or maybe something to share (for instance an operator not implemented that might be of interest for others), feel free to submit a [merge request]( on our gitlab.
249 250 251 252 253

MoReFEM follows the [integration manager]( workflow; internal developers are actually using a Redmine installed on a server of M3DISIM team.

# Continuous integration

MoReFEM provides continuous integration through the use of Gitlab-CI.

## Basic use: on each push

260 261 262
Each time you push code to your repository, CI should be kicked off. By default two stages are enabled:
     - One named 'build_and_test' that compiles the code and runs the test for a given environment (Ubuntu or Fedora, clang or gcc, debug or release...). It also runs Doxygen for three different configurations (basic, advanced and complete).
     - One named 'check_warnings' which checks there are no warnings in the compilation/Doxygen from previous stage.

## More thorough jobs

There are additional optional steps also available in CI:

- One named 'valgrind', which runs Valgrind on the embedded models and on few tests.
- One named 'verrou', which runs [Verrou]( (a floating-point precision tool) on the embedded models and on few tests.
- One named 'analysis', which runs several static analysis tools. Another stage 'sonarqube' aggregates the results and put them on a [Sonarqube](] instance.

There are three ways to kick in these stages:

- Push onto 'develop' branch of the main project (reserved for the integrator - i.e. myself only for the time being).
275 276
- Push onto a branch which include 'valgrind' (for Valgrind), 'sonarqube' (for the static analysis) or 'verrou' (for floating-point precision) in its name. Several may be combined: if you push to a branch named 10000_new_feature_sonarqube_valgrind both sonarqube and valgrind will be triggered (but not verrou),.
- Push onto a branch which name include 'full_ci_analysis'. This will trigger everything. 

278 279 280
For the Sonarqube stage, you need to define two environment variables in the gitlab fork of MoReFEM:
- `SONARQUBE_LOGIN` , and assign to it the token generated on your [Sonarqube account](
- `SONARQUBE_PROJECT` and assign to it what will be an identifier for your project. Typically put your identifier (i.e. `sgilles`; this variable was actually introduced for the main repository).

## Setting up CI for your fork

See <a href="Documentation/ContinuousIntegration/">the dedicated README</a> to see how to set up the CI pipeline for your own fork.

287 288 289 290 291

## Doxygen

Doxygen documentation is comprehensive and up-to-date; there are actually three flavors of them. For more informations, see the in Documentation/Doxygen directory.

292 293
Version generated at the last MoreFEM tag is available at [this address](; this version features the complete documentation.

294 295 296

## Introduction talks

Three explaining talks given in 2017 and 2018 to explain the structure of the library; they are in Documentation/IntroductionTalks directory.
298 299 300

Both were originally make in macOS Keynote; conversions to ppt and html are also given (with the later probably more useful). 'HappyHeart' was the codename of MoReFEM until late 2017.

301 302 303 304 305

## Lexicon

The main concepts of MoReFEM given on less than 3 slides each.

306 307 308 309
## ModelTutorial

You will find there a [tutorial](Documentation/ModelTutorial/ to build from scratch a simple elastic model with constant time step and volumic mass.

310 311
## Coding standards

Coding standards are provided in Documentation/CodingStandards. They are mostly inspired by those of [Verdandi](
313 314 315 316

# Getting help

The library was developed primarily by Sébastien Gilles (sebastien.gilles [AT], with also many contributions from Gautier Bureau, a very advanced model developer which is the author of most of the [advanced models](
318 319 320 321 322 323 324 325 326 327