GitLab

- adds a tab in settings window for the selections of Event and Link types to display, with future (not working yet) possibility to change the color. You can check the wanted states, and press reload to apply the changes to the trace. This needs testing.
- for splitted files, move entityvalues inside the .vite file to avoid replication and memory problems for reloading


This line, and those below, will be ignored--

M    src/render/Hook_event.cpp
M    src/interface/Settings_window.cpp
M    src/interface/settings.ui
M    src/interface/Settings_window.hpp
M    src/src.pro
M    src/common/Session.cpp
M    src/common/Session.hpp
M    src/parser/ParserSplitted.cpp
M    src/trace/Trace.cpp
M    src/trace/Serializer_types.hpp
M    src/trace/Trace.hpp
M    src/trace/Serializer_structs.hpp
M    src/trace/Serializer.cpp
M    src/trace/IntervalOfContainer.cpp
M    src/trace/DrawTree.hpp
M    src/trace/EntityValue.cpp
M    src/trace/EntityType.cpp
M    src/trace/DrawTrace.hpp
M    src/trace/Container.cpp
M    src/trace/EntityType.hpp

- re enable horizontal scrolling (ctrl + mouse wheel) which was broken since 1202
- avoid negative advancement values in parser splitted

add possibility to enter values greater than max_trace in interval selection, and removes a bug with variables and split files for containers' size computation

- add a second thread to separate OTF file operations and trace operations : not as much parallelism as in Paje as file operations are more efficient for OTF (as for Paje parsing, flag MT_PARSING)

- multithreaded loading of serialized files : moves loading from the main thread to multiple threads, as for serialization.

- beginning of the work with MPI : work can now be distributed and several intervals loaded from several machines and displayed there. 

How to use: the flag USE_MPI must be set, links  -lboost_mpi -lmpi -lmpi_cxx must be added in src.pro

mpicc and mpic++ have to be used instead of gcc and g++. This can be set manually in src/makefile, but will be reseted by the global makefile. Another solution is to add

unix{
QMAKESPEC = mpicc
QMAKE_CXX = mpic++
}

to the src.pro file.

To launch on a single machine :
mpirun -np nprocess vite path/to/file.vite2/configure

on several machines with separated screens and a machinefile to list them :
- allow ssh authentification without password (by key)
- allow display on each distant machine with "xhost +"
- have a split folder in a folder accessible with the same path for each machine (nfs or local)
- have vite in the path for each machine
- use mpirun -np nprocess -hostfile machinefile -mca orte_rsh_agent "ssh -X" -xDISPLAY=:0.0 vite -tInterval path/to/file.vite

This will split the given interval in nprocess parts and send them for display to the various nodes described, displaying on their local display

removes a few warnings, a bug in core that happened sometimes and prevented trace from displaying, a bug with variables in Container finish with itc, and adds a message when trying to use interval or node selection with no trace loaded

- better handling of trace reload with interval selection window

Feature : add an Interval Selection Window, allowing to select a displayable interval on the trace. (Preferences/Interval Selection)

Combined with IntervalOfContainers and Serialization, this allows to change the displayed interval and load only the data of the wanted interval, while freeing memory for non displayed zones.

Summary :

- trace can be dumped on disk while parsing
- data can be restored, loading only in memory the part of trace we want to display (by time and by containers).
- a light preview version of the whole trace can be displayed, allowing to chose the interval and actually load data from it


How it works :

- allow Serialization of IntervalOfContainers to the disk while parsing. Each finished IntervalOFContainer (containing 10000 states or links) can be dumped to the disk, using Boost serialization library, in a separate file, then its memory is freed, allowing to parse huge files (tested with 8Gb). Each type, container, linked in the IntervalOfContainer is assigned a unique id to avoid serializing too much data (file Serializer.hpp). If Boost with gzip is used, the resulting data is compressed. This is handled by Several SerializerWriter threads, and the assignment to each thread is done by a Singleton object, the SerializerDispatcher. The number of threads used is the number of CPUs found in the machine.

At the end of parsing all remaining loaded intervalOfContainers are dumped. File naming is "Unique ID of the container"_"IntervalOfContainer index in the container". They are saved in a folder named after the trace file, without extension.  

At the end of dumping, we have a folder containing many files. A file called "name of the trace".vite is created in this folder, which handles all containers, types, with their unique IDs. For each IntervalOFContainer of each Container, the beginning and end timings are also saved. This file will be used to correlate data from the multiple IntervalOfContainers files. It stores also the sum of all the times of all StateTypes encountered in each intervalOFContainers.

- we can now open this .vite file. A ParserSplitted is then used, allowing to restore the structure of the trace and all types. 

   - If the -t option is specified with a time interval, data is directly reloaded from the serialized files, loading in memory only the intervalofcontainers in the time interval.
   - If the -t option was not specified, we load the preview version of the trace, contained in the .vite file.

The preview version only stores states for the moment. When browsing the preview version, user can select a zone and press ctrl. This opens a new vite window, with the same zoom, but the data is then loaded from the serialized files.



How to use : 

needed libraries : libboost_serialization, libboost_thread, and libboost_iostreams . These libraries are in the standard boost package. In linux, they include the gzip library needed for compression and bindings. In windows, this library is not included and has to be included after, and boost recompiled.

- cmake : activate the option VITE_ENABLE_SERIALIZATION in order to check for boost libraries, and to add corresponding files 
- configure :  add the flag --use_boost if libraries are in /usr/lib, --boost_libdir=$dir else.
- by hand in the src.pro file :  add needed libraries ( -lboost_serialization -lboost_thread -lboost_iostreams ) and flags USE_ITC, BOOST_SERIALIZE, and BOOST_GZIP to activate everything



todo: 
- make preview + -c option work together ( -c and -t work together for the moment, -c and preview also, but not when loading actual data from disk)
- add other data to the preview (links events and variables)
- check if using lots of threads to compress is really useful
- better balance between those threads, without rebinding qt signal/slots each time
- tests, tests and tests.
- documentation and comments.

- removes a segfault occurring when clicking on the trace

add IntervalOfContainers which are described in https://gforge.inria.fr/plugins/wiki/index.php?NewDataStructPage&id=1596&type=g
A new IntervalOfContainer is built only when a certain (10000 for the moment) number of StateChanges or Links are attached to it. (initially only StateChanges were taken into account, but Links are created in parent containers without States, so they were all allocated in the same IntevalOfContainers, making them huge).

This is not activated right now by default, but can be turned on by setting the flag USE_ITC before compiling to perform tests.

- add a few messages and warning in core
- no longer try to parse .otf files with paje parser when otf is not compiled
- fix a bug with vertical line not disappearing when clicking on containers
- memory improvements :
    - do not store extra_fields for each entity as an empty map when they are not present but as a null pointer (need more checks to avoid segfaults, hope I didn't forget some in drawtrace and other files needing get_extra_fields)
    - remove a virtual destructor in Entity.h (virtual functions add memory overhead to their inheriting objects)
-> on my windows, sizeof(Link) was 72, it is now 56, and for a 55Mb trace I went from 325 Mb with 1.2 to 275 Mb of used memory with trunk

- change the way types are searched at each insertion (we iterated through lists at each insertion to find the right types, now we use maps which provide much better searching performance)

- a few warnings removed for windows again
- node selection while zooming now almost keeps the zoom (I don't get why the min changes a little bit, and don't know if it's possible to fix this)

new features:
-multithreaded Paje parser : this parser uses 3 threads :
   - the parsing thread to read the file and produce lines and tokens (lexical analysis), and aggregate them in blocks of 10000 lines 
   - the builder thread which handles these blocks of lines and calls store_event of the ParserEventPaje and transforms the tokens into the appropriate types, checks the correctness of the line (syntaxic analysis) but doesn't perform the calls to trace and the structural verifications 
  - the trace building thread which performs semantic analysis (if types, containers, exist) and performs the calls to the trace (adding events, states to the trace).

- file mapping : just for multithread version, the file is mapped into memory by chunks (100MB for the moment), which is faster and allows to handle larger files without using too much memory. The limit of 1Gb of the other version is removed, ViTE can now handle much larger files.


These features need more testing and feedback and can be activated at compile time by setting the flag MT_PARSING. 

note: the parser still uses a tokens number limit, this will be merged with the new version soon.


issues:
- gracefully stop parsing when cancel button is hit is not handled yet and causes segfaults

reset the vertical zoom before reloading, in order to update the size of the containers and fill the window when changing the set of containers.

modifications on the handling of the Parsing thread (preparing futures commits for multi-threaded parsing) => vite no longer waits for a full second to detect the end of parsing

slight render optimizations for events and links drawing 
colors are now stored on floats and no longer on double for rendering

various fixes, comments\n load_xml file was moved to Trace.cpp in order to separate this from the node selection interface => Now node selection work with SVG, and -e flag can be used with -c flag. \n Use of QCoreApplication instead of QApplication when we don't need an interface (-e flag for the moment, but will also be useful when -s flag will be added to split the trace) => ViTE now works without the need of an X server for svg export\n

Since it is use for initializing the QApplication, argc should be passed as a reference ("The data referred to by argc and argv must stay valid for the entire lifetime of the QApplication object").
This fix bug #10585 (at least it works for me)

- fix build on windows platforms where getopt is not present : add a version of getopt called xgetopt (license ok) when called on Windows.
- fix parsing issues with Paje traces on Windows, caused by the switch to std::getline, and the fact that it now removes the endline character (the character after the end of an std::line is seen as a \n on linux, so the bug is not apparent but still present)

feature :
- add a window that allows the user to select the containers to display, reorder them or hide them (by drag and drop and checking/unchecking). The selected display can be saved to an xml file, and reloaded for another trace. Found in Preferences/Node Selection
- add the flag -c to specify such an xml file to load initially for a trace

known issues and todo :
- only works with OpenGl render : separate interface from work on the xml file to allow use with SVC
- zoom is badly handled, and containers are not redrawn with new sizes yet
- no tests done with non Paje traces, should work though
- lacks comments and cleaning
- put the window as a plugin ?

fix start/end link for OTF trace files (traces generated with eztrace/gtg can now be visualized properly)

Replace options by a getopt / Now -t is working on opengl, probably not anymore on export, I have to fix something else

add a vertical helper line when a single click is made on the trace. Click again on the same spot or in the containers to hide it. The line can also be deactivated in the view menu, and isn't assigned on double clicks