Commit fbf0b6c1 authored by Augustin Degomme's avatar Augustin Degomme

A lot of testing and feedback on this one is needed, in order to improve the way it is done.

Summary :

- trace can be dumped on disk while parsing
- data can be restored, loading only in memory the part of trace we want to display (by time and by containers).
- a light preview version of the whole trace can be displayed, allowing to chose the interval and actually load data from it


How it works :

- allow Serialization of IntervalOfContainers to the disk while parsing. Each finished IntervalOFContainer (containing 10000 states or links) can be dumped to the disk, using Boost serialization library, in a separate file, then its memory is freed, allowing to parse huge files (tested with 8Gb). Each type, container, linked in the IntervalOfContainer is assigned a unique id to avoid serializing too much data (file Serializer.hpp). If Boost with gzip is used, the resulting data is compressed. This is handled by Several SerializerWriter threads, and the assignment to each thread is done by a Singleton object, the SerializerDispatcher. The number of threads used is the number of CPUs found in the machine.

At the end of parsing all remaining loaded intervalOfContainers are dumped. File naming is "Unique ID of the container"_"IntervalOfContainer index in the container". They are saved in a folder named after the trace file, without extension.  

At the end of dumping, we have a folder containing many files. A file called "name of the trace".vite is created in this folder, which handles all containers, types, with their unique IDs. For each IntervalOFContainer of each Container, the beginning and end timings are also saved. This file will be used to correlate data from the multiple IntervalOfContainers files. It stores also the sum of all the times of all StateTypes encountered in each intervalOFContainers.

- we can now open this .vite file. A ParserSplitted is then used, allowing to restore the structure of the trace and all types. 

   - If the -t option is specified with a time interval, data is directly reloaded from the serialized files, loading in memory only the intervalofcontainers in the time interval.
   - If the -t option was not specified, we load the preview version of the trace, contained in the .vite file.

The preview version only stores states for the moment. When browsing the preview version, user can select a zone and press ctrl. This opens a new vite window, with the same zoom, but the data is then loaded from the serialized files.



How to use : 

needed libraries : libboost_serialization, libboost_thread, and libboost_iostreams . These libraries are in the standard boost package. In linux, they include the gzip library needed for compression and bindings. In windows, this library is not included and has to be included after, and boost recompiled.

- cmake : activate the option VITE_ENABLE_SERIALIZATION in order to check for boost libraries, and to add corresponding files 
- configure :  add the flag --use_boost if libraries are in /usr/lib, --boost_libdir=$dir else.
- by hand in the src.pro file :  add needed libraries ( -lboost_serialization -lboost_thread -lboost_iostreams ) and flags USE_ITC, BOOST_SERIALIZE, and BOOST_GZIP to activate everything



todo: 
- make preview + -c option work together ( -c and -t work together for the moment, -c and preview also, but not when loading actual data from disk)
- add other data to the preview (links events and variables)
- check if using lots of threads to compress is really useful
- better balance between those threads, without rebinding qt signal/slots each time
- tests, tests and tests.
- documentation and comments.
parent a7c3c8f3
......@@ -15,6 +15,17 @@ INCLUDE(${QT_USE_FILE})
FIND_PACKAGE(GLU)
# search for Boost
# Components :
#filesystem, iostreams, programoptions, python, regex, serialization, signals
#system, thread, wave
OPTION(VITE_ENABLE_SERIALIZATION "Enable the support of Boost Serialization." OFF)
IF(VITE_ENABLE_SERIALIZATION)
FIND_PACKAGE( Boost COMPONENTS serialization thread iostreams)
ENDIF(VITE_ENABLE_SERIALIZATION)
OPTION(VITE_ENABLE_VBO "Enable the support of VBO." OFF)
IF(VITE_ENABLE_VBO)
INCLUDE(cmake/FindGLEW.cmake)
......
......@@ -88,6 +88,10 @@ tau :
tauclean :
"
#BOOST
with_boost=0
# QTColor
qtcolordir="$PWD/externals/qtcolorpicker"
qtcolorincdir="$qtcolordir/src"
......@@ -167,7 +171,13 @@ usage()
echo " include path to the TAU headers"
echo ""
echo " --tau_libdir=<path>"
echo " library path to the TAU libraries"
echo " library path to the TAU libraries" echo ""
echo ""
echo " --use_boost"
echo " use boost serialization in /usr/lib/"
echo ""
echo " --boost_libdir=<path>"
echo " library path to the boost libraries"
echo ""
}
......@@ -234,6 +244,14 @@ while [ 0 -lt $# ]; do
--qtcolor_libdir=*)
qtcolorlibdir=`echo $OPT | awk -F"--qtcolor_libdir=" '{print $2}'`
;;
--use_serialization*)
with_boost=1;
boost_lib_dir=/usr/lib/
;;
--serialization_libdir=*)
with_boost=2;
boost_lib_dir=`echo $OPT | awk -F"--boost_libdir=" '{print $2}'`
;;
*)
usage
echo "";
......@@ -291,6 +309,55 @@ if [ $with_tau -ge 1 ]; then
fi
fi
if [ $with_boost -ge 1 ]; then
# BOOST
boostser=0;
boostthread=0;
boostiostreams=0;
for p in $boost_lib_dir; do
echo "${p}"
if [ -f "${p}/libboost_serialization.so" ]; then
echo "${p}"
boostser=1;
libs="$libs -lboost_serialization"
fi
if [ -f "${p}/libboost_thread.so" ]; then
boostthread=1;
libs="$libs -lboost_thread"
fi
if [ -f "${p}/libboost_iostreams.so" ]; then
boostiostreams=1;
libs="$libs -lboost_iostreams"
fi
done
if [ $boostser -eq 0 ]; then
echo "$boostser ERROR: Boost serialization library not found in ${LIBPATH}"
exit;
fi
if [ $boostthread -eq 0 ]; then
echo "ERROR: Boost thread library not found in LD_LIBRARY_PATH"
exit;
fi
if [ $boostiostreams -eq 0 ]; then
echo "ERROR: Boost iostreams library not found in LD_LIBRARY_PATH"
exit;
fi
defines="$defines USE_ITC BOOST_SERIALIZE"
sources="$sources parser/ParserSplitted.cpp trace/IntervalOfContainer.cpp trace/SerializerWriter.cpp trace/SerializerDispatcher.cpp"
headers="$headers parser/ParserSplitted.hpp trace/IntervalOfContainer.hpp trace/SerializerWriter.hpp trace/SerializerDispatcher.hpp"
fi
# QT Color Picker
dependpath="$dependpath $qtcolorsrcdir"
includepath="$includepath $qtcolorincdir"
......@@ -453,4 +520,10 @@ else
echo TAU spport [DISABLE]
fi
if [ $with_boost -ge 1 ]; then
echo Serialization support [OK]
else
echo Serialization spport [DISABLE]
fi
echo ViTE is now configured for building. Just run \'$MAKE\'.
......@@ -313,6 +313,35 @@ ENDIF(VITE_ENABLE_TAU)
#ADD_DEFINITIONS(-DMEMORY_USAGE)
#ADD_DEFINITIONS(-DMEMORY_TRACE)
#############################################
# BOOST
#############################################
IF(VITE_ENABLE_SERIALIZATION)
link_directories ( ${Boost_LIBRARY_DIRS} )
INCLUDE_DIRECTORIES ( ${Boost_INCLUDE_DIRS} )
ADD_DEFINITIONS("-DUSE_ITC -DBOOST_SERIALIZE")
SET(VITE_HDRS
${VITE_HDRS}
parser/ParserSplitted.hpp
trace/IntervalOfContainer.hpp
trace/SerializerWriter.hpp
trace/SerializerDispatcher.hpp
)
SET(VITE_SRCS
${VITE_SRCS}
parser/ParserSplitted.cpp
trace/IntervalOfContainer.cpp
trace/SerializerWriter.cpp
trace/SerializerDispatcher.cpp
)
ENDIF(VITE_ENABLE_SERIALIZATION)
#############################################
# QT4
#############################################
......@@ -344,7 +373,15 @@ ADD_EXECUTABLE(vite ${VITE_SRCS} ${VITE_MOC} ${VITE_RCC_SRCS})
TARGET_LINK_LIBRARIES(vite
${QT_LIBRARIES}
${GLU_LIBRARY}
${Boost_LIBRARIES}
)
IF(${CMAKE_SYSTEM_NAME} MATCHES "Linux")
TARGET_LINK_LIBRARIES(vite
rt
)
ADD_DEFINITIONS("-DBOOST_GZIP")
ENDIF(${CMAKE_SYSTEM_NAME} MATCHES "Linux")
IF(VITE_ENABLE_VBO)
TARGET_LINK_LIBRARIES(vite
......
......@@ -46,6 +46,8 @@
#include "common/common.hpp"
#include "common/Info.hpp"
#include <string>
using namespace std;
unsigned int Info::Screen::width = 800;
unsigned int Info::Screen::height = 600;
......@@ -77,6 +79,15 @@ bool Info::Render::_no_events = false;
bool Info::Render::_shaded_states = true;/* By default, enable shaded state */
bool Info::Render::_vertical_line = true;/* By default, enable vertical line */
bool Info::Splitter::split = false;
bool Info::Splitter::load_splitted = false;
bool Info::Splitter::preview = false;
std::string Info::Splitter::path ;
std::string Info::Splitter::filename ;
Element_pos Info::Splitter::_x_min = 0.;
Element_pos Info::Splitter::_x_max = 0.;
int Info::Trace::depth=0;
void Info::release_all(){
......
......@@ -46,7 +46,7 @@
#ifndef INFO_HPP
#define INFO_HPP
#include <string>
/*!
* \brief Class used to store informations.
*
......@@ -272,6 +272,39 @@ public:
static int depth;
};
struct Splitter{
/*!
* \brief are we splitting ?
*/
static bool split;
/*!
* \brief are we loading from a splitted set of files (needed in core to know which action to initiate)
*/
static bool load_splitted;
/*!
* \brief are we loadinga preview version of a splitted file (needed in core to know which action to initiate)
*/
static bool preview;
/*!
* \brief path of the folder to split
*/
static std::string path;
/*!
* \brief filename of the splitted file, used to name the folder
*/
static std::string filename;
/*!
* \brief min time we want to load
*/
static Element_pos _x_min;
/*!
* \brief max time we want to load
*/
static Element_pos _x_max;
};
};
#endif
This diff is collapsed.
......@@ -217,6 +217,10 @@ public:
static const int _STATE_CLEAN_RENDER_AREA = 25;
static const int _STATE_ZOOM_IN_AN_INTERVAL=26;
static const int _STATE_SPLITTING=27;
static const int _LOAD_DATA = 28;
/*!
* \brief Launch an action according to the argument state value.
* \param state An integer corresponding to a kind of action which must be executed.
......@@ -480,19 +484,25 @@ public:
void export_variable(Variable *var, const std::string &filename);
//Qt signals fo the Parsing Thread
signals:
signals:
/*!
*\brief run_parsing()
* signal to launch the parsing
*/
void run_parsing();
*/
void run_parsing();
/*!
*\brief build_finished()
* signal to wait for the end of event loop
*/
*/
void build_finished();
/*!
*\brief dump())
* signal to dump last IntervalOfContainers and wait for the end of event loop
*/
void dump(std::string, std::string);
};
......
#ifdef WIN32
#include <windows.h>
#include <stdio.h>
......@@ -67,4 +68,6 @@ int getopt(int argc, char *argv[], char *optstring)
}
return c;
}
\ No newline at end of file
}
#endif
......@@ -266,6 +266,9 @@ void Stats_window::set_selected_nodes(string kind_of_state){
_trace->set_view_root_containers(_displayed_containers);
//_trace->set_selected_container(_trace->get_root_containers());
_console->get_console()->launch_action(Core:: _STATE_CLEAN_RENDER_AREA);
#if defined(USE_ITC) && defined(BOOST_SERIALIZE)
_trace->updateTrace(new Interval(zoom[0], zoom[1]));
#endif
_console->get_console()->draw_trace(_console->get_filename(),Core::_DRAW_OPENGL);
_console->get_console()->launch_action(Core:: _STATE_ZOOM_IN_AN_INTERVAL, &zoom);
_console->get_console()->launch_action(Core:: _STATE_RENDER_UPDATE);
......@@ -337,6 +340,9 @@ void Node_select::on_load_button_clicked(){
Info::Render::_x_min_visible = 0.0;
Info::Render::_x_max_visible = 0.0;
_console->get_console()->launch_action(Core:: _STATE_CLEAN_RENDER_AREA);
#if defined(USE_ITC) && defined(BOOST_SERIALIZE)
_trace->updateTrace(new Interval(it[0], it[1]));
#endif
_console->get_console()->draw_trace(_console->get_filename(),Core::_DRAW_OPENGL);
_console->get_console()->launch_action(Core:: _STATE_ZOOM_IN_AN_INTERVAL, &it);
......@@ -352,6 +358,7 @@ void Node_select::on_load_button_clicked(){
reassign_children_rec(*it,flg);
}
++it;
}
......
......@@ -70,6 +70,11 @@
#include "parser/ParserTau.hpp"
#endif //WITH_TAU
#ifdef BOOST_SERIALIZE
#include "ParserSplitted.hpp"
#endif
#include "ParserFactory.hpp"
using namespace std;
......@@ -108,7 +113,14 @@ bool ParserFactory::create(Parser **parser,
*Message::get_instance() << "Tau parser was not compiled. Use parser Paje by default" << Message::endw;
return false;
#endif //WITH_TAU
}
} else if(ext == ".vite") {
#ifdef BOOST_SERIALIZE
*parser = new ParserSplitted(filename);
#else
*Message::get_instance() << "Boost serialization was not compiled. We cannot parse .vite files" << Message::endw;
#endif //WITH_OTF
}
}
if(*parser == NULL) {
......
#include <iostream>
#include <fstream>
#include <string>
#include <set>
#include <map>
#include <queue>
#include <list>
#include <vector>
/* -- */
#include "common/common.hpp"
#include "common/Errors.hpp"
#include "common/Info.hpp"
/* -- */
#if defined(USE_ITC) && defined(BOOST_SERIALIZE)
/* -- */
#include "parser/Parser.hpp"
/* -- */
#include <boost/serialization/string.hpp>
#include <boost/serialization/base_object.hpp>
#include <boost/archive/tmpdir.hpp>
#include <boost/serialization/utility.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <boost/serialization/list.hpp>
#include <boost/serialization/assume_abstract.hpp>
#include "boost/serialization/map.hpp"
#include "trace/Serializer_values.hpp"
#include "trace/Serializer_types.hpp"
#include "trace/Serializer_structs.hpp"
#include "trace/Serializer_container.hpp"
#include "trace/values/Values.hpp"
#include "trace/tree/Interval.hpp"
#include "trace/EntityTypes.hpp"
#include "trace/Entitys.hpp"
#include "trace/Trace.hpp"
#include "trace/Container.hpp"
#include "parser/ParserSplitted.hpp"
using namespace std;
ParserSplitted::ParserSplitted(){}
ParserSplitted::ParserSplitted(const string &filename): Parser(filename){
}
ParserSplitted::~ParserSplitted() {
}
void ParserSplitted::parse(Trace &trace,
bool finish_trace_after_parse) {
Info::Splitter::load_splitted=true;
//dump containers
// char* filename=(char*)malloc(128*sizeof(char));
// sprintf(filename,"%s/%s.vite", Info::Splitter::path, );
std::list<ContainerType*> conts_types;
std::list<Container*> conts;
std::ifstream ifs(_file_to_parse.c_str());
boost::archive::text_iarchive ia(ifs);
ia.register_type(static_cast<StateType *>(NULL));
ia.register_type(static_cast<EventType *>(NULL));
ia.register_type(static_cast<VariableType *>(NULL));
ia.register_type(static_cast<LinkType *>(NULL));
ia.register_type(static_cast<ContainerType *>(NULL));
ia.register_type(static_cast<Container *>(NULL));
ia.register_type(static_cast<Color *>(NULL));
ia.register_type(static_cast<Date *>(NULL));
ia.register_type(static_cast<Double *>(NULL));
ia.register_type(static_cast<Hex *>(NULL));
ia.register_type(static_cast<Integer *>(NULL));
ia.register_type(static_cast<Name *>(NULL));
ia.register_type(static_cast<String *>(NULL));
ia >> conts_types;
trace.set_container_types(conts_types);
//int test;
//ia >> test;
// restore the schedule from the archive
ia >> conts;
// (trace.get_root_containers())->insert((trace.get_root_containers())->begin(), conts.begin(),conts.end() );
trace.set_root_containers(conts);
std::map<Name, StateType* > _state_types;
std::map<Name, EventType* > _event_types;
std::map<Name, LinkType* > _link_types;
std::map<Name, VariableType* > _variable_types;
ia >> _state_types;
trace.set_state_types(_state_types);
ia >> _event_types;
trace.set_event_types(_event_types);
ia >> _link_types;
trace.set_link_types(_link_types);
ia >> _variable_types;
trace.set_variable_types(_variable_types);
Date d;
ia >>d;
trace.set_max_date(d);
if(finish_trace_after_parse) {
//we need to build the display only if we are not splitting the trace
trace.finish();
finish();
}
// unsigned long long after = getCurrentTime();
// g_totalTime+=(after-before);
}
void ParserSplitted::releasefile(){
}
float ParserSplitted::get_percent_loaded() const {
return 0.0;
}
#endif
/*
** This file is part of the ViTE project.
**
** This software is governed by the CeCILL-A license under French law
** and abiding by the rules of distribution of free software. You can
** use, modify and/or redistribute the software under the terms of the
** CeCILL-A license as circulated by CEA, CNRS and INRIA at the following
** URL: "http://www.cecill.info".
**
** As a counterpart to the access to the source code and rights to copy,
** modify and redistribute granted by the license, users are provided
** only with a limited warranty and the software's author, the holder of
** the economic rights, and the successive licensors have only limited
** liability.
**
** In this respect, the user's attention is drawn to the risks associated
** with loading, using, modifying and/or developing or reproducing the
** software by the user in light of its specific status of free software,
** that may mean that it is complicated to manipulate, and that also
** therefore means that it is reserved for developers and experienced
** professionals having in-depth computer knowledge. Users are therefore
** encouraged to load and test the software's suitability as regards
** their requirements in conditions enabling the security of their
** systems and/or data to be ensured and, more generally, to use and
** operate it in the same conditions as regards security.
**
** The fact that you are presently reading this means that you have had
** knowledge of the CeCILL-A license and that you accept its terms.
**
**
** ViTE developers are (for version 0.* to 1.0):
**
** - COULOMB Kevin
** - FAVERGE Mathieu
** - JAZEIX Johnny
** - LAGRASSE Olivier
** - MARCOUEILLE Jule
** - NOISETTE Pascal
** - REDONDY Arthur
** - VUCHENER Clément
**
*/
/*!
* \file ParserPaje.hpp
* \brief The implementation of Parser for Paje traces.
*/
#ifndef PARSERSPLITTED_HPP
#define PARSERSPLITTED_HPP
#include <cstdio>
#include <QtCore>
#include <parser/Parser.hpp>
#include <parser/PajeFileManager.hpp>
class ParserDefinitionPaje;
class ParserEventPaje;
class PajeFileManager;
/*!
*
* \class ParserPaje
* \brief parse the input data format of Paje.
*
*/
class ParserSplitted : public Parser{
public:
/*!
* \fn ParserPaje()
*/
ParserSplitted();
ParserSplitted(const std::string &filename);
/*!
* \fn ~ParserPaje()
*/
~ParserSplitted();
/*!
* \fn parse(Trace &trace)
* \param trace : the structure of data to fill
* \param finish_trace_after_parse boolean set if we do not have to finish the trace after parsing
*/
void parse(Trace &trace,
bool finish_trace_after_parse = true);
void releasefile();
/*!
* \fn get_percent_loaded() const
* \brief return the size of the file already read.
* \return the scale of the size already loaded of the file by the parser. (between 0 and 1)
*/
float get_percent_loaded() const;
/*!
* Return the parser of definitions.
*
*/
ParserDefinitionPaje *get_parser_def() const;
signals:
void produced(int i, PajeLine* line);
void build_finish();
};
#endif // ParserSplitted_HPP
......@@ -63,7 +63,8 @@ ParsingThread::ParsingThread(Parser *p, Trace *t, QWaitCondition* finished,
void ParsingThread::run() {
try {
_parser->parse(*_trace);
_parser->parse(*_trace,false);//do not finish the build inside the parser, we will do this after, if we don't want to just dump the trace
_parser->finish();//set the parser's finished flag to true
}
catch (const std::string &) {
_parser->finish();
......@@ -75,5 +76,14 @@ void ParsingThread::run() {
void ParsingThread::finish_build(){
_is_finished=true;
_trace->finish();
_finished->wakeAll();
}
\ No newline at end of file
}
void ParsingThread::dump(std::string path, std::string filename){
_is_finished=true;
#if defined(USE_ITC) && defined(BOOST_SERIALIZE)
_trace->dump(path, filename);
#endif
_finished->wakeAll();
}
......@@ -91,5 +91,11 @@ protected slots:
*/
void finish_build();
/*!
* \fn dump(std::string path)
* \brief dumps the remaining IntervalOfContainer at path path
*/
void dump(std::string path,std::string filename);
};
#endif // PARSING_THREAD_HPP
......@@ -519,6 +519,18 @@ void Hook_event::apply_zoom_box(Element_pos x_min, Element_pos x_max, Element_po
_x_state_translate -= (x_scaled_middle_selection - x_distance_between_state_origin_and_render_middle);
_y_state_translate -= (y_scaled_middle_selection - y_distance_between_state_origin_and_render_middle);
}
#if defined(USE_ITC) && defined(BOOST_SERIALIZE)
if(Info::Splitter::preview==true && Info::Render::_key_ctrl == true){
//we are in preview mode, we have to load the actual data from the serialized files
Info::Splitter::_x_min= Info::Render::_x_min_visible;
Info::Splitter::_x_max= Info::Render::_x_max_visible;
// Info::Splitter::preview=false;
_core->launch_action(_core->_LOAD_DATA, NULL);
Info::Render::_key_ctrl =false;
// return;
}
#endif
/* Update render */
updateRender();
......
......@@ -56,11 +56,11 @@ DEPENDPATH += .
INCLUDEPATH += .
QMAKE_CFLAGS += -O2 -Wall -g
#-DMT_PARSING
QMAKE_CXXFLAGS += -O2 -Wall -g# -DMT_PARSING -DUSE_ITC
QMAKE_CXXFLAGS += -O2 -Wall -g -DMT_PARSING
#TRANSLATIONS = ViTE_fr.ts
#LIBS += -lGLEW
LIBS += -lz -lrt
LIBS += -lz -lrt -lXt
macx {
QMAKESPEC = macx-g++
......@@ -107,6 +107,7 @@ HEADERS += common/common.hpp \
parser/ParserEventPaje.hpp \
parser/ParserPaje.hpp \
parser/ParserVite.hpp \
#parser/ParserSplitted.hpp\
parser/mt_ParserEventPaje.hpp\
parser/mt_ParserPaje.hpp\
parser/mt_PajeFileManager.hpp\
......@@ -142,7 +143,9 @@ HEADERS += common/common.hpp \
trace/tree/BinaryTree.hpp \
trace/tree/Node.hpp \
trace/tree/Interval.hpp \
trace/IntervalOfContainer.hpp\
#trace/IntervalOfContainer.hpp\
#trace/SerializerWriter.hpp\
#trace/SerializerDispatcher.hpp\
# Statistics headers
statistics/Stats_window.hpp \
statistics/Statistic.hpp \
......@@ -195,7 +198,9 @@ SOURCES += common/Message.cpp \
trace/Container.cpp \
trace/tree/Interval.cpp \
trace/Trace.cpp \
trace/IntervalOfContainer.cpp\
#trace/IntervalOfContainer.cpp\
#trace/SerializerWriter.cpp\
#trace/SerializerDispatcher.cpp\
# Parser code files
parser/Parser.cpp \
parser/ParserFactory.cpp \
......@@ -206,6 +211,7 @@ SOURCES += common/Message.cpp \
parser/ParserEventPaje.cpp \
parser/ParserPaje.cpp \
parser/ParserVite.cpp \
#parser/ParserSplitted.cpp\
parser/mt_ParserEventPaje.cpp \
parser/mt_ParserPaje.cpp \
parser/mt_PajeFileManager.cpp \
......
This diff is collapsed.
......@@ -57,7 +57,10 @@ struct IntervalOfContainer;
* \class Container
* \brief Contains others containers or entities
*/
class Container {
friend class State;
public:
......@@ -90,6 +93,13 @@ private:
#ifdef USE_ITC
std::list<IntervalOfContainer* > _intervalsOfContainer;
/* QWaitCondition* _cond;