Commit c79a6ba0 authored by Johnny Jazeix's avatar Johnny Jazeix

Add the otf library. Modification of the interface console to take care

of 
otf traces. Modification of the parsers to have the otf one, beginning 
of it (containers are printed), adding a new statistics on counters.

New macros in the main_ressource.hpp for the ViTE version (major, minor, 
and a string one).

Modification of the configure script to handle all the changes, you 
should have to restart the script to be able to compile (make distclean 
&& ./configure && make should work).

parent d919016d
......@@ -28,7 +28,7 @@
# knowledge of the CeCILL-A license and that you accept its terms.
#
#
# ViTE developpers are (for version 0.* to 1.0):
# ViTE developers are (for version 0.* to 1.0):
#
# - COULOMB Kevin
# - FAVERGE Mathieu
......@@ -64,6 +64,7 @@ if [ -z "$MAKE" ]; then
fi
fi
OTFLIB_DIR=./src/general/otf
# Creates the makefile
cat > "$makefile" <<EOF
......@@ -75,7 +76,10 @@ cat > "$makefile" <<EOF
## ##
########################################################################
OTFLIB_DIR = ./src/general/otf
all:
make -C \$(OTFLIB_DIR)
EOF
if "$WHICH" qmake-qt4 >/dev/null 2>&1 ; then
......@@ -117,6 +121,7 @@ doc:
@echo " "
clean:
make -C \$(OTFLIB_DIR) clean
cd ./src && make clean
\$(RM) log.txt
@echo " "
......@@ -131,18 +136,15 @@ cleanall: clean
@(cd \$(DOCUMENTATION) && rm -rf html/ latex/)
distclean: cleanall
make -C \$(OTFLIB_DIR) distclean
\$(RM) Makefile
\$(RM) src/Makefile
#To remember how to create automaticaly a src.pro
#build:
# cd ./src && qmake-qt4 -project "RESOURCES= vite.qrc" "OBJECTS_DIR=../bin" "DESTDIR=../bin" "CONFIG+=uitools" "QT+=opengl" "LIBS += -lglut" && qmake-qt4 -makefile -o Makefile src.pro && make
# @echo " "
# @echo "Built and compiled! (release)"
# @echo " "
EOF
echo >&2 "Configuration of the otf library"
cd $OTFLIB_DIR && ./configure
# basename erase the path of acces of the command
MAKE=`basename $MAKE`
......
Andreas Knuepfer <andreas.knuepfer AT tu-dresden.de>
Ronny Brendel <ronny.brendel AT tu-dresden.de>
Matthias Jurenz <matthias.jurenz AT tu-dresden.de>
Johannes Spazier <johannes.spazier AT tu-dresden.de>
Michael Kluge <michael.kluge AT tu-dresden.de>
Holger Mickler <holger.mickler AT tu-dresden.de>
Holger Brunst <holger.brunst AT tu-dresden.de>
Hartmut Mix <hartmut.mix AT tu-dresden.de>
Copyright (c) 2005-2009, ZIH, Technische Universitaet Dresden,
Federal Republic of Germany
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
- Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
- Neither the name of ZIH, TU Dresden nor the names of its contributors
may be used to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
1.0
- initial version
1.1.1
- OTF_Reader now considers the return values of the handlers
- added OTF_VERBOSE, OTF_DEBUG macro for error treatment
- introduced 'UnknownRecord' handler which allows to catch
unknown record types
1.1.2
- inverted return value of call-back handlers:
'0' is non-error, '!= 0' is regarded as an error, now!
(this makes OTF conform with the VTF3 scheme.)
1.1.3
- fixed a minor bug in otfaux
1.1.4
- fixed a bug in OTF_Reader which might have caused the very first
time stamp of a trace to be not properly sorted
- introduced '--snapshots' and '--statistics' switches to do only
snapshots or statistics. for statistics a selective mode is allowed
which regards only some streams. By this means statistics can be created
in parallel by calling otfaux multiple times.
1.1.5
- have UnknownRecord report handle incomplete records or additional bytes at
the end of a file.
1.2
- introduce transparent zlib compression
1.2.1
- added progress functions using read bytes instead of timestamps
1.2.2
- important bugfix: definitionstream 0 was ignored since version 1.2
1.2.3
- bugfix: provided copy handlers returned wrong value
1.2.4
- bugfix: zlib compression bug, wrong sanity check fixed
1.2.5
- bugfix: correctly handle process groups with more than 1000 entries
1.2.6
- support shared libraries
1.2.7
- added progress functions to OTF_RStream
- added a progress counter to otfmerge
1.2.8
- allow suffix '.dylib' for zlib library file (from Mac OS X)
- removed configure warning
1.2.9
- changeable zlevel
- changeable zbuffersize
1.2.10
- bugfix: otfmerge does no longer accept traces with local streams
1.2.11
- changed OTF_RETURN*( 0=success, !0 = error )
- added these macros to all internal functions and tools for better
consistency
- fixed various memoryleaks in otf and otfmerge
- added otfconfig to tools. otfconfig shows installationparameters
important for developers
- updated documentation
1.2.12
- removed intel compiler warnings in otfmerge
- removed debug output
- fixed 64bit issue
1.2.13
- removed intel compiler warnings
- changed OTF_FILETYPE_*-macros
- fixed issues with OTF_getFilename()
1.2.14
- do not linke with '-lz' if '--with-zlibsymbols' was specified
- added zlib include line to otflib/Makefile.am
1.2.15
- 3 new records introduced (NOTE: these are experimental):
- DefFile
- DefFileGroup
- FileOperation
1.2.16
- fixed a problem with comments in otfmerge
- 2 new records introduced (NOTE: these are experimental):
- OpenFileSnapshot
- FileOperationSummary
1.2.17
- bugfixed parser (wrong treatment of unknown records)
- added FileGroupOperationSummary record
- changed FileOperation-, OpenFileSnapshot- and FileOperationSummary- record
NOTE: these records are still experimental
- included file operation records into all tools
NOTE: some parameters were added/changed
- the byte-parameter in the FileOperation record does now contain the
new position in the file file after the seek operation
- bugfixed otfmerge
1.2.18
- added fwrite check for writting less bytes than expected
- re-write of otfdump. Now, it uses OTF read lib instead of plain scanf.
It produces nicer output and numbers records in read order.
1.2.19
- finished otfdump
- removed debug output in OTF_Reader.c
- re-introduced DefVersion record as read only
1.2.20
- added own inttypes definitions in case there is no on the platform
(necessary on NEC SX6)
1.2.21
- fixed "--with-zlib-symbols"-bug
1.2.22
- fixed inttypes/stdint bug
1.2.23
- removed libtool from autoconf to make configure much faster,
libtool was not used by automake anyhow
- removed some autoconf-generated files from CVS,
you might need to re-run 'sh bootstrap' yourself
1.2.24
- minor change in otf[de|un]compress
- strictly avoid writing of unsorted time stamps. Now, there will be an
error message and the corresponding buffer/stream will be disabled for
further writing. The write routines provide an invlaid return code but
currently there is no way to "repair" the buffer/stream after an unsorted
time stamp has been issued.
1.3
- integrated libtool for shared and static linkage to otf
- removed the possibility to include zlib symbols into otf
- fixed warning which appeared when not using zlib
- added python bindings to the otf library
1.3.1
- if an fopen() call fails an error is now printed out every time
- OTF_READ_ERROR was introduced as an error return value for all
OTF_Reader_readXXX and OTF_RStream_readXXX functions. So (uint64_t)-1 is
now a reserved value, thus cannot be used as record limit.
The record limit always has to be smaller than or equal to
OTF_READ_MAXRECORDS.
- fixed issue in otfmerge, when using many streams
1.3.2
- added small functions to convert various variable types to common
counter type uint64_t. By this means, other types can be transported
as uint64_t and re-casted during reading.
This requires type information as additional counter property flags!
1.3.3
- fixed configure bug when searching for zlib on mac
- fixed bug in otfmerge, where the parameter "-n 0" caused an error
- added macros for specifing the data type of the counter
(OTF_COUNTER_VARTYPE_*)
- fixed otfconfig help text
- fixed bugs when using make dist
- python bindings disabled by default
- fixed configure bug, when using python bindings
- updated python bindings to support user defined counters
1.3.4
- bug fix in otfmerge: missing handling of DefCollectiveOperation
- bugfix in the documentation: overview.eps was broken
1.3.5
- added doxygen html documentation (docu/doxygen/html/index.html)
- fixed issue, where newlines were forbidden in def, stats and snaps
- added missing const to OTF_WStream_writeDefProcessGroup()
1.3.6
- OTF_{Keywords.h,Parse.h,Platform.h} will no longer be installed
- added a compile time assert making sure the correct sizes of the
integer types ([u]int*_T)
- OTF is now Windows Visual Studio compatible
1.3.7
- added example for using OTF_MasterControl to the doxygen documentation
- general improvement of the documentation
- improved autoconf's m4 scripts for cross buildings
1.3.8
- generate OTF_inttypes.h during configuring which provides
the integer types like int64_t, int8_t,...
- shared OTF library will be linked with zlib, thus additionally
linking with zlib is not necessary
- bugfix in otf2vtf: correctly handle function leave records with id 0
1.3.9
- fixed a portability bug: use own implementation of function
'strdup()'
- install a symbolic link 'otfdecompress' which is a synonym for
'otfcompress -d'
1.3.10
- added configure option '--[enable|disable]-binaries' to control
building of binaries (useful for multi-lib installations)
1.4
- added tool 'otfprofile' which can be used to generate CSV/Latex-
summaries about functions and P2P-messages of a trace
- build/install tool 'otfdump' by default
- install documentation
1.4.1
- bugfix: removed included system headers inside 'extern "C" {}'
- bugfix: added configure check for functions 'fseeko()' and 'ftello()'
(these functions are not available on NEC SX platforms)
- bugfix in otfprofile: fixed infinite loop while drawing the
statistics for "top 50 of counters"
1.5
- added file operation begin/end records
- added file operation types OTF_FILEOP_UNLINK, OTF_FILEOP_RENAME,
and OTF_FILEOP_OTHER
- use part of fileop for I/O flags (e.g. async, coll.)
- added collective operation begin/end records
- added remote memory access (RMA) operation records
- added marker records
1.5.1dolphin
- corrected package preparation for libtool
- build shared OTF library by default
- added version information to shared OTF library
1.5.2dolphin
- adapted python bindings to the current API (related to new record
types)
- if possible, otfprofile creates a PS-file
- fixed VPATH issue, when using python bindings
- fixed infinite recursion in python wrappers
pyOTF_COUNTER_VARTYPE_ISSIGNED and pyOTF_COUNTER_VARTYPE_ISUNSIGNED
1.6stingray
- added tool 'otfshrink' which can be used to reduce a trace to
specified processes
- introduced an error variable (otf_errno, otf_strerr) that indicates
whether an error occurred or not
- fixed build dependencies of otfprofile
- added file operation types OTF_FILEOP_DUP, OTF_FILEOP_SYNC,
OTF_FILEOP_LOCK, and OTF_FILEOP_UNLOCK
- changed datatype for sent/received data volume of collective
operation begin/end records from 'uint32_t' to 'uint64_t'
- replaced usage of PATH_MAX by VT_PATH_MAX to avoid compile errors
This diff is collapsed.
ACLOCAL_AMFLAGS = -I config/m4
if AMHAVEPYTHONSWIG
OTFLIB_PY = otflib_py
else
OTFLIB_PY =
endif
SUBDIRS = \
otflib \
$(OTFLIB_PY) \
tools \
docu
EXTRA_DIST= \
otf_vc08.sln \
config/otf_get_version.sh \
tests/hello/Makefile \
tests/hello/hello_otf.c \
tests/otf_python/groups.py \
tests/otf_python/test_read.py \
tests/otf_python/test_read_qt.py \
tests/otf_python/test_write.py \
VERSION
This diff is collapsed.
Open Trace Format (OTF)
Detailed program analysis of massively parallel programs requires the recording of event based performance data during run-time and its visualisation with appropriate software tools like the Vampir framework.
The specification of a convenient and powerful trace file format has to fulfill a large number of requirements. It must allow an efficient collection of the event based performance data on a parallel program environment. On the other hand it has to provide a very fast and convenient access to the large amount of performance data and the corresponding event definitions by the analysis software tools.
To improve scalability for very large and massively parallel traces the Open Trace Format (OTF) has been developed at ZIH as a successor format to the Vampir Trace Format (VTF3).
The Open Trace Format makes use of a portable ASCII encoding. It distributes single traces to multiple so called streams with one or more files each. Merging of records from multiple files is done transparently by the OTF library. The number of possible streams is not limited by the number of available file handles.
The provided read/write library should be used as a convenient interface for third party software. The library supports efficient parallel and distributed access to trace data and offers selective reading access regarding arbitrary time intervals, process selection and record types.
Optional auxiliary information can assist this selective access.
The software package contains additional tools for trace date conversion or preparation.
More information can be found in the provided documentation.
OTF version is available as open source software under the BSD license, see 'COPYING'. The software comes as a source code package. It can be installed by using autoconf procedures, see 'INSTALL'.
- re-design OTF's error/success return values for handlers and functions
- ideas for future development
- add memory allocation/deallocation record types, how?
- have more semantics
- have less semantics
- add general purpose record types like (type/owner,key,value) with no semantics,
-- can be used freely, ignored if unknown,
-- key/value are either uint64_t, array of uint64_t or text
- add record to write wall clock time or epoch or year:month:day:hour:minute:second
with this, VT can record time at start and end of trace (request by Robert Henschel)
- create a (graphical) tool for changing definition records:
- grouping functions
- ...
- solve the problem that when only reading statistic records one cannot
determine the begin and end of a process !!!!!
You might introduce a new record for this. ( maybe like begin/endprocess
for events? )
You could also write a statistics comment record when the first event appears,
(in otfaux)
- add new statistics record type that gives the caller-callee relation
- only write this at the end of a trace run
- use asynchronous write (aio_write) and maybe even read for performance
# This is the VERSION file for OTF, describing the precise version of OTF in
# this distribution. The various components of the version number below are
# combined to form a single version number string.
# major, minor, and sub are generally combined in the form
# <major>.<minor>.<sub>. If sub is zero, then it is omitted.
major=1
minor=6
sub=0
# string is used for alpha, beta, or release tags. If it is non-empty, it will
# be appended to the version number.
#
# history of release tags:
# 0.2.* octopussy
# 0.3.*-1.1.* starfish
# 1.2.* pufferfish
# 1.3.* jellyfish
# 1.4.* shark
# 1.5.* dolphin
# 1.6.* stingray
#
string=stingray
# library is used for the library versioning. These three numbers will be added
# to the name of the library (e.g. libotf.so.1.2.3).
# The meaning of these numbers is as follows (from left to right):
#
# current The number of the current API exported by the library.
# That must be incremented, if the API has changed.
# revision The implementation number of the most recent API
# exported by the library. A value of '0' means that this
# is the first implementation of the API.
# If any of the source for the library has changed,
# revision must be incremented.
# age The number of previous additional APIs supported by the
# library. If age is '2', the library can be linked into
# executables built with a release of the library that
# exported the current API number (current), or any of
# the preceding two APIs.
# If the new API is backward-compatible with the preceding
# release, age must be incremented. Otherwise, reset age
# to '0'.
library=0:1:0
m4_include(config/m4/acinclude.debug.m4)
m4_include(config/m4/acinclude.math.m4)
m4_include(config/m4/acinclude.omp.m4)
m4_include(config/m4/acinclude.pydev.m4)
m4_include(config/m4/acinclude.swig.m4)
m4_include(config/m4/acinclude.swig_python.m4)
m4_include(config/m4/acinclude.vtf3.m4)
m4_include(config/m4/acinclude.verbose.m4)
m4_include(config/m4/acinclude.zlib.m4)
m4_include(config/m4/acarchive/ax_openmp.m4)
This diff is collapsed.
/* config.h.in. Generated from configure.in by autoheader. */
/* Define to 1 if you have the `asprintf' function. */
#undef HAVE_ASPRINTF
/* Define to 1 if you have the <dlfcn.h> header file. */
#undef HAVE_DLFCN_H
/* Define to 1 if you have the `fseeko' function. */
#undef HAVE_FSEEKO
/* Define to 1 if you have the `ftello' function. */
#undef HAVE_FTELLO
/* Define to 1 if you have the <inttypes.h> header file. */
#undef HAVE_INTTYPES_H
/* Define to 1 if you have the <memory.h> header file. */
#undef HAVE_MEMORY_H
/* Define to 1 if you have the `snprintf' function. */
#undef HAVE_SNPRINTF
/* Define to 1 if you have the <stddef.h> header file. */
#undef HAVE_STDDEF_H
/* Define to 1 if you have the <stdint.h> header file. */
#undef HAVE_STDINT_H
/* Define to 1 if you have the <stdlib.h> header file. */
#undef HAVE_STDLIB_H
/* Define to 1 if you have the `strdup' function. */
#undef HAVE_STRDUP
/* Define to 1 if you have the <strings.h> header file. */
#undef HAVE_STRINGS_H
/* Define to 1 if you have the <string.h> header file. */
#undef HAVE_STRING_H
/* Define to 1 if you have the <sys/param.h> header file. */
#undef HAVE_SYS_PARAM_H
/* Define to 1 if you have the <sys/stat.h> header file. */
#undef HAVE_SYS_STAT_H
/* Define to 1 if you have the <sys/time.h> header file. */
#undef HAVE_SYS_TIME_H
/* Define to 1 if you have the <sys/types.h> header file. */
#undef HAVE_SYS_TYPES_H
/* Define to 1 if you have the `__va_copy' function. */
#undef HAVE_UNDERSCORE_VA_COPY
/* Define to 1 if you have the <unistd.h> header file. */
#undef HAVE_UNISTD_H
/* Define to 1 if you have the `vasprintf' function. */
#undef HAVE_VASPRINTF
/* Define to 1 if you have the `va_copy' function. */
#undef HAVE_VA_COPY
/* Define to 1 if you have the `vsnprintf' function. */
#undef HAVE_VSNPRINTF
/* */
#undef HAVE_VTF3
/* */
#undef HAVE_ZLIB
/* Define to the sub-directory in which libtool stores uninstalled libraries.
*/
#undef LT_OBJDIR
/* */
#undef OTF_DEBUG
/* */
#undef OTF_VERBOSE
/* Name of package */
#undef PACKAGE
/* Define to the address where bug reports for this package should be sent. */
#undef PACKAGE_BUGREPORT
/* Define to the full name of this package. */
#undef PACKAGE_NAME
/* Define to the full name and version of this package. */
#undef PACKAGE_STRING
/* Define to the one symbol short name of this package. */
#undef PACKAGE_TARNAME
/* Define to the version of this package. */
#undef PACKAGE_VERSION
/* The size of `long', as computed by sizeof. */
#undef SIZEOF_LONG
/* Define to 1 if you have the ANSI C header files. */
#undef STDC_HEADERS
/* Version number of package */
#undef VERSION
/* Define to `unsigned int' if <sys/types.h> does not define. */
#undef size_t
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
dnl @synopsis AX_OPENMP([ACTION-IF-FOUND[, ACTION-IF-NOT-FOUND]])
dnl
dnl @summary determine how to compile programs using OpenMP
dnl
dnl This macro tries to find out how to compile programs that use
dnl OpenMP a standard API and set of compiler directives for parallel
dnl programming (see http://www-unix.mcs/)
dnl
dnl On success, it sets the
dnl OPENMP_CFLAGS/OPENMP_CXXFLAGS/OPENMP_F77FLAGS output variable to
dnl the flag (e.g. -omp) used both to compile *and* link OpenMP
dnl programs in the current language.
dnl
dnl NOTE: You are assumed to not only compile your program with these
dnl flags, but also link it with them as well.
dnl
dnl If you want to compile everything with OpenMP, you should set:
dnl
dnl CFLAGS="$CFLAGS $OPENMP_CFLAGS"
dnl #OR# CXXFLAGS="$CXXFLAGS $OPENMP_CXXFLAGS"
dnl #OR# FFLAGS="$FFLAGS $OPENMP_FFLAGS"
dnl
dnl (depending on the selected language).
dnl
dnl The user can override the default choice by setting the
dnl corresponding environment variable (e.g. OPENMP_CFLAGS).
dnl
dnl ACTION-IF-FOUND is a list of shell commands to run if an OpenMP
dnl flag is found, and ACTION-IF-NOT-FOUND is a list of commands to run
dnl it if it is not found. If ACTION-IF-FOUND is not specified, the
dnl default action will define HAVE_OPENMP.
dnl
dnl @category InstalledPackages
dnl @author Steven G. Johnson <stevenj@alum.mit.edu>
dnl @version 2006-01-24
dnl @license GPLWithACException
AC_DEFUN([AX_OPENMP], [
AC_PREREQ(2.59) dnl for _AC_LANG_PREFIX
AC_CACHE_CHECK([for OpenMP flag of _AC_LANG compiler], ax_cv_[]_AC_LANG_ABBREV[]_openmp, [save[]_AC_LANG_PREFIX[]FLAGS=$[]_AC_LANG_PREFIX[]FLAGS
ax_cv_[]_AC_LANG_ABBREV[]_openmp=unknown
# Flags to try: -fopenmp (gcc), -openmp (icc), -mp (SGI & PGI),
# -xopenmp (Sun), -omp (Tru64), -qsmp=omp (AIX),
# -Popenmp (NECSX), none
ax_openmp_flags="-fopenmp -openmp -mp -xopenmp -omp -qsmp=omp -Popenmp none"
if test "x$OPENMP_[]_AC_LANG_PREFIX[]FLAGS" != x; then
ax_openmp_flags="$OPENMP_[]_AC_LANG_PREFIX[]FLAGS $ax_openmp_flags"
fi
for ax_openmp_flag in $ax_openmp_flags; do