E

evaluation

This project contains the information related to the evaluation carried out during CRV 14.

Name Last Update
DataMillScripts Loading commit data...
SpreadSheet Loading commit data...
README.md Loading commit data...

About

This GitLab project hosts the evaluation scripts used during the First International Competition on Runtime Verification (CRV 14).

CRV 14 was held in September 2014, in Toronto, Canada, as a satellite event of the 14th international conference on Runtime Verification (RV’14).

Repository content

This repository contains:

  1. the scripts to be run on DataMill produced by the participants to assess their tools on the benchmarks of the competition (see the other GitLab project hosting benchmarks).
  2. a spreadsheet with the raw experimental data and which also implements score calculation.

Scripts to be run on DataMill

The scripts (see scripts directory) are organised in directories. There is one directory per track (Java, C, Offline). Each subdirectory contains subdirectories associated to participants. Each participant directory contains the scripts of the participants. There is one script for each benchmark the participant participated to.

Spreadsheet

The spreadsheet contains the raw experimental data that is, for each track, for each participant, for each benchmark the participant participated to:

  • the execution time obtained by the participant
  • the memory consumption obtained by the participant.

The speadsheet also contains calculation of scores as per the method described in the competition papers. Note, to obtain full details on the score calculation, one can unmask some columns of the spreadsheet.

Note, the speadsheet can also be downloaded here.

Note

As of March 13, 2017, we have been informed by the RiTHM team, participating to the C track, that they are unable to provide the competition scripts.