Figure out which testcases tend to fail on which system (ubuntu / debian)
There are some test cases that tend to fail more often than others. Since we are facing strange behaviour especially regarding testing on debian we need to investigate a bit more which testcases tend to fail and in which configuration.
-
check if the ci api works in our gitlab instance (https://docs.gitlab.com/ee/api/jobs.html) [yes, see melissa/melissa#26] -
check if there are ready to use tools for the following purposes -
download the logs of the ci jobs beginning for jobs performed in 2021 and figure out which testcases failed in each testcase - create a dataset looking a bit like this:
job | commit | ubuntu/debian | failed testcase | configuration comment | date |
---|
- note that there could be more than one line per job if multiple testcases failed for this job (there was a time where we did not stop ci after the first testcase failed)
for the configuration comment we need to put which configuration (runner cores, server cores, runner amount) failed for test-different-paralelism
- I think for the other testcases this is not so interesting.)
I think this will be also useful in future if we have testcases that periodically fail. So would be cool if the analysis is fully automated so we can regenerate it if necessary.