Here is [an example of report for pyfaust](https://faustgrp.gitlabpages.inria.fr/-/faust/-/jobs/3163959/artifacts/build_FaustLinuxPython/python_pyfaust_test_output.html)
(if this link doesn't work, the associated pipeline has been deleted on Gitlab,
please look [here](https://gitlab.inria.fr/faustgrp/faust/-/jobs) for a recent ``ctest_python``/``ctest_matlab`` output).
please look [here](https://gitlab.inria.fr/faustgrp/faust/-/jobs) for a recent ``ctest_python``/``ctest_matlab``
output).
- On the end of ``ctest`` CI job execution a test report in HTML for the C++ tests too.
It is automatically uploaded to the project Gitlab pages (as a hidden page).
The link is displayed in the CI job output (the finished jobs are listed
Here is [an example of report for C++ tests](https://faustgrp.gitlabpages.inria.fr/-/faust/-/jobs/3193880/artifacts/build_FaustLinux/cpp_test_report.html)
(if this link doesn't work, the associated pipeline has been deleted on Gitlab,
please look [here](https://gitlab.inria.fr/faustgrp/faust/-/jobs) for a recent ``ctest``
output).
At the time of writing a group of 167 C++ tests are ran by the ctest CI job. It includes
many unit tests and all these tests passed the latest pipeline.
- Alternative test reports are available here: [FAµST CDash](https://cdash-ci.inria.fr/index.php?project=faust),
they are uploaded by ctest/cdash (CDash configuration is available in ``CDashConfScript.cmake``
at the root of the project). It gives additional information about the
building part of tests (which is important in case of errors, that's where you might look up
to understand why a C++ test or a wrapper failed to build).
#### Doctest
The pyfaust/matfaust API comes with many examples integrated on the [documentation](https://faustgrp.gitlabpages.inria.fr/faust/last-doc/html/index.html).
They are all fully tested in the
``doctest_nightly_pyfaust``/``doctest_nightly_matfaust`` ci job (cf. ``gitlab-ci.yml``)
and comply with the job passing policy exposed in [1.](#gitlab_pol_test)(any doctest failure means the ci pipeline
fails). These ci jobs are [scheduled](https://gitlab.inria.fr/faustgrp/faust/-/pipeline_schedules)
to run every night.
If any of the tested pyfaust/matfaust submodules or functions fails (and they are all tested at the time of
writing), the erroneous tests will be displayed in the ci job output.
The pyfaust/matfaust API comes with many examples integrated in the [documentation](https://faustgrp.gitlabpages.inria.fr/faust/last-doc/html/index.html).
They are all fully tested in the ``doctest_nightly_pyfaust``/``doctest_nightly_matfaust``
ci job (cf. ``gitlab-ci.yml``) and comply with the job passing policy exposed in [1.](#gitlab_pol_test)
(any doctest failure means normally that the CI pipeline fails). These CI jobs are [scheduled](https://gitlab.inria.fr/faustgrp/faust/-/pipeline_schedules)
to run by night (cf. Gitlab schedule interface). If any of the tested pyfaust/matfaust submodules or functions
fails (and they are all tested at the time of writing), the erroneous tests will be
displayed in the CI job output.
#### Test Coverage
It really matters to keep a metric of how much code is covered by the tests in the project.
- The ci job ``pyfaust_test_code_coverage`` measures the test coverage for
pyfaust (using the same tests as ``ctest_python`` and
``doctest_nightly_pyfaust``). At the moment I'm writing these lines, the cover
is about 72%. The same ci job produces a report available on gitlab-pages
too as job's artifact (e.g.: [report](https://faustgrp.gitlabpages.inria.fr/-/faust/-/jobs/3169837/artifacts/htmlcov/index.html) -- note: this link won't last forever). The report is also available directly as text in the ci job output.
- The CI job ``pyfaust_test_code_coverage`` measures the test coverage for
pyfaust (using the same tests as ``ctest_python`` and
``doctest_nightly_pyfaust``). Look at the badge below to know what is the
current coverage of pyfaust tests. The same CI job produces a report available on gitlab-pages
too as job's artifact (e.g.: [report](https://faustgrp.gitlabpages.inria.fr/-/faust/-/jobs/3169837/artifacts/htmlcov/index.html)
-- note: this link won't last forever). The report is also available directly as text in the CI job output.
- For matfaust the ci job is ``matfaust_test_code_coverage`` it uses MOCov to
produce a coverage report as this [one](https://faustgrp.gitlabpages.inria.fr/-/faust/-/jobs/3186956/artifacts/coverage_html/index.html)
(this link might be deleted in the future, in which case you might look at artifacts of recent ci
[jobs](https://gitlab.inria.fr/faustgrp/faust/-/jobs)). The tests used for coverage calculation
are pretty much the same as the ones in ``ctest_matlab`` and ``doctest_nightly_matfaust``.
When writing this doc, the coverage was about 48.6%, it is partially due to the fact that the
code for mex float (single in matlab) is not auto-tested but there is no reason to think it
would fail more often than the code for double which is tested. However it would be advisable
to pursue for a larger coverage.
- About GPU code: unfortunately no automatic testing is made for GPU code
because no NVIDIA GPU is available on VMs/Docker daemons that run the project gitlab-runners.
- For matfaust the CI job is ``matfaust_test_code_coverage`` it uses MOCov to
produce a coverage report as this [one](https://faustgrp.gitlabpages.inria.fr/-/faust/-/jobs/3186956/artifacts/coverage_html/index.html)
(this link might be deleted in the future, in which case you might look at artifacts of recent ci
[jobs](https://gitlab.inria.fr/faustgrp/faust/-/jobs)). The tests used for coverage calculation
are pretty much the same as the ones in ``ctest_matlab`` and ``doctest_nightly_matfaust``.
Look at the badge below to know what is the current coverage of matfaust tests. If it is not so good
it is partially due to the fact that the code for mex float (single in matlab) is not auto-tested
but there is no reason to think it would fail more often than the code for double which is tested.
However it would be advisable to pursue for a larger coverage.
### FAµST C++ core tests
#### Tests
- C++ test coverage: the calculation of the coverage for C++ tests is
integrated directly in the ``ctest`` CI job (see the badge in the next to know
the coverage rate).
- Directory location: ``misc/test/src/C++`` and ``misc/test/src/C++/unit``
- About GPU code: GPU code is tested mainly in C++ tests but ``pyfaust.tests``
submodule provides the same tests on GPU and CPU (except for the CPU only API).
matfaust is not so well tested on GPU code, another reason for its poor coverage rate.
Not all VMs are capable to run GPU tests (cf. [README](README.md) for requirements), the gitlab-runners
must be tagged 'cuda' if they are capable of running the GPU code (this condition is
verified dynamically in the concerned CI jobs).
#### Gitlab continuous integration (ci) jobs
- Gitlab badges: Gitlab provides the badge feature for indicating clearly the test coverage rate. The
pyfaust/matfaust/C++ core coverage badges are displayed above and below.
- Gitlab ci job: ``ctest`` performs C++ tests.
- Note that for a matter of pipeline speed this ci job runs only if there are changes in ``src/**/*``or ``misc/test/src/C++/**/*``.
For the same reason another ci job named ``ctest_nightly`` is made to run the tests that are very slow/long. They are enabled through the cmake option
``SLOW_TESTS`` (which is defaultly set to ``OFF``). There is one job version for each supported OS (``ctest_nightly_linux``, ``ctest_nightly_macos``,
``ctest_nightly_win10``), for the moment the macos' is disabled mainly because the macos' runs in monothread (see ``.gitlab-ci.yml``).
- For more details about ctest, take a look at [2.2](#py_mat_test_ci_jobs).
### Package tests and automatic releases
#### Test report on Gitlab pages
Several CI jobs (defined in ``.gitlab-ci.yml``) are responsible to build and
generate different packages.
#### Jobs for revision packages
- On the end of ``ctest`` ci job execution a test report in HTML
is automatically uploaded to the project Gitlab pages (as a hidden page).
The link is displayed in the ci job output (the finished jobs are listed
Here is [an example of report for C++ tests](https://faustgrp.gitlabpages.inria.fr/-/faust/-/jobs/3193880/artifacts/build_FaustLinux/cpp_test_report.html)
(if this link doesn't work, the associated pipeline has been deleted on Gitlab,
please look [here](https://gitlab.inria.fr/faustgrp/faust/-/jobs) for a recent ``ctest`` output).
They are triggered on each push of commits. The goal of the packages is to work
on the project without the need to produce a new release. They are named with the
SHA256 prefix of the corresponding git-commit. The secondary interest is to
avoid to find out too late (at release time) that one package doesn't build.
- Jobs for system packages which embed both pyfaust and matfaust.
*``pkg_linux``: generates Linux packages \*.rpm (Fedora, Redhat, Centos, etc.) and
\*.deb Linux packages (Debian, Ubuntu, etc.).
*``pkg_win``: generates \*.exe Windows 10 package.
*``pkg_macos``: generates \*.pkg Mac OS X package.
- Jobs for wheel PIP packages for pyfaust only (one per system type):
*``pkg_linux_purepy_rev``,
*``pkg_win_purepy_rev``,
*``pkg_macos_purepy_rev``.
#### Jobs for release packages
They are triggered on each git-tag push. The goal is to produce a release of
all packages available of the tagged version. The same kind of packages as the
ones generated on revision/commit are generated with additionally anaconda packages.
The jobs are listed below. See description of equivalent jobs for revision packages.
- Jobs for system packages which embed both pyfaust and matfaust
*``pkg_linux_release``
*``pkg_linux_release_static``: does the same as the previous one except that
matio library is built statically in the wrappers (no need to install matio
separately on the system)
*``pkg_macos_release``
*``pkg_win_release``
- Jobs for wheel PIP packages for pyfaust only (one per system type):
*``pkg_linux_purepy_release``
*``pkg_macos_purepy_release``
*``pkg_win_purepy_release``
- Jobs for anaconda packages: ``conda_linux_pub``, ``conda_macosx_pub``,
``conda_win_pub`` that produce anaconda packages based on wheel packages.
- Jobs for extra feature packages.
* Another version of Python is used to build the wrappers in these jobs:
