should issueshttps://gitlab.inria.fr/vidjil/should/-/issues2020-04-23T16:42:11+02:00https://gitlab.inria.fr/vidjil/should/-/issues/36Interleaving commands and tests2020-04-23T16:42:11+02:00Mathieu GiraudInterleaving commands and testsWhen interleaving commands and tests we (may?) want that the tests are performed only on the state reached so far and not after launching all the commands in the `should` file.
An example of what such a test could look like:
```
dir=$(m...When interleaving commands and tests we (may?) want that the tests are performed only on the state reached so far and not after launching all the commands in the `should` file.
An example of what such a test could look like:
```
dir=$(mktemp -d)
ls $dir
$ Check that directory is empty
0:.
touch $dir/file
ls -1 $dir
$ Directory should now contain file `file`
1:file
rm -f $dir/file
$ Directory should be empty again
0:.
rmdir $dir
```2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/33Release 2.0.02019-04-08T15:57:16+02:00Mathieu GiraudRelease 2.0.0The 1.0.0 release was in July 2018. We should release soon and complete `CHANGELOG`The 1.0.0 release was in July 2018. We should release soon and complete `CHANGELOG`2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/30Implement a NO_EXTRA2019-03-29T09:04:18+01:00Mathieu GiraudImplement a NO_EXTRAAs suggested by @mikael\-sAs suggested by @mikael\-s2.0.0Mathieu GiraudMathieu Giraudhttps://gitlab.inria.fr/vidjil/should/-/issues/28--only-f / --only-expected-to-fail2019-03-29T09:04:19+01:00Mathieu Giraud--only-f / --only-expected-to-failIt could be worth to have options launching only *some* tests.
For example
- `--only-f` / `--only-expected-to-fail`
- `--skip-f` / `--skip-expected-to-fail`
and possibly other filters if we identify other pertinent categories of tests.It could be worth to have options launching only *some* tests.
For example
- `--only-f` / `--only-expected-to-fail`
- `--skip-f` / `--skip-expected-to-fail`
and possibly other filters if we identify other pertinent categories of tests.2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/27Allow tests te be failed ?2022-01-26T00:27:36+01:00Mathieu GiraudAllow tests te be failed ?We say that a test with `f` is *expected* to be failed, and we are thinking on should#20.
But sometimes we would like to *allow* a test to fail. It's not expected, but under some circonstances, it may fail. In this case, we do not want ...We say that a test with `f` is *expected* to be failed, and we are thinking on should#20.
But sometimes we would like to *allow* a test to fail. It's not expected, but under some circonstances, it may fail. In this case, we do not want to label `f` if should#20 is resolved, we do not want to break a ~"dev\-ci" pipeline... (but the user may want to be informed, it could be displayed at the end of the pipeline, like the current situation for `TODO-but-ok` before should#20.)
Do we need a new letter for these tests ?2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/25Timeout doesn't kill the command2019-03-26T16:34:07+01:00Mathieu GiraudTimeout doesn't kill the commandWhen a test times out, the test isn't killed. This can be an issue as the test continues consuming resources, which may overload the test server.When a test times out, the test isn't killed. This can be an issue as the test continues consuming resources, which may overload the test server.2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/23should.py doesn't timeout with launcher2019-03-26T16:34:07+01:00Mathieu Giraudshould.py doesn't timeout with launcherOn current version of Vidjil just launch:
`python3 should.py --launcher 'valgrind --tool=memcheck --leak-check=full --show-reachable=yes --trace-children=yes' should-get-tests/stanford-germlines.should-get`
Valgrind produces lots of ou...On current version of Vidjil just launch:
`python3 should.py --launcher 'valgrind --tool=memcheck --leak-check=full --show-reachable=yes --trace-children=yes' should-get-tests/stanford-germlines.should-get`
Valgrind produces lots of output (see #19) and `should.py` hangs.2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/21--var FOO=BAR should override a --var from should.cfg2019-03-26T16:34:07+01:00Mathieu Giraud--var FOO=BAR should override a --var from should.cfg2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/20Fail tests with TODO-but-ok2019-04-01T08:15:58+02:00Mathieu GiraudFail tests with TODO-but-okI think that by default `should.py` should fail when having `TODO-but-ok` tests. We need to know when a test doesn't fail anymore. With automatic testing there is no chance that we notice this unless `should.py` complains.I think that by default `should.py` should fail when having `TODO-but-ok` tests. We need to know when a test doesn't fail anymore. With automatic testing there is no chance that we notice this unless `should.py` complains.2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/19Timeout with large outputs2019-03-26T16:34:07+01:00Mathieu GiraudTimeout with large outputsWhen the output (either on stdout or stderr) is quite large the test systematically timeouts.
Here is a simple example:
```
# Displays 65,537 random ACGT
cat /dev/urandom | tr -dc ACGT | head -c 65537
rwl65537: [ACGT]
```
There is no...When the output (either on stdout or stderr) is quite large the test systematically timeouts.
Here is a simple example:
```
# Displays 65,537 random ACGT
cat /dev/urandom | tr -dc ACGT | head -c 65537
rwl65537: [ACGT]
```
There is no timeout for 65,536 characters (hmm… this value reminds me of something…)2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/4--timeout2019-03-26T16:34:07+01:00Mathieu Giraud--timeoutThe timeout should be configureable.The timeout should be configureable.2.0.0