should issueshttps://gitlab.inria.fr/vidjil/should/-/issues2020-06-01T17:58:08+02:00https://gitlab.inria.fr/vidjil/should/-/issues/34Hierarchical tests / sub-tests2020-06-01T17:58:08+02:00Mathieu GiraudHierarchical tests / sub-testsIdea by @mikael\-s :
> I have `3: foo` and I have to further test something in these matching lines :
```
3: foo
2:: foo bar
r:: foo .*z
::: barz
```Idea by @mikael\-s :
> I have `3: foo` and I have to further test something in these matching lines :
```
3: foo
2:: foo bar
r:: foo .*z
::: barz
```https://gitlab.inria.fr/vidjil/should/-/issues/33Release 2.0.02019-04-08T15:57:16+02:00Mathieu GiraudRelease 2.0.0The 1.0.0 release was in July 2018. We should release soon and complete `CHANGELOG`The 1.0.0 release was in July 2018. We should release soon and complete `CHANGELOG`2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/32Kubernetes ?2019-03-06T08:37:47+01:00Mathieu GiraudKubernetes ?Évoqué, à creuser.Évoqué, à creuser.https://gitlab.inria.fr/vidjil/should/-/issues/31Allows to check that EXIT_CODE is not 02022-01-25T22:10:11+01:00Mathieu GiraudAllows to check that EXIT_CODE is not 0There should be a way to test whether the program failed, without giving the precise exit code.There should be a way to test whether the program failed, without giving the precise exit code.https://gitlab.inria.fr/vidjil/should/-/issues/30Implement a NO_EXTRA2019-03-29T09:04:18+01:00Mathieu GiraudImplement a NO_EXTRAAs suggested by @mikael\-sAs suggested by @mikael\-s2.0.0Mathieu GiraudMathieu Giraudhttps://gitlab.inria.fr/vidjil/should/-/issues/29Detect when the input is not a .should file2019-02-27T21:50:25+01:00Mathieu GiraudDetect when the input is not a .should fileWhen the input does not contain any test line (`^[modifier]*:`), `should.py` should .
exit and warn the user.When the input does not contain any test line (`^[modifier]*:`), `should.py` should .
exit and warn the user.https://gitlab.inria.fr/vidjil/should/-/issues/28--only-f / --only-expected-to-fail2019-03-29T09:04:19+01:00Mathieu Giraud--only-f / --only-expected-to-failIt could be worth to have options launching only *some* tests.
For example
- `--only-f` / `--only-expected-to-fail`
- `--skip-f` / `--skip-expected-to-fail`
and possibly other filters if we identify other pertinent categories of tests.It could be worth to have options launching only *some* tests.
For example
- `--only-f` / `--only-expected-to-fail`
- `--skip-f` / `--skip-expected-to-fail`
and possibly other filters if we identify other pertinent categories of tests.2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/27Allow tests te be failed ?2022-01-26T00:27:36+01:00Mathieu GiraudAllow tests te be failed ?We say that a test with `f` is *expected* to be failed, and we are thinking on should#20.
But sometimes we would like to *allow* a test to fail. It's not expected, but under some circonstances, it may fail. In this case, we do not want ...We say that a test with `f` is *expected* to be failed, and we are thinking on should#20.
But sometimes we would like to *allow* a test to fail. It's not expected, but under some circonstances, it may fail. In this case, we do not want to label `f` if should#20 is resolved, we do not want to break a ~"dev\-ci" pipeline... (but the user may want to be informed, it could be displayed at the end of the pipeline, like the current situation for `TODO-but-ok` before should#20.)
Do we need a new letter for these tests ?2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/25Timeout doesn't kill the command2019-03-26T16:34:07+01:00Mathieu GiraudTimeout doesn't kill the commandWhen a test times out, the test isn't killed. This can be an issue as the test continues consuming resources, which may overload the test server.When a test times out, the test isn't killed. This can be an issue as the test continues consuming resources, which may overload the test server.2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/24Incomplete output2019-04-01T10:46:45+02:00Mathieu GiraudIncomplete outputFollowing fd4e2721, on some architectures (depending on Python version?) the output is incomplete.
In `vidjil`, this is the case for `vidjil-h-examples.should-get` or `stanford-k14.should-get` where the log outputs are not complete on s...Following fd4e2721, on some architectures (depending on Python version?) the output is incomplete.
In `vidjil`, this is the case for `vidjil-h-examples.should-get` or `stanford-k14.should-get` where the log outputs are not complete on some slaves (CentOS 6.3 amd64, Fedora 25 amd64, FreeBSD). Moreover those tests still timeout on those architectures.
Would a flush of the output be required?https://gitlab.inria.fr/vidjil/should/-/issues/23should.py doesn't timeout with launcher2019-03-26T16:34:07+01:00Mathieu Giraudshould.py doesn't timeout with launcherOn current version of Vidjil just launch:
`python3 should.py --launcher 'valgrind --tool=memcheck --leak-check=full --show-reachable=yes --trace-children=yes' should-get-tests/stanford-germlines.should-get`
Valgrind produces lots of ou...On current version of Vidjil just launch:
`python3 should.py --launcher 'valgrind --tool=memcheck --leak-check=full --show-reachable=yes --trace-children=yes' should-get-tests/stanford-germlines.should-get`
Valgrind produces lots of output (see #19) and `should.py` hangs.2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/22Add a --requires option2018-08-07T13:06:23+02:00Mathieu GiraudAdd a --requires optionThis option should trigger the same behaviour than the `!REQUIRES` directive.This option should trigger the same behaviour than the `!REQUIRES` directive.https://gitlab.inria.fr/vidjil/should/-/issues/21--var FOO=BAR should override a --var from should.cfg2019-03-26T16:34:07+01:00Mathieu Giraud--var FOO=BAR should override a --var from should.cfg2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/20Fail tests with TODO-but-ok2019-04-01T08:15:58+02:00Mathieu GiraudFail tests with TODO-but-okI think that by default `should.py` should fail when having `TODO-but-ok` tests. We need to know when a test doesn't fail anymore. With automatic testing there is no chance that we notice this unless `should.py` complains.I think that by default `should.py` should fail when having `TODO-but-ok` tests. We need to know when a test doesn't fail anymore. With automatic testing there is no chance that we notice this unless `should.py` complains.2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/19Timeout with large outputs2019-03-26T16:34:07+01:00Mathieu GiraudTimeout with large outputsWhen the output (either on stdout or stderr) is quite large the test systematically timeouts.
Here is a simple example:
```
# Displays 65,537 random ACGT
cat /dev/urandom | tr -dc ACGT | head -c 65537
rwl65537: [ACGT]
```
There is no...When the output (either on stdout or stderr) is quite large the test systematically timeouts.
Here is a simple example:
```
# Displays 65,537 random ACGT
cat /dev/urandom | tr -dc ACGT | head -c 65537
rwl65537: [ACGT]
```
There is no timeout for 65,536 characters (hmm… this value reminds me of something…)2.0.0https://gitlab.inria.fr/vidjil/should/-/issues/18What shell is used for launching commands?2020-04-21T15:43:37+02:00Mathieu GiraudWhat shell is used for launching commands?Depending on what shell is used the tests won't be written in the same way. This should be documented
I guess this is `sh`. Is it? Could this be customisable?Depending on what shell is used the tests won't be written in the same way. This should be documented
I guess this is `sh`. Is it? Could this be customisable?https://gitlab.inria.fr/vidjil/should/-/issues/17Upcoming release2019-03-31T20:50:06+02:00Mathieu GiraudUpcoming releaseWe should release to ease the distribution of `should`.
I suggest to go with Semantic Versionning https://semver.org/
Being already used in oe released production software (https://vidjil.org),
I think that we should label the release a...We should release to ease the distribution of `should`.
I suggest to go with Semantic Versionning https://semver.org/
Being already used in oe released production software (https://vidjil.org),
I think that we should label the release as `1.0.0`.
See https://semver.org/#how-do-i-know-when-to-release-100
This would mean that we take care seriously of our "API" (in our case, this is both what is in `demo/` and the command-line options, we should be explicit on that), and thus that we pledge to avoid incompatible API changes in the `1.x.x` releases.https://gitlab.inria.fr/vidjil/should/-/issues/14Combination of several non-True/False status2019-04-19T19:09:29+02:00Mathieu GiraudCombination of several non-True/False statusHow should a `TestSuite` be tagged when the underlying `TestCase`s have various status ?
Now it is the status of the first non-`True`/`False` status.
It could be the smallest element relative to the `True > SKIP > TODO-but-ok > TODO > ...How should a `TestSuite` be tagged when the underlying `TestCase`s have various status ?
Now it is the status of the first non-`True`/`False` status.
It could be the smallest element relative to the `True > SKIP > TODO-but-ok > TODO > False` order.https://gitlab.inria.fr/vidjil/should/-/issues/13Add an argument to every tested command2018-07-06T10:40:02+02:00Mathieu GiraudAdd an argument to every tested command@mikael\-s suggested to have a mechanism to add an arbitray string, such as `--foo bar`, to every command that is tested.
A way to go could be to use a variable, and replace `cmd arg1 arg2` with `cmd $EXTRA arg1 arg2` (with a default e...@mikael\-s suggested to have a mechanism to add an arbitray string, such as `--foo bar`, to every command that is tested.
A way to go could be to use a variable, and replace `cmd arg1 arg2` with `cmd $EXTRA arg1 arg2` (with a default empty value for `$EXTRA`).
Perhaps this mechanism could behave like `LAUNCHER` : `$EXTRA` should not be put when it is already present.https://gitlab.inria.fr/vidjil/should/-/issues/12Better displaying stdout2018-07-06T10:40:02+02:00Mathieu GiraudBetter displaying stdoutWe now display the first 100 lines (`MAX_DUMP_LINES`) of stdout when a test is failing (or with `-vv`)
Several times, I did not find the relevant information there.
Should we rather:
- dump all lines ? (it can be huge...)
- dump th...We now display the first 100 lines (`MAX_DUMP_LINES`) of stdout when a test is failing (or with `-vv`)
Several times, I did not find the relevant information there.
Should we rather:
- dump all lines ? (it can be huge...)
- dump the the last 100 lines ?