2022-02-07 17:35:29 +01:00
|
|
|
# Testing Inferno
|
|
|
|
|
|
|
|
The inferno codebases uses different kinds of tests:
|
|
|
|
|
|
|
|
- We used to have server-only tests, but we don't have any right now. Those could go in `src/test`.
|
|
|
|
|
|
|
|
- The example client has tests in `client/test`:
|
|
|
|
+ [TestF.ml][] has small unit tests for type-checking of System F terms
|
|
|
|
+ [TestML.ml][] has small unit tests for ML programs
|
|
|
|
+ [TestMLRandom.ml][] performs property-based testing on a small fragment
|
|
|
|
of the client language (we don't know how to generate pattern-matching code randomly).
|
|
|
|
+ [suite.t/run.t][] uses dune [Cram tests][] for more unit tests
|
|
|
|
of ML programs, including failure scenarios.
|
|
|
|
|
|
|
|
[TestF.ml]: client/test/TestF.ml
|
|
|
|
[TestML.ml]: client/test/TestML.ml
|
|
|
|
[TestMLRandom.ml]: (client/test/TestMLRandom.ml)
|
|
|
|
[Cram tests]: https://dune.readthedocs.io/en/stable/tests.html#cram-tests
|
|
|
|
[suite.t]: client/tests/suite.t/run.t
|
|
|
|
[BenchMLRandom.ml]: client/test/BenchMLRandom.ml
|
|
|
|
|
|
|
|
Cram tests provide a readable way to specify test outcomes, and they
|
|
|
|
encourage us to develop a user-friendly binary frontend for the
|
|
|
|
client. Future ML unit tests should go to [suite.t][] rather than
|
|
|
|
[TestML.ml][]. To work with [suite.t][], we recommend reading the dune
|
|
|
|
documentation on [Cram tests][] first.
|
|
|
|
|
|
|
|
Finally, performance benchmarks are included in [BenchMLRandom.ml]. It
|
|
|
|
is a good idea to run them to evaluate changes that have a potential
|
|
|
|
for degrading performances.
|
|
|
|
|
|
|
|
## Running tests
|
|
|
|
|
|
|
|
Some tests are rather slow, so we have a "fast" and "slow" mode to run
|
|
|
|
the tests. The "fast" tests should be pleasant to use during
|
|
|
|
interactive edit-test loops, a few seconds at most. The "slow" tests
|
|
|
|
currently take about 30s on a typical machine.
|
|
|
|
|
|
|
|
You can run tests using either `make` or `dune`. Using the makefile:
|
|
|
|
|
|
|
|
```
|
|
|
|
make quick # only the fast tests
|
|
|
|
make test # all tests
|
|
|
|
```
|
|
|
|
|
|
|
|
Using dune:
|
|
|
|
|
|
|
|
```
|
|
|
|
INFERNO_SLOW_TESTS=0 dune runtest # only the fast tests
|
|
|
|
dune runtest # all tests
|
|
|
|
```
|
|
|
|
|
|
|
|
Note that `dune` runs tests incrementally: it will only re-run tests
|
|
|
|
that depend on code that has changed. If you want to run all tests
|
|
|
|
each time, use `dune runtest --force`. The Makefile rules pass
|
|
|
|
`--force` by default.
|
|
|
|
|
|
|
|
### Running individual tests
|
|
|
|
|
|
|
|
You can run each test individually. (You may want to pass `--force` if the code has not changed.)
|
|
|
|
|
|
|
|
```
|
|
|
|
dune exec client/test/TestF.exe
|
|
|
|
dune exec client/test/TestML.exe
|
|
|
|
dune exec client/test/TestMLRandom.exe # consider passing INFERNO_SLOW_TESTS=0
|
|
|
|
```
|
|
|
|
|
|
|
|
The dune invocation is a bit different from cram tests:
|
|
|
|
|
|
|
|
```
|
|
|
|
dune build @client/test/suite
|
|
|
|
```
|
|
|
|
|
|
|
|
Note that this command shows nothing if the tests pass. If you are not
|
|
|
|
sure it tests as expected, go break the [suite.t] testsuite and see
|
|
|
|
what happens :-)
|
|
|
|
|
|
|
|
### Promoting cram tests
|
|
|
|
|
|
|
|
[Cram tests] compare the current behavior with a pre-recorded reference
|
|
|
|
output. If the output of a test changes, but the change is
|
|
|
|
correct/expected, you can fix the test automatically by running
|
|
|
|
|
|
|
|
```
|
|
|
|
dune promote
|
|
|
|
```
|
|
|
|
|
|
|
|
which will update the reference output with the current behavior.
|
|
|
|
|
|
|
|
For more details, see the [dune documentation][Cram tests].
|
|
|
|
|
|
|
|
|
2022-02-07 17:47:45 +01:00
|
|
|
## Running the `midml` program
|
|
|
|
|
|
|
|
[midml.ML] is a very simple frontend program that parses, infers and
|
|
|
|
type-checks a source file in the "mid-ML" language supported by our
|
|
|
|
example client. You can run it to check small programs.
|
|
|
|
|
|
|
|
This program can be invoked using dune. You can refer to it using
|
|
|
|
either its internal path, `client/bin/midml.exe`, or its installation
|
|
|
|
path, `midml`:
|
|
|
|
|
|
|
|
```
|
|
|
|
echo "let id = fun x -> x" > foo.midml
|
|
|
|
dune exec -- client/bin/midml.exe foo.midml
|
|
|
|
dune exec -- midml foo.midml
|
|
|
|
```
|
|
|
|
|
2022-02-07 17:35:29 +01:00
|
|
|
## Running the benchmark
|
|
|
|
|
|
|
|
```
|
2022-02-07 21:22:42 +01:00
|
|
|
make bench
|
2022-02-07 17:35:29 +01:00
|
|
|
```
|
|
|
|
|
|
|
|
will print performance statistics. If you do a performance-impacting
|
|
|
|
change, you are expected to run the benchmark before and after, and
|
|
|
|
compare the results by yourself. (Merge Requests to make this more
|
|
|
|
convenient are of course welcome.)
|