F5/unison
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

doc: Update test documentation

Browse Source 
Adding --only-fullness and examples-as-tests entries
This commit is contained in:
Gabriel Ferreira
2025-09-14 23:18:52 +02:00
parent 5ec1ce9556
commit 280ea6cada
1 changed files with 105 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files

119
doc/manual/source/test-framework.rst
View File

@@ -101,49 +101,50 @@ if you run ``test.py --help`` you should see a command summary like:
.. sourcecode:: text .. sourcecode:: text
Usage: test.py [options] usage: test.py [-h] [-b BUILDPATH] [-c CONSTRAIN] [-d] [-e EXAMPLE] [-u]
[-f {QUICK,EXTENSIVE,TAKES_FOREVER} | -of {QUICK,EXTENSIVE,TAKES_FOREVER}]
[-g] [-k] [-l] [-m] [-n] [-p PYEXAMPLE] [-r] [-s SUITE] [-t TEXT-FILE]
[-v] [--verbose-failed] [-w HTML-FILE] [-x XML-FILE] [--nocolor]
[--jobs PROCESS_LIMIT] [--rerun-failed]
Options: options:
-h, --help show this help message and exit -h, --help show this help message and exit
-b BUILDPATH, --buildpath=BUILDPATH -b, --buildpath BUILDPATH
specify the path where ns-3 was built (defaults to the specify the path where ns-3 was built (defaults to the build
build directory for the current variant) directory for the current variant)
-c KIND, --constrain=KIND -c, --constrain CONSTRAIN
constrain the test-runner by kind of test constrain the test-runner by kind of test
-e EXAMPLE, --example=EXAMPLE
specify a single example to run (no relative path is
needed)
-d, --duration print the duration of each test suite and example -d, --duration print the duration of each test suite and example
-e EXAMPLE, --example=EXAMPLE -e, --example EXAMPLE
specify a single example to run (no relative path is specify a single example to run (no relative path is needed)
needed) -u, --update-data If examples use reference data files, get them to re-generate
-u, --update-data If examples use reference data files, get them to re- them
generate them -f, --fullness {QUICK,EXTENSIVE,TAKES_FOREVER}
-f FULLNESS, --fullness=FULLNESS choose the duration of tests to run: QUICK, EXTENSIVE, or
choose the duration of tests to run: QUICK, EXTENSIVE, TAKES_FOREVER, where EXTENSIVE includes QUICK and TAKES_FOREVER
or TAKES_FOREVER, where EXTENSIVE includes QUICK and includes QUICK and EXTENSIVE (only QUICK tests are run by
TAKES_FOREVER includes QUICK and EXTENSIVE (only QUICK default)
tests are run by default) -of, --only-fullness {QUICK,EXTENSIVE,TAKES_FOREVER}
choose the duration of tests to run: QUICK, EXTENSIVE, or
TAKES_FOREVER (only tests marked with fullness will be executed)
-g, --grind run the test suites and examples using valgrind -g, --grind run the test suites and examples using valgrind
-k, --kinds print the kinds of tests available -k, --kinds print the kinds of tests available
-l, --list print the list of known tests -l, --list print the list of known tests
-m, --multiple report multiple failures from test suites and test -m, --multiple report multiple failures from test suites and test cases
cases -n, --no-build do not build before starting testing
-n, --nobuild do not run ns3 before starting testing -p, --pyexample PYEXAMPLE
-p PYEXAMPLE, --pyexample=PYEXAMPLE specify a single python example to run (with relative path)
specify a single python example to run (with relative -r, --retain retain all temporary files (which are normally deleted)
path) -s, --suite SUITE specify a single test suite to run
-r, --retain retain all temporary files (which are normally -t, --text TEXT-FILE write detailed test results into TEXT-FILE.txt
deleted)
-s TEST-SUITE, --suite=TEST-SUITE
specify a single test suite to run
-t TEXT-FILE, --text=TEXT-FILE
write detailed test results into TEXT-FILE.txt
-v, --verbose print progress and informational messages -v, --verbose print progress and informational messages
-w HTML-FILE, --web=HTML-FILE, --html=HTML-FILE --verbose-failed print progress and informational messages for failed jobs
-w, --web, --html HTML-FILE
write detailed test results into HTML-FILE.html write detailed test results into HTML-FILE.html
-x XML-FILE, --xml=XML-FILE -x, --xml XML-FILE write detailed test results into XML-FILE.xml
write detailed test results into XML-FILE.xml --nocolor do not use colors in the standard output
--jobs PROCESS_LIMIT limit number of worker threads
--rerun-failed rerun failed tests
If one specifies an optional output style, one can generate detailed descriptions If one specifies an optional output style, one can generate detailed descriptions
of the tests and status. Available styles are ``text`` and ``HTML``. of the tests and status. Available styles are ``text`` and ``HTML``.
@@ -266,10 +267,6 @@ will result in a list of the test suite being displayed, similar to
.. sourcecode:: text .. sourcecode:: text
Waf: Entering directory `/home/craigdo/repos/ns-3-allinone-test/ns-3-dev/build'
Waf: Leaving directory `/home/craigdo/repos/ns-3-allinone-test/ns-3-dev/build'
'build' finished successfully (0.939s)
Test Type Test Name Test Type Test Name
--------- --------- --------- ---------
performance many-uniform-random-variables-one-get-value-call performance many-uniform-random-variables-one-get-value-call
@@ -338,6 +335,40 @@ results in that single example being run.
PASS: Example examples/udp/udp-echo PASS: Example examples/udp/udp-echo
You can also execute all instances of declared examples using different parameters,
as defined in ``examples-to-run.py`` by including a star at the end of the example name.
::
$ ./test.py --example=wifi-phy-configuration*
results in that example being run with the following parameters:
.. sourcecode:: text
[1/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=0
[2/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=8
[3/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=10
[4/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=12
[5/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=1
[6/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=13
[7/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=4
[8/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=3
[9/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=14
[10/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=5
[11/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=15
[12/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=6
[13/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=7
[14/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=11
[15/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=2
[16/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=9
[17/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=16
[18/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=19
[19/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=18
[20/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=17
[21/21] PASS: Example src/wifi/examples/wifi-phy-configuration --testCase=20
21 of 21 tests passed (21 passed, 0 skipped, 0 failed, 0 crashed, 0 valgrind errors)
You can specify the directory where |ns3| was built using the You can specify the directory where |ns3| was built using the
``--buildpath`` option as follows. ``--buildpath`` option as follows.
@@ -459,6 +490,12 @@ Note that specifying EXTENSIVE fullness will also run tests in QUICK category.
Specifying TAKES_FOREVER will run tests in EXTENSIVE and QUICK categories. Specifying TAKES_FOREVER will run tests in EXTENSIVE and QUICK categories.
By default, only QUICK tests are ran. By default, only QUICK tests are ran.
Alternatively, one can specify only-fullness option, which runs tests
from just that particular category. This allows for tiered execution, both in
time constrained jobs (starting from TAKES_FOREVER to give them more time,
then EXTENSIVE, and then QUICK), or to progressively run more expensive tests
(QUICK, then EXTENSIVE, then TAKES_FOREVER).
As a rule of thumb, tests that must be run to ensure |ns3| coherence should be As a rule of thumb, tests that must be run to ensure |ns3| coherence should be
QUICK (i.e., take a few seconds). Tests that could be skipped, but are nice to do QUICK (i.e., take a few seconds). Tests that could be skipped, but are nice to do
can be EXTENSIVE; these are tests that typically need minutes. TAKES_FOREVER is can be EXTENSIVE; these are tests that typically need minutes. TAKES_FOREVER is
@@ -565,7 +602,7 @@ You should see something like the following
.. sourcecode:: text .. sourcecode:: text
Usage: /home/craigdo/repos/ns-3-allinone-test/ns-3-dev/build/utils/ns3-dev-test-runner-debug [OPTIONS] Usage: /mnt/c/tools/sources/ns-3-dev/build/utils/ns3-dev-test-runner-debug [OPTIONS]
Options: Options:
--help : print these options --help : print these options
@@ -587,6 +624,9 @@ You should see something like the following
includes QUICK and TAKES_FOREVER includes includes QUICK and TAKES_FOREVER includes
QUICK and EXTENSIVE (only QUICK tests are QUICK and EXTENSIVE (only QUICK tests are
run by default) run by default)
--only-fullness=FULLNESS : choose the duration of tests to run: QUICK,
EXTENSIVE, TAKES_FOREVER (only tests marked
with fullness will be executed)
--verbose : print details of test execution --verbose : print details of test execution
--xml : format test run output as xml --xml : format test run output as xml
--tempdir=DIR : set temp dir for tests to store output files --tempdir=DIR : set temp dir for tests to store output files
@@ -742,6 +782,7 @@ of functionality. As described above, test suites can be classified as,
* Unit Tests * Unit Tests
* System Tests * System Tests
* Examples * Examples
* Examples-as-Tests
* Performance Tests * Performance Tests
This classification is exported from the TestSuite class. This class is quite This classification is exported from the TestSuite class. This class is quite