Add documentation about tests
This commit is contained in:
Tom Henderson
@@ -77,6 +77,39 @@ Output such as below should be produced:
|
||||
|
||||
See src/lte/test/test-lte-antenna.cc for a worked example.
|
||||
|
||||
Test macros
|
||||
***********
|
||||
|
||||
There are a number of macros available for checking test program
|
||||
output with expected output. These macros are defined in
|
||||
``src/core/model/test.h``.
|
||||
|
||||
The main set of macros that are used include the following:
|
||||
|
||||
::
|
||||
|
||||
NS_TEST_ASSERT_MSG_EQ(actual, limit, msg)
|
||||
NS_TEST_ASSERT_MSG_NE(actual, limit, msg)
|
||||
NS_TEST_ASSERT_MSG_LT(actual, limit, msg)
|
||||
NS_TEST_ASSERT_MSG_GT(actual, limit, msg)
|
||||
NS_TEST_ASSERT_MSG_EQ_TOL(actual, limit, tol, msg)
|
||||
|
||||
The first argument ``actual`` is the value under test, the second value
|
||||
``limit`` is the expected value (or the value to test against), and the
|
||||
last argument ``msg`` is the error message to print out if the test fails.
|
||||
|
||||
The first four macros above test for equality, inequality, less than, or
|
||||
greater than, respectively. The fifth macro above tests for equality,
|
||||
but within a certain tolerance. This variant is useful when testing
|
||||
floating point numbers for equality against a limit, where you want to
|
||||
avoid a test failure due to rounding errors.
|
||||
|
||||
Finally, there are variants of the above where the keyword ``ASSERT``
|
||||
is replaced by ``EXPECT``. These variants are designed specially for
|
||||
use in methods (especially callbacks) returning void. Reserve their
|
||||
use for callbacks that you use in your test programs; otherwise,
|
||||
use the ``ASSERT`` variants.
|
||||
|
||||
How to add an example program to the test suite
|
||||
***********************************************
|
||||
|
||||
|
||||
@@ -165,6 +165,12 @@ to be specified. Running the command
|
||||
|
||||
./test.py --suite=ns3-wifi-propagation-loss-models
|
||||
|
||||
or equivalently
|
||||
|
||||
::
|
||||
|
||||
./test.py -s ns3-wifi-propagation-loss-models
|
||||
|
||||
results in that single test suite being run.
|
||||
|
||||
::
|
||||
@@ -456,11 +462,35 @@ Performance Tests
|
||||
Performance tests are those which exercise a particular part of the system
|
||||
and determine if the tests have executed to completion in a reasonable time.
|
||||
|
||||
As of ns-3.15, there are no performance tests that are formally part of
|
||||
the test framework (contributed tests would be helpful here).
|
||||
|
||||
Running Tests
|
||||
+++++++++++++
|
||||
*************
|
||||
|
||||
Tests are typically run using the high level ``test.py`` program. To get a list of the available command-line options, run ``test.py --help``
|
||||
|
||||
|
||||
The test program ``test.py`` will run both tests and those examples that
|
||||
have been added to the list to check. The difference between tests
|
||||
and examples is as follows. Tests generally check that specific simulation
|
||||
output or events conforms to expected behavior. In contrast, the output
|
||||
of examples is not checked, and the test program merely checks the exit
|
||||
status of the example program to make sure that it runs without error.
|
||||
|
||||
Briefly, to run all tests, first one must configure tests during configuration
|
||||
stage, and also (optionally) examples if examples are to be checked:
|
||||
|
||||
::
|
||||
|
||||
./waf --configure --enable-examples --enable-tests
|
||||
|
||||
Then, build ns-3, and after it is built, just run ``test.py``. ``test.py -h``
|
||||
will show a number of configuration options that modify the behavior
|
||||
of test.py.
|
||||
|
||||
The program ``test.py`` invokes, for C++ tests and examples, a lower-level
|
||||
C++ program called ``test-runner`` to actually run the tests. As discussed
|
||||
below, this ``test-runner`` can be a helpful way to debug tests.
|
||||
|
||||
Debugging Tests
|
||||
***************
|
||||
@@ -541,7 +571,10 @@ directory tree looking for a directory file with files named ``VERSION`` and
|
||||
``LICENSE.`` If it finds one, it assumes that must be the basedir and provides
|
||||
it for you.
|
||||
|
||||
Similarly, many test suites need to write temporary files (such as pcap files)
|
||||
Test output
|
||||
+++++++++++
|
||||
|
||||
Many test suites need to write temporary files (such as pcap files)
|
||||
in the process of running the tests. The tests then need a temporary directory
|
||||
to write to. The Python test utility (test.py) will provide a temporary file
|
||||
automatically, but if run stand-alone this temporary directory must be provided.
|
||||
@@ -563,6 +596,20 @@ The time is provided as a hint so that you can relatively easily reconstruct
|
||||
what directory was used if you need to go back and look at the files that were
|
||||
placed in that directory.
|
||||
|
||||
Another class of output is test output like pcap traces that are generated
|
||||
to compare to reference output. The test program will typically delete
|
||||
these after the test suites all run. To disable the deletion of test
|
||||
output, run ``test.py`` with the "retain" option:
|
||||
|
||||
::
|
||||
|
||||
./test.py -r
|
||||
|
||||
and test output can be found in the ``testpy-output/`` directory.
|
||||
|
||||
Reporting of test failures
|
||||
++++++++++++++++++++++++++
|
||||
|
||||
When you run a test suite using the test-runner it will run the test quietly
|
||||
by default. The only indication that you will get that the test passed is
|
||||
the *absence* of a message from ``waf`` saying that the program
|
||||
@@ -621,6 +668,41 @@ also not a complete XML file since test suites are designed to have their
|
||||
output appended to a master XML status file as described in the ``test.py``
|
||||
section.
|
||||
|
||||
Debugging test suite failures
|
||||
+++++++++++++++++++++++++++++
|
||||
|
||||
To debug test crashes, such as::
|
||||
|
||||
CRASH: TestSuite ns3-wifi-interference
|
||||
|
||||
You can access the underlying test-runner program via gdb as follows, and
|
||||
then pass the "--basedir=`pwd`" argument to run (you can also pass other
|
||||
arguments as needed, but basedir is the minimum needed)::
|
||||
|
||||
./waf --command-template="gdb %s" --run "test-runner"
|
||||
Waf: Entering directory `/home/tomh/hg/sep09/ns-3-allinone/ns-3-dev-678/build'
|
||||
Waf: Leaving directory `/home/tomh/hg/sep09/ns-3-allinone/ns-3-dev-678/build'
|
||||
'build' finished successfully (0.380s)
|
||||
GNU gdb 6.8-debian
|
||||
Copyright (C) 2008 Free Software Foundation, Inc.
|
||||
L cense GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
|
||||
This is free software: you are free to change and redistribute it.
|
||||
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
|
||||
and "show warranty" for details.
|
||||
This GDB was configured as "x86_64-linux-gnu"...
|
||||
(gdb) r --basedir=`pwd`
|
||||
Starting program: <..>/build/debug/utils/test-runner --basedir=`pwd`
|
||||
[Thread debugging using libthread_db enabled]
|
||||
assert failed. file=../src/core/model/type-id.cc, line=138, cond="uid <= m_information.size () && uid != 0"
|
||||
...
|
||||
|
||||
Here is another example of how to use valgrind to debug a memory problem
|
||||
such as::
|
||||
|
||||
VALGR: TestSuite devices-mesh-dot11s-regression
|
||||
|
||||
./waf --command-template="valgrind %s --basedir=`pwd` --suite=devices-mesh-dot11s-regression" --run test-runner
|
||||
|
||||
Class TestRunner
|
||||
****************
|
||||
|
||||
@@ -706,7 +788,7 @@ case a name and override the ``DoRun`` method to run the test.
|
||||
}
|
||||
|
||||
Utilities
|
||||
+++++++++
|
||||
*********
|
||||
|
||||
There are a number of utilities of various kinds that are also part of the
|
||||
testing framework. Examples include a generalized pcap file useful for
|
||||
@@ -714,37 +796,6 @@ storing test vectors; a generic container useful for transient storage of
|
||||
test vectors during test execution; and tools for generating presentations
|
||||
based on validation and verification testing results.
|
||||
|
||||
Debugging test suite failures
|
||||
+++++++++++++++++++++++++++++
|
||||
|
||||
To debug test crashes, such as::
|
||||
|
||||
CRASH: TestSuite ns3-wifi-interference
|
||||
|
||||
You can access the underlying test-runner program via gdb as follows, and
|
||||
then pass the "--basedir=`pwd`" argument to run (you can also pass other
|
||||
arguments as needed, but basedir is the minimum needed)::
|
||||
|
||||
./waf --command-template="gdb %s" --run "test-runner"
|
||||
Waf: Entering directory `/home/tomh/hg/sep09/ns-3-allinone/ns-3-dev-678/build'
|
||||
Waf: Leaving directory `/home/tomh/hg/sep09/ns-3-allinone/ns-3-dev-678/build'
|
||||
'build' finished successfully (0.380s)
|
||||
GNU gdb 6.8-debian
|
||||
Copyright (C) 2008 Free Software Foundation, Inc.
|
||||
L cense GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
|
||||
This is free software: you are free to change and redistribute it.
|
||||
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
|
||||
and "show warranty" for details.
|
||||
This GDB was configured as "x86_64-linux-gnu"...
|
||||
(gdb) r --basedir=`pwd`
|
||||
Starting program: <..>/build/debug/utils/test-runner --basedir=`pwd`
|
||||
[Thread debugging using libthread_db enabled]
|
||||
assert failed. file=../src/core/model/type-id.cc, line=138, cond="uid <= m_information.size () && uid != 0"
|
||||
...
|
||||
|
||||
Here is another example of how to use valgrind to debug a memory problem
|
||||
such as::
|
||||
|
||||
VALGR: TestSuite devices-mesh-dot11s-regression
|
||||
|
||||
./waf --command-template="valgrind %s --basedir=`pwd` --suite=devices-mesh-dot11s-regression" --run test-runner
|
||||
These utilities are not documented here, but for example, please see
|
||||
how the TCP tests found in ``src/test/ns3tcp/`` use pcap files and reference
|
||||
output.
|
||||
|
||||
Reference in New Issue
Block a user