F5/unison
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

doc: update python bindings docs to use cppyy

Browse Source
This commit is contained in:
Gabriel Ferreira
2022-08-01 17:34:51 -03:00
parent 58c5a94daa
commit 6ac76bd9a2
7 changed files with 120 additions and 683 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGES.md
View File

@@ -35,6 +35,7 @@ Changes from ns-3.36 to ns-3.37
### Changes to build system
* Replaced the Pybindgen python bindings framework with Cppyy.
* Enabled precompiled headers (`NS3_PRECOMPILE_HEADERS`) by default when CCache is found.
* Added a `./ns3 show targets` option to list buildable/runnable targets.
* Replaced `./ns3 --check-config` with `./ns3 show config`.

6
doc/contributing/source/coding-style.rst
View File

@@ -337,11 +337,7 @@ Language features
*****************
As of ns-3.36, |ns3| permits the use of C++-17 (or earlier) features
in the implementation files. Because the |ns3| public APIs are scanned by a
a `Python bindings framework <https://github.com/gjcarneiro/pybindgen>`_ that
doesn't support even all of the C++-11 features, the |ns3| project is more
conservative in using newer C++ standards in the APIs, and may ask
to rework some APIs to retain compatibility with pybindgen.
in the implementation files.
If a developer would like to propose to raise this bar to include more
features than this, please email the developers list. We will move this

558
doc/manual/source/python.rst
View File

@@ -25,35 +25,18 @@ Python, to allow integration of |ns3| with other Python tools and workflows.
The intent is not to provide a different language choice to author new
|ns3| models implemented in Python.
Python bindings for |ns3| use a tool called PyBindGen (https://github.com/gjcarneiro/pybindgen)
to create Python modules from the C++ libraries built by CMake. The Python bindings that PyBindGen
uses are maintained in a ``bindings`` directory in each module, and must be maintained to match the
C++ API of that ns-3 module. If the C++ API changes, the Python bindings file
must either be modified by hand accordingly, or the bindings must be
regenerated by an automated scanning process.
Python bindings for |ns3| use a tool called Cppyy (https://cppyy.readthedocs.io/en/latest/)
to create a Python module from the C++ libraries built by CMake. The Python bindings that Cppyy
uses are built at runtime, by importing the C++ libraries and headers for each ns-3 module.
This means that even if the C++ API changes, the Python bindings will adapt to them
without requiring any preprocessing or scanning.
If a user is not interested in Python, no action is needed; by default, Python
bindings are disabled (and must be explicitly enabled at
CMake configure time). In this case, changes to the C++
API of a provided module will not cause the module to fail to compile.
If a user is not interested in Python, no action is needed; the Python bindings
are only built on-demand by Cppyy.
The process for automatically generating Python bindings relies on a toolchain
involving a development installation of the Clang compiler, a program called
CastXML (https://github.com/CastXML/CastXML), and a program called
pygccxml (https://github.com/gccxml/pygccxml). The toolchain can be installed
using the ns-3 ``bake`` build tool.
As of ns-3.36, the Python bindings framework (including many of its
constituent tools) has lacked active maintainers for several years, and there
are constraints on the compatible Python version and on the C++ language
features that may be used in the C++ header files. The API scanning framework
(in particular, pygccxml) requires Python 3.6 through 3.8 but is
not compatible with Python 3.9 or 3.10. Compiling the existing bindings
should work through Python 3.10. Additionally, many newer C++ language features
beyond C++11 may not be supported by pybindgen.
Python bindings may be removed from future ns-3 releases if new maintainers
are not found.
As of ns-3.37, the previous Python bindings framework based on Pybindgen has been
removed due to a lack of active maintainers. The Cppyy frameword that replaced
it may also be removed from future ns-3 releases if new maintainers are not found.
An Example Python Script that Runs |ns3|
****************************************
@@ -62,11 +45,7 @@ Here is some example code that is written in Python and that runs |ns3|, which i
::
import ns.applications
import ns.core
import ns.internet
import ns.network
import ns.point_to_point
from ns import ns
ns.core.LogComponentEnable("UdpEchoClientApplication", ns.core.LOG_LEVEL_INFO)
ns.core.LogComponentEnable("UdpEchoServerApplication", ns.core.LOG_LEVEL_INFO)
@@ -84,9 +63,10 @@ Here is some example code that is written in Python and that runs |ns3|, which i
stack.Install(nodes)
address = ns.internet.Ipv4AddressHelper()
address.SetBase(ns.network.Ipv4Address("10.1.1.0"), ns.network.Ipv4Mask("255.255.255.0"))
address.SetBase(ns.network.Ipv4Address("10.1.1.0"),
ns.network.Ipv4Mask("255.255.255.0"))
interfaces = address.Assign (devices);
interfaces = address.Assign(devices)
echoServer = ns.applications.UdpEchoServerHelper(9)
@@ -94,9 +74,10 @@ Here is some example code that is written in Python and that runs |ns3|, which i
serverApps.Start(ns.core.Seconds(1.0))
serverApps.Stop(ns.core.Seconds(10.0))
echoClient = ns.applications.UdpEchoClientHelper(interfaces.GetAddress(1), 9)
address = ns.addressFromIpv4Address(interfaces.GetAddress(1))
echoClient = ns.applications.UdpEchoClientHelper(address, 9)
echoClient.SetAttribute("MaxPackets", ns.core.UintegerValue(1))
echoClient.SetAttribute("Interval", ns.core.TimeValue(ns.core.Seconds (1.0)))
echoClient.SetAttribute("Interval", ns.core.TimeValue(ns.core.Seconds(1.0)))
echoClient.SetAttribute("PacketSize", ns.core.UintegerValue(1024))
clientApps = echoClient.Install(nodes.Get(0))
@@ -106,6 +87,8 @@ Here is some example code that is written in Python and that runs |ns3|, which i
ns.core.Simulator.Run()
ns.core.Simulator.Destroy()
Running Python Scripts
**********************
@@ -113,10 +96,11 @@ First, we need to enable the build of Python bindings:
.. sourcecode:: bash
$ ./ns3 configure --enable-python-bindings
$ ./ns3 configure
Other options such as ``--enable-examples`` may be passed to the above command.
ns3 contains some options that automatically update the python path to find the ns3 module. To run example programs, there are two ways to use ns3 to take care of this. One is to run a ns3 shell; e.g.:
ns3 contains some options that automatically update the python path to find the ns3 module.
To run example programs, there are two ways to use ns3 to take care of this. One is to run a ns3 shell; e.g.:
.. sourcecode:: bash
@@ -154,61 +138,104 @@ To run your own Python script that calls |ns3| and that has this path, ``/path/t
Caveats
*******
In addition to limitations in Python and C++ versions supported (as discussed
above), some additional limitations are listed here.
Some of the limitations of the Cppyy-based byindings are listed here.
Incomplete Coverage
===================
First of all, keep in mind that not 100% of the API is supported in Python. Some of the reasons are:
#. some of the APIs involve pointers, which require knowledge of what kind of memory passing semantics (who owns what memory). Such knowledge is not part of the function signatures, and is either documented or sometimes not even documented. Annotations are needed to bind those functions;
#. Sometimes a unusual fundamental data type or C++ construct is used which is not yet supported by PyBindGen;
#. CastXML does not report template based classes unless they are instantiated.
Memory-management issues
########################
Most of the missing APIs can be wrapped, given enough time, patience, and expertise, and will likely be wrapped if bug reports are submitted. However, don't file a bug report saying "bindings are incomplete", because the project does not have maintainers to maintain every API.
Some of the APIs involve pointers, which require knowledge of what kind of memory passing semantics (who owns what memory).
Such knowledge is not part of the function signatures, and is either documented or sometimes not even documented.
You may need to workaround these issues by instantiating variables on the C++ side with a Just-In-Time (JIT) compiled function.
Conversion Constructors
=======================
For example, when handling command-line arguments, we could set additional parameters like in the following code:
`Conversion constructors <http://publib.boulder.ibm.com/infocenter/compbgpl/v9v111/topic/com.ibm.xlcpp9.bg.doc/language_ref/cplr384.htm>`_ are not fully supported yet by PyBindGen, and they always act as explicit constructors when translating an API into Python. For example, in C++ you can do this:
.. sourcecode:: python
.. sourcecode:: cpp
# Import the ns-3 C++ modules with Cppyy
from ns import ns
Ipv4AddressHelper ipAddrs;
ipAddrs.SetBase ("192.168.0.0", "255.255.255.0");
ipAddrs.Assign (backboneDevices);
# ns.cppyy.cppdef compiles the code defined in the block
# The defined types, values and functions are available in ns.cppyy.gbl
ns.cppyy.cppdef("""
using namespace ns3;
In Python, you have to do:
CommandLine& GetCommandLine(std::string filename, int& nCsma, bool& verbose)
{
static CommandLine cmd = CommandLine(filename);
cmd.AddValue("nCsma", "Number of extra CSMA nodes/devices", nCsma);
cmd.AddValue("verbose", "Tell echo applications to log if true", verbose);
return cmd;
}
""")
::
# To pass the addresses of the Python variables to c++, we need to use ctypes
from ctypes import c_int, c_bool
nCsma = c_int(3)
verbose = c_bool(True)
ipAddrs = ns.internet.Ipv4AddressHelper()
ipAddrs.SetBase(ns.network.Ipv4Address("192.168.0.0"), ns.network.Ipv4Mask("255.255.255.0"))
ipAddrs.Assign(backboneDevices)
# Pass the addresses of Python variables to C++
cmd = ns.cppyy.gbl.GetCommandLine(__file__, nCsma, verbose)
cmd.Parse(sys.argv)
CommandLine
===========
If you find a segmentation violation, be sure to wait for the stacktrace provided by Cppyy
and try to find the root cause of the issue. If you have multiple cores, the number of
stacktraces will correspond to the number of threads being executed by Cppyy. To limit them,
define the environment variable `OPENBLAS_NUM_THREADS=1`.
:cpp:func:`CommandLine::AddValue` works differently in Python than it does in |ns3|. In Python, the first parameter is a string that represents the command-line option name. When the option is set, an attribute with the same name as the option name is set on the :cpp:func:`CommandLine` object. Example:
Operators
#########
::
Cppyy may fail to map C++ operators due to the implementation style used by ns-3.
This happens for the fundamental type `Time`. To provide the expected behavior, we
redefine these operators from the Python side during the setup of the ns-3 bindings
module (`ns-3-dev/bindings/python/ns__init__.py`).
NUM_NODES_SIDE_DEFAULT = 3
.. sourcecode:: python
cmd = ns3.CommandLine()
# Redefine Time operators
cppyy.cppdef("""
using namespace ns3;
bool Time_ge(Time& a, Time& b){ return a >= b;}
bool Time_eq(Time& a, Time& b){ return a == b;}
bool Time_ne(Time& a, Time& b){ return a != b;}
bool Time_le(Time& a, Time& b){ return a <= b;}
bool Time_gt(Time& a, Time& b){ return a > b;}
bool Time_lt(Time& a, Time& b){ return a < b;}
""")
cppyy.gbl.ns3.Time.__ge__ = cppyy.gbl.Time_ge
cppyy.gbl.ns3.Time.__eq__ = cppyy.gbl.Time_eq
cppyy.gbl.ns3.Time.__ne__ = cppyy.gbl.Time_ne
cppyy.gbl.ns3.Time.__le__ = cppyy.gbl.Time_le
cppyy.gbl.ns3.Time.__gt__ = cppyy.gbl.Time_gt
cppyy.gbl.ns3.Time.__lt__ = cppyy.gbl.Time_lt
cmd.NumNodesSide = None
cmd.AddValue("NumNodesSide", "Grid side number of nodes (total number of nodes will be this number squared)")
cmd.Parse(argv)
A different operator used by ns-3 is `operator Address()`, used to
convert different types of Addresses into the generic type Address.
This is not supported by Cppyy and requires explicit conversion.
Some helpers have been added to handle the common cases.
[...]
.. sourcecode:: python
# Define ns.cppyy.gbl.addressFromIpv4Address and others
cppyy.cppdef("""using namespace ns3;
Address addressFromIpv4Address(Ipv4Address ip){ return Address(ip); };
Address addressFromInetSocketAddress(InetSocketAddress addr){ return Address(addr); };
Address addressFromPacketSocketAddress(PacketSocketAddress addr){ return Address(addr); };
""")
# Expose addressFromIpv4Address as a member of the ns3 namespace (equivalent to ns)
setattr(cppyy.gbl.ns3, "addressFromIpv4Address", cppyy.gbl.addressFromIpv4Address)
setattr(cppyy.gbl.ns3, "addressFromInetSocketAddress", cppyy.gbl.addressFromInetSocketAddress)
setattr(cppyy.gbl.ns3, "addressFromPacketSocketAddress", cppyy.gbl.addressFromPacketSocketAddress)
Most of the missing APIs can be wrapped, given enough time, patience, and expertise, and will likely be wrapped if bug reports are submitted.
However, don't file a bug report saying "bindings are incomplete", because the project does not have maintainers to maintain every API.
if cmd.NumNodesSide is None:
num_nodes_side = NUM_NODES_SIDE_DEFAULT
else:
num_nodes_side = int(cmd.NumNodesSide)
Tracing
=======
@@ -217,17 +244,20 @@ Callback based tracing is not yet properly supported for Python, as new |ns3| AP
Pcap file writing is supported via the normal API.
ASCII tracing is supported via the normal C++ API translated to Python. However, ASCII tracing requires the creation of an ostream object to pass into the ASCII tracing methods. In Python, the C++ std::ofstream has been minimally wrapped to allow this. For example:
ASCII tracing is supported via the normal C++ API translated to Python.
However, ASCII tracing requires the creation of an ostream object to pass into the ASCII tracing methods.
In Python, the C++ std::ofstream has been minimally wrapped to allow this. For example:
::
ascii = ns3.ofstream("wifi-ap.tr") # create the file
ns3.YansWifiPhyHelper.EnableAsciiAll(ascii)
ns3.Simulator.Run()
ns3.Simulator.Destroy()
ascii = ns.ofstream("wifi-ap.tr") # create the file
ns.YansWifiPhyHelper.EnableAsciiAll(ascii)
ns.Simulator.Run()
ns.Simulator.Destroy()
ascii.close() # close the file
There is one caveat: you must not allow the file object to be garbage collected while |ns3| is still using it. That means that the 'ascii' variable above must not be allowed to go out of scope or else the program will crash.
There is one caveat: you must not allow the file object to be garbage collected while |ns3| is still using it.
That means that the 'ascii' variable above must not be allowed to go out of scope or else the program will crash.
Working with Python Bindings
****************************
@@ -241,14 +271,14 @@ The python bindings are generated into an 'ns' namespace. Examples:
::
from ns.network import Node
n1 = Node()
from ns import ns
n1 = ns.network.Node()
or
::
import ns.network
from ns import*
n1 = ns.network.Node()
The best way to explore the bindings is to look at the various example
@@ -257,373 +287,9 @@ example. There is no structured documentation for the Python bindings
like there is Doxygen for the C++ API, but the Doxygen can be consulted
to understand how the C++ API works.
Python Bindings Workflow
========================
The process by which Python bindings are handled is the following:
#. Periodically a developer uses a CastXML (https://github.com/CastXML/CastXML) based API scanning script, which saves the scanned API definition as ``bindings/python/ns3_module_*.py`` files or as Python files in each modules' ``bindings`` directory. These files are kept under version control in the main |ns3| repository;
#. Other developers clone the repository and use the already scanned API definitions;
#. When configuring |ns3|, pybindgen will be automatically downloaded if not already installed. Released |ns3| tarballs will ship a copy of pybindgen.
If something goes wrong with compiling Python bindings and you just want to ignore them and move on with C++, you can disable Python by configuring |ns3| without Python bindings enabled.
Regenerating the Python bindings
================================
|ns3| will fail to successfully compile the Python bindings if the C++
headers are changed and no longer align with the stored Python bindings.
In this case, the developer has two main choices: 1) disable Python
as described above, or 2) update the bindings to align with the new C++
API.
Process Overview
################
|ns3| has an automated process to regenerate Python bindings from the C++
header files. The process is only supported for Linux at the moment
because the project has not found a contributor yet to test and
document the capability on macOS. In short, the process currently
requires the following steps on a Linux machine.
1. Prepare the system for scanning by installing the prerequisites,
including a development version of ``clang``, the ``CastXML`` package,
``pygccxml``, and ``pybindgen``.
2. Perform a scan of the module of interest or all modules
Installing a clang development environment
##########################################
Make sure you have a development version of the clang compiler installed
on your system. This can take a long time to build from source. Linux
distributions provide binary library packages such as ``libclang-dev``
(Ubuntu) or ``clang-devel`` (Fedora).
Installing other prerequisites
##############################
``cxxfilt`` is a new requirement, typically installed using ``pip`` or ``pip3``; e.g.
.. sourcecode:: bash
pip3 install --user cxxfilt
See also the wiki for installation notes for your system.
Set up a ``bake`` build environment
###################################
Try the following commands:
.. sourcecode:: bash
$ cd bake
$ export PATH=`pwd`/build/bin:$PATH
$ export LD_LIBRARY_PATH=`pwd`/build/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
$ export PYTHONPATH=`pwd`/build/lib${PYTHONPATH:+:$PYTHONPATH}
$ mkdir -p build/lib
Configure
#########
Perform a configuration at the bake level:
.. sourcecode:: bash
$ ./bake.py configure -e ns-3-dev -e pygccxml
The output of ``./bake.py show`` should show something like this:
.. sourcecode:: bash
$ ./bake.py show
Should say (note: some are OK to be 'Missing' for Python bindings scanning):
.. sourcecode:: text
-- System Dependencies --
> clang-dev - OK
> cmake - OK
> cxxfilt - OK
> g++ - OK
> gi-cairo - OK or Missing
> gir-bindings - OK or Missing
> llvm-dev - OK
> pygobject - OK or Missing
> pygraphviz - OK or Missing
> python3-dev - OK
> python3-setuptools - OK
> qt - OK or Missing
> setuptools - OK
Note that it is not harmful for Python bindings if some of the items above
report as Missing. For Python bindings, the important
prerequisites are clang-dev, cmake, cxxfilt, llvm-dev, python3-dev,
and python3-setuptools. In the following process, the following programs
and libraries will be locally installed: castxml, pybindgen, pygccxml,
and |ns3|.
Note also that the `ns-3-allinone` target for bake will also include the
`pygccxml` and `ns-3-dev` targets (among other libraries) and can be
used instead, e.g.:
.. sourcecode:: bash
$ ./bake.py configure -e ns-3-allinone
Download
########
Issue the following download command. Your output may vary depending on what
is present or missing on your system.
.. sourcecode:: bash
$ ./bake.py download
>> Searching for system dependency llvm-dev - OK
>> Searching for system dependency clang-dev - OK
>> Searching for system dependency qt - Problem
> Problem: Optional dependency, module "qt" not available
This may reduce the functionality of the final build.
However, bake will continue since "qt" is not an essential dependency.
For more information call bake with -v or -vvv, for full verbose mode.
>> Searching for system dependency g++ - OK
>> Searching for system dependency cxxfilt - OK
>> Searching for system dependency setuptools - OK
>> Searching for system dependency python3-setuptools - OK
>> Searching for system dependency gi-cairo - Problem
> Problem: Optional dependency, module "gi-cairo" not available
This may reduce the functionality of the final build.
However, bake will continue since "gi-cairo" is not an essential dependency.
For more information call bake with -v or -vvv, for full verbose mode.
>> Searching for system dependency gir-bindings - Problem
> Problem: Optional dependency, module "gir-bindings" not available
This may reduce the functionality of the final build.
However, bake will continue since "gir-bindings" is not an essential dependency.
For more information call bake with -v or -vvv, for full verbose mode.
>> Searching for system dependency pygobject - Problem
> Problem: Optional dependency, module "pygobject" not available
This may reduce the functionality of the final build.
However, bake will continue since "pygobject" is not an essential dependency.
For more information call bake with -v or -vvv, for full verbose mode.
>> Searching for system dependency pygraphviz - Problem
> Problem: Optional dependency, module "pygraphviz" not available
This may reduce the functionality of the final build.
However, bake will continue since "pygraphviz" is not an essential dependency.
For more information call bake with -v or -vvv, for full verbose mode.
>> Searching for system dependency python3-dev - OK
>> Searching for system dependency cmake - OK
>> Downloading castxml - OK
>> Downloading netanim - OK
>> Downloading pybindgen - OK
>> Downloading pygccxml - OK
>> Downloading ns-3-dev - OK
Build
#####
Next, try the following command:
.. sourcecode:: bash
$ ./bake.py build
A build report should be printed for each package, such as:
::
>> Building castxml - OK
>> Building netanim - OK
>> Building pybindgen - OK
>> Building pygccxml - OK
>> Building ns-3-dev - OK
However, if there is a problem with the bindings compilation (or with the
C++ code), it will report a failure instead:
::
>> Building ns-3-dev - Problem
> Error: Critical dependency, module "ns-3-dev" failed
For more information call Bake with --debug and/or -v, -vvv, for full verbose mode (bake --help)
At this point, it is recommended to change into the ns-3-dev directory and
work further from there, because the API scanning dependencies have been
built and installed successfully into the `build` directory.
The output of './ns3 configure' can be inspected to see if Python API scanning
support is enabled:
.. sourcecode:: text
Python API Scanning Support : enabled
It may say something like this, if the support is not active or something
went wrong in the build process:
.. sourcecode:: text
Python API Scanning Support : not enabled (Missing 'pygccxml' Python module)
In this case, the user must take additional steps to resolve. For the
API scanning support to be detected, the castxml binary must be in the
shell's PATH, and pygccxml must be in the PYTHONPATH.
LP64 vs ILP32 bindings
######################
Linux (64-bit, as most modern installations use) and MacOS use different
data models, as explained here: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.cbcpx01/datatypesize64.htm
Linux uses the LP64 model, and MacOS (as well as 32-bit Linux) use the ILP32
model. Users will note that there are two versions of bindings files in
each ns-3 module directory; one with an ILP32.py suffix and one with
an LP64.py suffix. Only one is used on any given platform. The main
difference is in the representation of the 64 bit integer type as either
a 'long' (LP64) or 'long long' (ILP32).
The process (only supported on Linux at present) generates the LP64
bindings using the toolchain and then copies the LP64 bindings to the
ILP32 bindings with some type substitutions automated by CMake scripts.
Rescanning a module
###################
To re-scan a module, it is recommended to clean the installation, and then
to configure with Python scanning enabled:
.. sourcecode:: bash
$ cd source/ns-3-dev
$ ./ns3 clean
$ ./ns3 configure -- -DNS3_SCAN_PYTHON_BINDINGS=ON
Ensure in the configure output that pygccxml and castxml were found. To
scan an individual module, such as wifi, append `-apiscan` to the module
name:
.. sourcecode:: bash
$ ./ns3 build wifi-apiscan
To re-scan all modules (which can take some time):
.. sourcecode:: bash
$ cd source/ns-3-dev
$ ./ns3 build apiscan-all
Then, to check whether the rescanned bindings can be compiled, enable
the Python bindings in the build:
.. sourcecode:: bash
$ ./ns3 configure --enable-python-bindings
$ ./ns3 build
Generating bindings on MacOS
############################
In principle, this should work (and should generate the 32-bit bindings).
However, maintainers have not been available to complete this port to date.
We would welcome suggestions on how to enable scanning for MacOS.
Regenerating the Python bindings using gitlab-ci-local
======================================================
.. _gitlab-ci-local: https://github.com/firecow/gitlab-ci-local
The |ns3| GitLab.com continuous integration (CI) system can be run on a
local machine using `gitlab-ci-local`_.
From within the ns-3 directory:
.. sourcecode:: bash
$ gitlab-ci-local --file ./utils/tests/gitlab-ci.yml pybindgen-21.04
If everything works, the diff with the re-scanned bindings will be in
``./gitlab-ci-local/artifacts/pybindgen-21.04``.
Regenerating the Python bindings using Docker
=============================================
An alternative to the above Bake install or gitlab-ci-local is to use a
Docker container. Basically, the below steps are the low-level commands
used by the gitlab-ci-local approach above.
The following steps are used by the |ns3| CI system on an Ubuntu 20.04 image.
As a prerequisite, install Docker using a package manager; on Ubuntu, use:
.. sourcecode:: bash
$ apt install docker.io
To allow an unprivileged user to use Docker, perform the following:
.. sourcecode:: bash
$ sudo chmod 666 /var/run/docker.sock
The following command will obtain a raw shell for an Ubuntu 20.04 image:
.. sourcecode:: bash
$ docker run -it ubuntu:20.04
However, if you prefer to work on your current directory from the container
shell, use the ``-v`` option to request a bind mount of a host directory
to a container directory, such as follows:
.. sourcecode:: bash
$ docker run -it -v /path/to/your/current/directory:/path/to/container/mount ubuntu:20.04
$ cd /path/to/container/mount
where ``/path/to/container/mount`` can be something like ``/ns-3-dev``.
Then, from within a directory that contains ns-3, perform the following
steps to rescan the APIs and test them:
.. sourcecode:: bash
$ apt-get update
$ apt-get install -y g++ cmake ninja-build ccache libgsl-dev libgtk-3-dev libboost-dev wget git python3 python3-pip
$ pip install cxxfilt pygccxml pybindgen castxml
$ ./ns3 clean
$ ./ns3 configure -G Ninja -- -DNS3_SCAN_PYTHON_BINDINGS=ON
$ ./ns3 build apiscan-all
$ ./ns3 configure --enable-python-bindings
$ ./ns3 build
$ ./ns3 run ./utils/python-unit-tests.py
If you only want to scan one module, specify ``MODULENAME-apiscan``
(such as ``wifi-apiscan``) instead of ``apiscan-all`` in the above.
If you exit the container, the package state will be lost, but from another
terminal window, you may be able to further test and evaluate the new bindings.
Note that the new bindings and build files will have root file ownership.
Organization of the Modular Python Bindings
===========================================
The ``src/<module>/bindings`` directory may contain the following files, some of them optional:
* ``callbacks_list.py``: this is a scanned file, DO NOT TOUCH. Contains a list of Callback<...> template instances found in the scanned headers;
* ``modulegen__gcc_LP64.py``: this is a scanned file, DO NOT TOUCH. Scanned API definitions for the GCC, LP64 architecture (64-bit)
* ``modulegen__gcc_ILP32.py``: this is a scanned file, DO NOT TOUCH. Scanned API definitions for the GCC, ILP32 architecture (32-bit)
* ``modulegen_customizations.py``: you may optionally add this file in order to customize the pybindgen code generation
* ``scan-header.h``: you may optionally add this file to customize what header file is scanned for the module. Basically this file is scanned instead of ns3/<module>-module.h. Typically, the first statement is #include "ns3/<module>-module.h", plus some other stuff to force template instantiations;
* ``module_helpers.cc``: you may add additional files, such as this, to be linked to python extension module. They will be automatically scanned;
* ``<module>.py``: if this file exists, it becomes the "frontend" python module for the ns3 module, and the extension module (.so file) becomes _<module>.so instead of <module>.so. The <module>.py file has to import all symbols from the module _<module> (this is more tricky than it sounds, see src/core/bindings/core.py for an example), and then can add some additional pure-python definitions.
Historical Information
**********************
If you are a developer and need more background information on |ns3|'s Python bindings, please see the `Python Bindings wiki page <http://www.nsnam.org/wiki/NS-3_Python_Bindings>`_. Please note, however, that some information on that
page is stale.
If you are a developer and need more background information on |ns3|'s Python bindings,
please see the `Python Bindings wiki page <http://www.nsnam.org/wiki/NS-3_Python_Bindings>`_.
Please note, however, that some information on that page is stale.

229
doc/manual/source/working-with-cmake.rst
View File

@@ -1871,235 +1871,6 @@ CMakeLists.txt file, creating the examples. It also scans for python examples.
# ...
endfunction()
The following block creates the ``lib${BLIB_LIBNAME}-apiscan`` CMake target.
When the target is built, it runs modulescan-modular to extract the python
bindings for the module using pybindgen.
.. sourcecode:: cmake
function(build_lib)
# ...
# Get architecture pair for python bindings
if((${CMAKE_SIZEOF_VOID_P} EQUAL 8) AND (NOT APPLE))
set(arch gcc_LP64)
set(arch_flags -m64)
else()
set(arch gcc_ILP32)
set(arch_flags)
endif()
# Add target to scan python bindings
if(${ENABLE_SCAN_PYTHON_BINDINGS}
AND EXISTS ${CMAKE_HEADER_OUTPUT_DIRECTORY}/${BLIB_LIBNAME}-module.h
)
set(bindings_output_folder ${PROJECT_SOURCE_DIR}/${FOLDER}/bindings)
file(MAKE_DIRECTORY ${bindings_output_folder})
set(module_api_ILP32 ${bindings_output_folder}/modulegen__gcc_ILP32.py)
set(module_api_LP64 ${bindings_output_folder}/modulegen__gcc_LP64.py)
set(modulescan_modular_command
${Python_EXECUTABLE}
${PROJECT_SOURCE_DIR}/bindings/python/ns3modulescan-modular.py
)
set(header_map "")
# We need a python map that takes header.h to module e.g. "ptr.h": "core"
foreach(header ${BLIB_HEADER_FILES})
# header is a relative path to the current working directory
get_filename_component(
header_name ${CMAKE_CURRENT_SOURCE_DIR}/${header} NAME
)
string(APPEND header_map "\"${header_name}\":\"${BLIB_LIBNAME}\",")
endforeach()
set(ns3-headers-to-module-map "${ns3-headers-to-module-map}${header_map}"
CACHE INTERNAL "Map connecting headers to their modules"
)
# API scan needs the include directories to find a few headers (e.g. mpi.h)
get_target_includes(${lib${BLIB_LIBNAME}} modulegen_include_dirs)
set(module_to_generate_api ${module_api_ILP32})
set(LP64toILP32)
if("${arch}" STREQUAL "gcc_LP64")
set(module_to_generate_api ${module_api_LP64})
set(LP64toILP32
${Python_EXECUTABLE}
${PROJECT_SOURCE_DIR}/build-support/pybindings_LP64_to_ILP32.py
${module_api_LP64} ${module_api_ILP32}
)
endif()
add_custom_target(
${lib${BLIB_LIBNAME}}-apiscan
COMMAND
${modulescan_modular_command} ${CMAKE_OUTPUT_DIRECTORY} ${BLIB_LIBNAME}
${PROJECT_BINARY_DIR}/header_map.json ${module_to_generate_api}
\"${arch_flags} ${modulegen_include_dirs}\"
COMMAND ${LP64toILP32}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
DEPENDS ${lib${BLIB_LIBNAME}}
)
add_dependencies(apiscan-all ${lib${BLIB_LIBNAME}}-apiscan)
endif()
# ...
endfunction()
The targets can be built to execute the scanning using one of the following commands:
.. sourcecode:: console
~/cmake-cache$ cmake --build . --target libcore-apiscan
~/ns-3-dev/$ ./ns3 build core-apiscan
To re-scan all bindings, use ``./ns3 build apiscan-all``.
The next block runs the ``modulegen-modular`` and consumes the results of the previous step.
However, this step runs at configuration time, while the previous runs at build time.
This means the output directory needs to be cleared after running ``apiscan``, to regenerate
up-to-date bindings source files.
.. sourcecode:: cmake
function(build_lib)
# ...
# Build pybindings if requested and if bindings subfolder exists in
# NS3/src/BLIB_LIBNAME
if(${ENABLE_PYTHON_BINDINGS} AND EXISTS
"${CMAKE_CURRENT_SOURCE_DIR}/bindings"
)
set(bindings_output_folder ${CMAKE_OUTPUT_DIRECTORY}/${FOLDER}/bindings)
file(MAKE_DIRECTORY ${bindings_output_folder})
set(module_src ${bindings_output_folder}/ns3module.cc)
set(module_hdr ${bindings_output_folder}/ns3module.h)
string(REPLACE "-" "_" BLIB_LIBNAME_sub ${BLIB_LIBNAME}) # '-' causes
# problems (e.g.
# csma-layout), replace with '_' (e.g. csma_layout)
# Set prefix of binding to _ if a ${BLIB_LIBNAME}.py exists, and copy the
# ${BLIB_LIBNAME}.py to the output folder
set(prefix)
if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/bindings/${BLIB_LIBNAME}.py)
set(prefix _)
file(COPY ${CMAKE_CURRENT_SOURCE_DIR}/bindings/${BLIB_LIBNAME}.py
DESTINATION ${CMAKE_OUTPUT_DIRECTORY}/bindings/python/ns
)
endif()
# Run modulegen-modular to generate the bindings sources
if((NOT EXISTS ${module_hdr}) OR (NOT EXISTS ${module_src})) # OR TRUE) # to
# force
# reprocessing
string(REPLACE ";" "," ENABLED_FEATURES
"${ns3-libs};${BLIB_MODULE_ENABLED_FEATURES}"
)
set(modulegen_modular_command
GCC_RTTI_ABI_COMPLETE=True NS3_ENABLED_FEATURES="${ENABLED_FEATURES}"
${Python_EXECUTABLE}
${PROJECT_SOURCE_DIR}/bindings/python/ns3modulegen-modular.py
)
execute_process(
COMMAND
${CMAKE_COMMAND} -E env
PYTHONPATH=${CMAKE_OUTPUT_DIRECTORY}:$ENV{PYTHONPATH}
${modulegen_modular_command} ${CMAKE_CURRENT_SOURCE_DIR} ${arch}
${prefix}${BLIB_LIBNAME_sub} ${module_src}
TIMEOUT 60
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
OUTPUT_FILE ${module_hdr}
ERROR_FILE ${bindings_output_folder}/ns3modulegen.log
RESULT_VARIABLE error_code
)
if(${error_code} OR NOT (EXISTS ${module_hdr}))
message(
FATAL_ERROR
"Something went wrong during processing of the python bindings of module ${BLIB_LIBNAME}."
" Make sure you have the latest version of Pybindgen."
)
if(EXISTS ${module_src})
file(REMOVE ${module_src})
endif()
endif()
endif()
# Add core module helper sources
set(python_module_files ${module_hdr} ${module_src})
if(${BLIB_LIBNAME} STREQUAL "core")
list(APPEND python_module_files
${CMAKE_CURRENT_SOURCE_DIR}/bindings/module_helpers.cc
${CMAKE_CURRENT_SOURCE_DIR}/bindings/scan-header.h
)
endif()
set(bindings-name lib${BLIB_LIBNAME}-bindings)
add_library(${bindings-name} SHARED "${python_module_files}")
target_include_directories(
${bindings-name} PUBLIC ${Python_INCLUDE_DIRS} ${bindings_output_folder}
)
target_compile_options(${bindings-name} PRIVATE -Wno-error)
# If there is any, remove the "lib" prefix of libraries (search for
# "set(lib${BLIB_LIBNAME}")
list(LENGTH ns_libraries_to_link num_libraries)
if(num_libraries GREATER "0")
string(REPLACE ";" "-bindings;" bindings_to_link
"${ns_libraries_to_link};"
) # add -bindings suffix to all lib${name}
endif()
target_link_libraries(
${bindings-name}
PUBLIC ${LIB_AS_NEEDED_PRE} ${lib${BLIB_LIBNAME}} "${bindings_to_link}"
"${BLIB_LIBRARIES_TO_LINK}" ${LIB_AS_NEEDED_POST}
PRIVATE ${Python_LIBRARIES}
)
target_include_directories(
${bindings-name} PRIVATE ${PROJECT_SOURCE_DIR}/src/core/bindings
)
set(suffix)
if(APPLE)
# Python doesn't like Apple's .dylib and will refuse to load bindings
# unless its an .so
set(suffix SUFFIX .so)
endif()
# Set binding library name and output folder
set_target_properties(
${bindings-name}
PROPERTIES OUTPUT_NAME ${prefix}${BLIB_LIBNAME_sub}
PREFIX ""
${suffix} LIBRARY_OUTPUT_DIRECTORY
${CMAKE_OUTPUT_DIRECTORY}/bindings/python/ns
)
set(ns3-python-bindings-modules
"${bindings-name};${ns3-python-bindings-modules}"
CACHE INTERNAL "list of modules python bindings"
)
# Make sure all bindings are built before building the visualizer module
# that makes use of them
if(${ENABLE_VISUALIZER} AND (visualizer IN_LIST libs_to_build))
if(NOT (${BLIB_LIBNAME} STREQUAL visualizer))
add_dependencies(${libvisualizer} ${bindings-name})
endif()
endif()
endif()
# ...
endfunction()
The recommended steps to scan the bindings and then use them is the following:
.. sourcecode:: console
~/ns-3-dev$ ./ns3 clean
~/ns-3-dev$ ./ns3 configure -- -DNS3_SCAN_PYTHON_BINDINGS=ON
~/ns-3-dev$ ./ns3 build apiscan-all
~/ns-3-dev$ ./ns3 configure --enable-python-bindings
~/ns-3-dev$ ./ns3 build
In the next code block we add the library to the ns3ExportTargets, later used for installation.
We also print an additional message the folder just finished being processed if NS3_VERBOSE is set to ON.

2
doc/manual/source/working-with-git.rst
View File

@@ -17,7 +17,7 @@ The ns-3 project used Mercurial in the past as its source code control system, b
The ns-3 project is officially hosted on GitLab.com at https://gitlab.com/nsnam/. For convenience and historical reasons, ns-3-dev mirrors are currently posted on Bitbucket.com and GitHub.com, and kept in sync with the official repository periodically via cron jobs. We recommend that users who have been working from one of these mirrors repoint their remotes so that they pull origin or upstream from GitLab.com (see below explanation about how to configure remotes).
This section of the manual provides common tips for both users and maintainers. Since the first part is shared, in this manual section we will start with a personal repository and then explain what to do in some typical cases. ns-3 users often combine ns-3-dev with other repositories (pybindgen, netanim, apps from the app store). This manual chapter does not cover this use case; it only focuses on the single ns-3-dev repository. See other project documentation such as the ns-3 tutorial for descriptions on bundled releases distributed as source archives, or on the bake build tool for managing multiple repositories. The guidelines listed below also largely pertain to the user who is using (and cloning) bake from the GitLab.com repository.
This section of the manual provides common tips for both users and maintainers. Since the first part is shared, in this manual section we will start with a personal repository and then explain what to do in some typical cases. ns-3 users often combine ns-3-dev with other repositories (netanim, apps from the app store). This manual chapter does not cover this use case; it only focuses on the single ns-3-dev repository. See other project documentation such as the ns-3 tutorial for descriptions on bundled releases distributed as source archives, or on the bake build tool for managing multiple repositories. The guidelines listed below also largely pertain to the user who is using (and cloning) bake from the GitLab.com repository.
ns-3's Git workflow in a nutshell
*********************************

5
doc/manual/source/working-with-gitlab-ci-local.rst
View File

@@ -66,7 +66,7 @@ the ns-3 jobs can be listed using the following command:
...
weekly-build-clang-11-optimized build on_success false
pybindgen build on_success false
cppyy-22.04 build on_success false
per-commit-compile-debug build on_success false
per-commit-compile-release build on_success false
per-commit-compile-optimized build on_success false
@@ -117,6 +117,9 @@ to run after weekly jobs are successfully executed.
Another option is to run the specific job that produces the required artifact. In this case
the ``pybindgen`` job.
Note: Pybindgen has been replaced by Cppyy, which does not produce artifacts to be consumed
by other jobs. However, the example is kept for reference.
.. sourcecode:: console
~/ns-3-dev$ gitlab-ci-local --file ./utils/tests/gitlab-ci.yml pybindgen

2
doc/release_steps.txt
View File

@@ -33,7 +33,7 @@ and/or the buildbots have been testing
- ensure that tests pass (./test.py -g) and make sure that the buildbots
are reporting 'pass' state, based on the tip of the repository
- revise and check in AUTHORS, RELEASE_NOTES.md, and CHANGES.html
- required versions for related libraries (netanim, pybindgen)
- required versions for related libraries (netanim, cppyy)
are correct
- confirm that Doxygen builds cleanly (./ns3 doxygen),
- confirm that the new bake configurations for the release work correctly