From e0f6baf9980e5d53037e9dc52225b60f90e4519a Mon Sep 17 00:00:00 2001
From: Tom Henderson <tomh@tomh.org>
Date: Mon, 18 Sep 2017 13:46:16 -0700
Subject: [PATCH] bindings:  Updates corresponding to project use of CastXML

---
 CHANGES.html                             |  2 +
 bindings/python/ns3modulescan-modular.py |  1 -
 bindings/python/wscript                  |  2 +-
 doc/manual/source/python.rst             | 97 ++++++++++++++++--------
 4 files changed, 69 insertions(+), 33 deletions(-)

diff --git a/CHANGES.html b/CHANGES.html
index e576bc771..fb41f0d64 100644
--- a/CHANGES.html
+++ b/CHANGES.html
@@ -172,6 +172,8 @@ is now exported by WifiNetDevice.</li>
 </ul>
 <h2>Changes to build system:</h2>
 <ul>
+<li> The API scanning process for Python bindings now relies on CastXML, and only 64-bit scans are presently supported (Linux 64-bit systems).  Generation of 32-bit scans is documented in the Python chapter of the ns-3 manual.
+</li>
 <li> Modules can now be located in the 'contrib/' directory in addition to 'src/'
 </li>
 <li> Behavior for running Python programs was aligned with that of C++ programs; the list of modules built is no longer printed out.
diff --git a/bindings/python/ns3modulescan-modular.py b/bindings/python/ns3modulescan-modular.py
index de4785338..e3a904918 100644
--- a/bindings/python/ns3modulescan-modular.py
+++ b/bindings/python/ns3modulescan-modular.py
@@ -259,7 +259,6 @@ def ns3_module_scan(top_builddir, module_name, headers_map, output_file_name, cf
 
     module_parser.parse_init([scan_header],
                              None, whitelist_paths=[top_builddir],
-                             #includes=['"ns3/everything.h"'],
                              pygen_sink=output_sink,
                              castxml_options=castxml_options)
     module_parser.scan_types()
diff --git a/bindings/python/wscript b/bindings/python/wscript
index fa9c0f592..1dd4a5785 100644
--- a/bindings/python/wscript
+++ b/bindings/python/wscript
@@ -52,7 +52,7 @@ def options(opt):
                    dest='python_disable')
     opt.add_option('--apiscan',
                    help=("Rescan the API for the indicated module(s), for Python bindings.  "
-                         "Needs working GCCXML / pygccxml environment.  "
+                         "Needs working CastXML / pygccxml environment.  "
                          "The metamodule 'all' expands to all available ns-3 modules."),
                    default=None, dest='apiscan', metavar="MODULE[,MODULE...]")
     opt.add_option('--with-pybindgen',
diff --git a/doc/manual/source/python.rst b/doc/manual/source/python.rst
index 50d58fdce..7788e79e4 100644
--- a/doc/manual/source/python.rst
+++ b/doc/manual/source/python.rst
@@ -18,7 +18,22 @@ Python, to allow integration of |ns3| with other Python tools and workflows.
 The intent is not to provide a different language choice to author new
 |ns3| models implemented in Python.
 
-Python bindings for |ns3| use a tool called PyBindGen (https://github.com/gjcarneiro/pybindgen).
+Python bindings for |ns3| use a tool called PyBindGen (https://github.com/gjcarneiro/pybindgen) to create Python modules from the C++ libraries built by
+Waf.  The Python bindings that PyBindGen uses are maintained in a 
+``bindings`` directory in each module, and must be maintained to match the 
+C++ API of that ns-3 module.  If the C++ API changes, the Python bindings file
+must either be modified by hand accordingly, or the bindings must be
+regenerated by an automated scanning process.
+
+If a user is not interested in Python, he or she may disable the use of
+Python bindings at Waf configure time.  In this case, changes to the C++
+API of a provided module will not cause the module to fail to compile. 
+
+The process for automatically generating Python bindings relies on a toolchain
+involving a development installation of the Clang compiler, a program called
+CastXML (https://github.com/CastXML/CastXML), and a program called 
+pygccxml (https://github.com/gccxml/pygccxml).  The toolchain can be installed
+using the ns-3 ``bake`` build tool.
 
 An Example Python Script that Runs |ns3|
 ****************************************
@@ -113,7 +128,7 @@ First of all, keep in mind that not 100% of the API is supported in Python.  Som
 
 #. some of the APIs involve pointers, which require knowledge of what kind of memory passing semantics (who owns what memory). Such knowledge is not part of the function signatures, and is either documented or sometimes not even documented.  Annotations are needed to bind those functions;
 #. Sometimes a unusual fundamental data type or C++ construct is used which is not yet supported by PyBindGen;
-#. GCC-XML does not report template based classes unless they are instantiated.
+#. GCC-XML does not report template based classes unless they are instantiated. (Note:  Need to confirm this limitation still exists with CastXML)
 
 Most of the missing APIs can be wrapped, given enough time, patience, and expertise, and will likely be wrapped if bug reports are submitted.  However, don't file a bug report saying "bindings are incomplete", because we do not have manpower to complete 100% of the bindings.
 
@@ -178,22 +193,6 @@ ASCII tracing is supported since |ns3|.4 via the normal C++ API translated to Py
 
 There is one caveat: you must not allow the file object to be garbage collected while |ns3| is still using it.  That means that the 'ascii' variable above must not be allowed to go out of scope or else the program will crash.
 
-Cygwin limitation
-+++++++++++++++++
-
-Python bindings do not work on Cygwin.  This is due to a gccxml bug.
-
-You might get away with it by re-scanning API definitions from within the
-cygwin environment (./waf --apiscan=all).  However the most likely solution
-will probably have to be that we disable python bindings in CygWin.
-
-If you really care about Python bindings on Windows, try building with mingw and native
-python instead.  Or else, to build without python bindings, disable python bindings in the configuration stage:  
-
-.. sourcecode:: bash
-
-  $ ./waf configure --disable-python
-
 Working with Python Bindings
 ****************************
 
@@ -204,7 +203,7 @@ Python Bindings Workflow
 
 The process by which Python bindings are handled is the following:
 
-#. Periodically a developer uses a GCC-XML (http://www.gccxml.org) based API scanning script, which saves the scanned API definition as ``bindings/python/ns3_module_*.py`` files or as Python files in each modules' ``bindings`` directory.  These files are kept under version control in the main |ns3| repository;
+#. Periodically a developer uses a CastXML (https://github.com/CastXML/CastXML) based API scanning script, which saves the scanned API definition as ``bindings/python/ns3_module_*.py`` files or as Python files in each modules' ``bindings`` directory.  These files are kept under version control in the main |ns3| repository;
 #. Other developers clone the repository and use the already scanned API definitions;
 #. When configuring |ns3|, pybindgen will be automatically downloaded if not already installed.  Released |ns3| tarballs will ship a copy of pybindgen.
 
@@ -251,7 +250,7 @@ to be accessible from Python) or is changing the C++ API in a way that
 breaks the compilation of the existing Python bindings.
 
 There are two steps.  First, the bindings toolchain must be enabled in
-the |ns3| build.  This requires that the gccxml and pygccxml tools be
+the |ns3| build.  This requires that the castxml and pygccxml tools be
 installed on the system or using the bake build tool.  Second, Waf
 can be used to update the bindings.
 
@@ -268,8 +267,8 @@ It may say something like this, if the support is not active:
 
   Python API Scanning Support   : not enabled (Missing 'pygccxml' Python module)
 
-In this case, the user must take steps to install gccxml and pygccxml;
-gccxml binary must be in the shell's path, and pygccxml must be in the
+In this case, the user must take steps to install castxml and pygccxml;
+castxml binary must be in the shell's path, and pygccxml must be in the
 Python path.
 
 An automated setup for this is provided by the `bake` build system, if the
@@ -278,28 +277,24 @@ release number.  For example:
 
 ::
 
-  ./bake.py configure -e ns-allinone-3.26
+  ./bake.py configure -e ns-allinone-3.27
   ./bake.py download
   ./bake.py build
 
-At present, this toolchain is only supported for gcc version 4; gcc-5
-and gcc-6 are not supported due to the gccxml project stopping maintenance
-a few years ago.  clang compiler is also not supported.  The |ns3| project
-plans to convert to the CastXML project to replace gccxml in the future,
-in which case newer versions of gcc as well as clang will be supported.
-
 Once API scanning support is enabled, to scan the modular Python bindings 
 for the core module, for example, do the following:
 
 .. sourcecode:: bash
 
-  $ ./waf --apiscan=core
+  $ ./waf --apiscan=core --no32bit-scan
 
 To scan the modular Python bindings for all of the modules, do the following:
 
 .. sourcecode:: bash
 
-  $ ./waf --apiscan=all
+  $ ./waf --apiscan=all --no32bit-scan
+
+The ``--no32bit-scan`` flag is described below.
 
 Adding Modular Bindings To A Existing or New Module
 +++++++++++++++++++++++++++++++++++++++++++++++++++
@@ -310,6 +305,46 @@ To add support for modular bindings to an existing or new |ns3| module, simply a
 
     bld.ns3_python_bindings()
 
+One must also provide the bindings files (usually by running the scanning
+framework).
+
+Differences between MacOS and Linux bindings
+++++++++++++++++++++++++++++++++++++++++++++
+
+Linux (64-bit, as most modern installations use) and MacOS use different
+data models, as explained here:  https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.cbcpx01/datatypesize64.htm
+
+Linux uses the LP64 model, and MacOS (as well as 32-bit Linux) use the ILP32
+model.  Users will note that there are two versions of bindings files in
+each ns-3 module directory; one with an ILP32.py suffix and one with 
+an LP64.py suffix.  Only one is used on any given platform.  The main
+difference is in the representation of the 64 bit integer type as either
+a 'long' (LP64) or 'long long' (ILP32).  
+  
+As of ns-3.27, the CastXML framework can generate LP64 bindings by passing
+the ``--no32bit-scan`` flag.  However, the framework does not automatically
+generate 32bit scans at the moment.  Instead, users must manually generate
+the ILP32.py equivalents by taking all instances of 'unsigned long' in the
+bindings file and converting them to 'unsigned long long', such as:
+
+::
+
+  -    cls.add_instance_attribute('nMarkedBytes', 'std::map< std::string, unsigned long >', is_const=False)
+  +    cls.add_instance_attribute('nMarkedBytes', 'std::map< std::string, unsigned long long >', is_const=False)
+
+In summary, to generate LP64 bindings for Linux 64-bit systems, it is
+sufficient to call:
+
+::
+
+  $ ./waf --apiscan=core --no32bit-scan
+
+To generate ILP32 bindings, one first must generate the LP64.py file as above,
+and then copy the file to be named with an ILP32.py suffix, and then 
+hand-edit that file, replacing all instances of 'unsigned long' with
+'unsigned long long'.  |ns3| maintainers are working to better automate
+this process for future releases.
+
 Organization of the Modular Python Bindings
 +++++++++++++++++++++++++++++++++++++++++++