The Tracing System
******************

Background
==========

As mentioned in the Using the Tracing System section, the whole point of running an ns-3 simulation is to generate output for study.  You have two basic strategies to work with in ns-3: using generic pre-defined bulk output mechanisms and parsing their content to extract interesting information; or somehow developing an output mechanism that conveys exactly (and perhaps only) the information wanted.

Using pre-defined bulk output mechansims has the advantage of not requiring any changes to ns-3, but it does require programming.  Often, pcap or NS_LOG output messages are gathered during simulation runs and separately run through scripts that use grep, sed or awk to parse the messages and reduce and transform the data to a manageable form.  Programs must be written to do the transformation, so this does not come for free.  Of course, if the information of interest in does not exist in any of the pre-defined output mechanisms, this approach fails.

If you need to add some tidbit of information to the pre-defined bulk mechanisms, this can certainly be done; and if you use one of the ns-3 mechanisms, you may get your code added as a contribution.

ns-3 provides another mechanism, called Tracing, that avoids some of the problems inherent in the bulk output mechanisms.  It has several important advantages.  First, you can reduce the amount of data you have to manage by only tracing the events of interest to you.  Second, if you use this method, you can control the format of the output directly so you avoid the postprocessing step with sed or awk script.  If you desire, your output can be formatted directly into a form acceptable by gnuplot, for example.  You can add hooks in the core which can then be accessed by other users, but which will produce no information unless explicitly asked to do so.  For these reasons, we believe that the ns-3 Tracing system is the best way to get information out of a simulation and is also therefore one of the most important mechanisms to understand in ns-3.

Blunt Instruments
-----------------

There are many ways to get information out of a program.  The most straightforward way is to just directly print the information to the standard output, as in,

  #include <iostream>
  ...
  void
  SomeFunction (void)
  {
    uint32_t x = SOME_INTERESTING_VALUE;
    ...
    std::cout << "The value of x is " << x << std::endl;
    ...
  }

Nobody is going to prevent you from going deep into the core of ns-3 and adding print statements.  This is insanely easy to do and, after all, you have complete control of your own ns-3 branch.  This will probably not turn out to be very satisfactory in the long term, though.

As the number of print statements increases in your programs, the task of dealing with the large number of outputs will become more and more complicated. Eventually, you may feel the need to control what information is being printed in some way; perhaps by turning on and off certain categories of prints, or increasing or decreasing the amount of information you want.  If you continue down this path you may discover that you have re-implemented the NS_LOG mechanism.  In order to avoid that, one of the first things you might consider is using NS_LOG itself.

We mentioned above that one way to get information out of ns-3 is to parse existing NS_LOG output for interesting information.  If you discover that some tidbit of information you need is not present in existing log output, you could edit the core of ns-3 and simply add your interesting information to the output stream.  Now, this is certainly better than adding your own print statements since it follows ns-3 coding conventions and could potentially be useful to other people as a patch to the existing core.

Let's pick a random example.  If you wanted to add more logging to the ns-3 TCP socket (tcp-socket-impl.cc) you could just add a new message down in the implementation.  Notice that in TcpSocketImpl::ProcessAction() there is no log message for the ACK_TX case.  You could simply add one, changing the code from:

  bool TcpSocketImpl::ProcessAction (Actions_t a)
  { // These actions do not require a packet or any TCP Headers
    NS_LOG_FUNCTION (this << a);
    switch (a)
    {
      case NO_ACT:
        NS_LOG_LOGIC ("TcpSocketImpl " << this <<" Action: NO_ACT");
        break;
      case ACK_TX:
        SendEmptyPacket (TcpHeader::ACK);
        break;
      ...

to add a new NS_LOG_LOGIC in the appropriate case statement:

  bool TcpSocketImpl::ProcessAction (Actions_t a)
  { // These actions do not require a packet or any TCP Headers
    NS_LOG_FUNCTION (this << a);
    switch (a)
    {
      case NO_ACT:
        NS_LOG_LOGIC ("TcpSocketImpl " << this << " Action: NO_ACT");
        break;
      case ACK_TX:
        NS_LOG_LOGIC ("TcpSocketImpl " << this << " Action: ACK_TX");
        SendEmptyPacket (TcpHeader::ACK);
        break;
      ...

This may seem fairly simple and satisfying at first glance, but something to consider is that you will be writing code to add the NS_LOG statement and you will also have to write code (as in grep, sed or awk scripts) to parse the log output in order to isolate your information.  This is because even though you have some control over what is output by the logging system, you only have control down to the log component level.

If you are adding code to an existing module, you will also have to live with the output that every other developer has found interesting.  You may find that in order to get the small amount of information you need, you may have to wade through huge amounts of extraneous messages that are of no interest to you.  You may be forced to save huge log files to disk and process them down to a few lines whenever you want to do anything.

Since there are no guarantees in ns-3 about the stability of NS_LOG messages, you may also discover that pieces of log output on which you depend disappear or change between releases.  If you depend on the structure of the output, you may find other messages being added or deleted which may affect your parsing code.

For these reasons, we consider prints to std::cout and NS_LOG messages simple ways to get more information out of ns-3, but they are really unstable and quite blunt instruments.

It is desirable to have a stable facility using stable APIs that allow one to reach into the core system and only get the information required.  It is desirable to be able to do this without having to change and recompile the core system.  Even better would be a system that notified the user when an item of interest changed or an interesting event happened so the user doesn't have to actively go poke around in the system looking for things.

The ns-3 tracing system is designed to work along those lines and is well-integrated with the Attribute and Config subsystems allowing for relatively simple use scenarios.

Overview
========

The ns-3 tracing system is built on the concepts of independent tracing sources and tracing sinks; along with a uniform mechanism for connecting sources to sinks.

Trace sources are entities that can signal events that happen in a simulation and provide access to interesting underlying data.  For example, a trace source could indicate when a packet is received by a net device and provide access to the packet contents for interested trace sinks.  A trace source might also indicate when an iteresting state change happens in a model.  For example, the congestion window of a TCP model is a prime candidate for a trace source.

Trace sources are not useful by themselves; they must be connected to other pieces of code that actually do something useful with the information provided by the source. The entities that consume trace information are called trace sinks.  Trace sources are generators of events and trace sinks are consumers.  This explicit division allows for large numbers of trace sources to be scattered around the system in places which model authors believe might be useful.

There can be zero or more consumers of trace events generated by a trace source. One can think of a trace source as a kind of point-to-multipoint information link. Your code looking for trace events from a particular piece of core code could happily coexist with other code doing something entirely different from the same information.

Unless a user connects a trace sink to one of these sources, nothing is output.  By using the tracing system, both you and other people at the same trace source are getting exactly what they want and only what they want out of the system.  Neither of you are impacting any other user by changing what information is output by the system.  If you happen to add a trace source, your work as a good open-source citizen may allow other users to provide new utilities that are perhaps very useful overall, without making any changes to the ns-3 core.

A Simple Low-Level Example
--------------------------

Let's take a few minutes and walk through a simple tracing example.  We are going to need a little background on Callbacks to understand what is happening in the example, so we have to take a small detour right away.

Callbacks
.........

The goal of the Callback system in ns-3 is to allow one piece of code to call a function (or method in C++) without any specific inter-module dependency. This ultimately means you need some kind of indirection -- you treat the address of the called function as a variable.  This variable is called a pointer-to-function variable.  The relationship between function and pointer-to-function pointer is really no different that that of object and pointer-to-object.

In C the canonical example of a pointer-to-function is a pointer-to-function-returning-integer (PFI).  For a PFI taking one int parameter, this could be declared like,

  int (*pfi)(int arg) = 0;

What you get from this is a variable named simply ``pfi'' that is initialized to the value 0.  If you want to initialize this pointer to something meaningful, you have to have a function with a matching signature.  In this case, you could provide a function that looks like,

  int MyFunction (int arg) {}

If you have this target, you can initialize the variable to point to your function:

  pfi = MyFunction;

You can then call MyFunction indirectly using the more suggestive form of the call,

  int result = (*pfi) (1234);

This is suggestive since it looks like you are dereferencing the function pointer just like you would dereference any pointer.  Typically, however, people take advantage of the fact that the compiler knows what is going on and will just use a shorter form,

  int result = pfi (1234);

This looks like you are calling a function named ``pfi,'' but the compiler is smart enough to know to call through the variable pfi indirectly to the function MyFunction.

Conceptually, this is almost exactly how the tracing system will work. Basically, a trace source is a callback.  When a trace sink expresses interest in receiving trace events, it adds a Callback to a list of Callbacks internally held by the trace source.  When an interesting event happens, the trace source invokes its operator() providing zero or more parameters. The operator() eventually wanders down into the system and does something remarkably like the indirect call you just saw.  It provides zero or more parameters (the call to ``pfi'' above passed one parameter to the target function MyFunction.

The important difference that the tracing system adds is that for each trace source there is an internal list of Callbacks.  Instead of just making one indirect call, a trace source may invoke any number of Callbacks.  When a trace sink expresses interest in notifications from a trace source, it basically just arranges to add its own function to the callback list.

If you are interested in more details about how this is actually arranged in ns-3, feel free to peruse the Callback section of the manual.

Example Code
............

We have provided some code to implement what is really the simplest example of tracing that can be assembled.  You can find this code in the tutorial directory as fourth.cc.  Let's walk through it.

  /* -*- Mode:C++; c-file-style:"gnu"; indent-tabs-mode:nil; -*- */
  /*
   * This program is free software; you can redistribute it and/or modify
   * it under the terms of the GNU General Public License version 2 as
   * published by the Free Software Foundation;
   *
   * This program is distributed in the hope that it will be useful,
   * but WITHOUT ANY WARRANTY; without even the implied warranty of
   * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   * GNU General Public License for more details.
   *
   * You should have received a copy of the GNU General Public License
   * along with this program; if not, write to the Free Software
   * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
   */

  #include "ns3/object.h"
  #include "ns3/uinteger.h"
  #include "ns3/traced-value.h"
  #include "ns3/trace-source-accessor.h"

  #include <iostream>

  using namespace ns3;

Most of this code should be quite familiar to you.  As mentioned above, the trace system makes heavy use of the Object and Attribute systems, so you will need to include them.  The first two includes above bring in the declarations for those systems explicitly.  You could use the core module header, but this illustrates how simple this all really is.

The file, traced-value.h brings in the required declarations for tracing of data that obeys value semantics.  In general, value semantics just means that you can pass the object around, not an address.  In order to use value semantics at all you have to have an object with an associated copy constructor and assignment operator available.  We extend the requirements to talk about the set of operators that are pre-defined for plain-old-data (POD) types.  Operator=, operator++, operator---, operator+, operator==, etc.

What this all really means is that you will be able to trace changes to a C++ object made using those operators.

Since the tracing system is integrated with Attributes, and Attributes work with Objects, there must be an ns-3 Object for the trace source to live in.  The next code snippet declares and defines a simple Object we can work with.

  class MyObject : public Object
  {
  public:
    static TypeId GetTypeId (void)
    {
      static TypeId tid = TypeId ("MyObject")
        .SetParent (Object::GetTypeId ())
        .AddConstructor<MyObject> ()
        .AddTraceSource ("MyInteger",
                         "An integer value to trace.",
                         MakeTraceSourceAccessor (&MyObject::m_myInt))
        ;
      return tid;
    }

    MyObject () {}
    TracedValue<int32_t> m_myInt;
  };

The two important lines of code, above, with respect to tracing are the .AddTraceSource and the TracedValue declaration of m_myInt.

The .AddTraceSource provides the ``hooks'' used for connecting the trace source to the outside world through the config system.  The TracedValue declaration provides the infrastructure that overloads the operators mentioned above and drives the callback process.

  void
  IntTrace (int32_t oldValue, int32_t newValue)
  {
    std::cout << "Traced " << oldValue << " to " << newValue << std::endl;
  }

This is the definition of the trace sink.  It corresponds directly to a callback function.  Once it is connected, this function will be called whenever one of the overloaded operators of the TracedValue is executed.

We have now seen the trace source and the trace sink.  What remains is code to connect the source to the sink.

  int
  main (int argc, char *argv[])
  {
    Ptr<MyObject> myObject = CreateObject<MyObject> ();
    myObject->TraceConnectWithoutContext ("MyInteger", MakeCallback(&IntTrace));

    myObject->m_myInt = 1234;
  }

Here we first create the Object in which the trace source lives.

The next step, the TraceConnectWithoutContext, forms the connection between the trace source and the trace sink.  Notice the MakeCallback template function.  This function does the magic required to create the underlying ns-3 Callback object and associate it with the function IntTrace.  TraceConnect makes the association between your provided function and the overloaded operator() in the traced variable referred to by the ``MyInteger'' Attribute.  After this association is made, the trace source will ``fire'' your provided callback function.

The code to make all of this happen is, of course, non-trivial, but the essence is that you are arranging for something that looks just like the pfi() example above to be called by the trace source.  The declaration of the TracedValue<int32_t> m_myInt; in the Object itself performs the magic needed to provide the overloaded operators (++, ---, etc.) that will use the operator() to actually invoke the Callback with the desired parameters. The .AddTraceSource performs the magic to connect the Callback to the Config system, and TraceConnectWithoutContext performs the magic to connect your function to the trace source, which is specified by Attribute name.

Let's ignore the bit about context for now.

Finally, the line,

   myObject->m_myInt = 1234;

should be interpreted as an invocation of operator= on the member variable m_myInt with the integer 1234 passed as a parameter.

It turns out that this operator is defined (by TracedValue) to execute a callback that returns void and takes two integer values as parameters --- an old value and a new value for the integer in question.  That is exactly the function signature for the callback function we provided --- IntTrace.

To summarize, a trace source is, in essence, a variable that holds a list of callbacks.  A trace sink is a function used as the target of a callback.  The Attribute and object type information systems are used to provide a way to connect trace sources to trace sinks.  The act of ``hitting'' a trace source is executing an operator on the trace source which fires callbacks.  This results in the trace sink callbacks registering interest in the source being called with the parameters provided by the source.

If you now build and run this example,

  ./waf --run fourth

you will see the output from the IntTrace function execute as soon as the trace source is hit:

  Traced 0 to 1234

When we executed the code, myObject->m_myInt = 1234;, the trace source fired and automatically provided the before and after values to the trace sink. The function IntTrace then printed this to the standard output.  No problem.

Using the Config Subsystem to Connect to Trace Sources
------------------------------------------------------

The TraceConnectWithoutContext call shown above in the simple example is actually very rarely used in the system.  More typically, the Config subsystem is used to allow selecting a trace source in the system using what is called a config path.  We saw an example of this in the previous section where we hooked the ``CourseChange'' event when we were playing with third.cc.

Recall that we defined a trace sink to print course change information from the mobility models of our simulation.  It should now be a lot more clear to you what this function is doing.

  void
  CourseChange (std::string context, Ptr<const MobilityModel> model)
  {
    Vector position = model->GetPosition ();
    NS_LOG_UNCOND (context <<
      " x = " << position.x << ", y = " << position.y);
  }

When we connected the ``CourseChange'' trace source to the above trace sink, we used what is called a ``Config Path'' to specify the source when we arranged a connection between the pre-defined trace source and the new trace sink:

  std::ostringstream oss;
  oss <<
    "/NodeList/" << wifiStaNodes.Get (nWifi - 1)->GetId () <<
    "/$ns3::MobilityModel/CourseChange";

  Config::Connect (oss.str (), MakeCallback (&CourseChange));

Let's try and make some sense of what is sometimes considered relatively mysterious code.  For the purposes of discussion, assume that the node number returned by the GetId() is ``7''.  In this case, the path above turns out to be,

  "/NodeList/7/$ns3::MobilityModel/CourseChange"

The last segment of a config path must be an Attribute of an Object.  In fact, if you had a pointer to the Object that has the ``CourseChange'' Attribute handy, you could write this just like we did in the previous example.  You know by now that we typically store pointers to our nodes in a NodeContainer.  In the third.cc example, the Nodes of interest are stored in the wifiStaNodes NodeContainer.  In fact, while putting the path together, we used this container to get a Ptr<Node> which we used to call GetId() on.  We could have used this Ptr<Node> directly to call a connect method directly:

  Ptr<Object> theObject = wifiStaNodes.Get (nWifi - 1);
  theObject->TraceConnectWithoutContext ("CourseChange", MakeCallback (&CourseChange));

In the third.cc example, we actually want an additional ``context'' to be delivered along with the Callback parameters (which will be explained below) so we could actually use the following equivalent code,

  Ptr<Object> theObject = wifiStaNodes.Get (nWifi - 1);
  theObject->TraceConnect ("CourseChange", MakeCallback (&CourseChange));

It turns out that the internal code for Config::ConnectWithoutContext and Config::Connect actually do find a Ptr<Object> and call the appropriate TraceConnect method at the lowest level.

The Config functions take a path that represents a chain of Object pointers.  Each segment of a path corresponds to an Object Attribute.  The last segment is the Attribute of interest, and prior segments must be typed to contain or find Objects.  The Config code parses and ``walks'' this path until it gets to the final segment of the path.  It then interprets the last segment as an Attribute on the last Object it found while walking the path.  The Config functions then call the appropriate TraceConnect or TraceConnectWithoutContext method on the final Object.  Let's see what happens in a bit more detail when the above path is walked.

The leading ``/'' character in the path refers to a so-called namespace.  One of the predefined namespaces in the config system is ``NodeList'' which is a list of all of the nodes in the simulation.  Items in the list are referred to by indices into the list, so ``/NodeList/7'' refers to the eighth node in the list of nodes created during the simulation.  This reference is actually a Ptr<Node> and so is a subclass of an ns3::Object.

As described in the Object Model section of the ns-3 manual, we support Object Aggregation.  This allows us to form an association between different Objects without any programming.  Each Object in an Aggregation can be reached from the other Objects.

The next path segment being walked begins with the ``$'' character.  This indicates to the config system that a GetObject call should be made looking for the type that follows.  It turns out that the MobilityHelper used in third.cc arranges to Aggregate, or associate, a mobility model to each of the wireless Nodes.  When you add the ``$'' you are asking for another Object that has presumably been previously aggregated.  You can think of this as switching pointers from the original Ptr<Node> as specified by ``/NodeList/7'' to its associated mobility model --- which is of type ``$ns3::MobilityModel''.  If you are familiar with GetObject, we have asked the system to do the following:

  Ptr<MobilityModel> mobilityModel = node->GetObject<MobilityModel> ()

We are now at the last Object in the path, so we turn our attention to the Attributes of that Object.  The MobilityModel class defines an Attribute called ``CourseChange.''  You can see this by looking at the source code in src/mobility/mobility-model.cc and searching for ``CourseChange'' in your favorite editor.  You should find,

  .AddTraceSource (``CourseChange'',
                   ``The value of the position and/or velocity vector changed'',
                   MakeTraceSourceAccessor (&MobilityModel::m_courseChangeTrace))

which should look very familiar at this point.

If you look for the corresponding declaration of the underlying traced variable in mobility-model.h you will find

  TracedCallback<Ptr<const MobilityModel> > m_courseChangeTrace;

The type declaration TracedCallback identifies m_courseChangeTrace as a special list of Callbacks that can be hooked using the Config functions described above.

The MobilityModel class is designed to be a base class providing a common interface for all of the specific subclasses.  If you search down to the end of the file, you will see a method defined called NotifyCourseChange():

  void
  MobilityModel::NotifyCourseChange (void) const
  {
    m_courseChangeTrace(this);
  }

Derived classes will call into this method whenever they do a course change to support tracing.  This method invokes operator() on the underlying m_courseChangeTrace, which will, in turn, invoke all of the registered Callbacks, calling all of the trace sinks that have registered interest in the trace source by calling a Config function.

So, in the third.cc example we looked at, whenever a course change is made in one of the RandomWalk2dMobilityModel instances installed, there will be a NotifyCourseChange() call which calls up into the MobilityModel base class.  As seen above, this invokes operator() on m_courseChangeTrace, which in turn, calls any registered trace sinks. In the example, the only code registering an interest was the code that provided the config path.  Therefore, the CourseChange function that was hooked from Node number seven will be the only Callback called.

The final piece of the puzzle is the ``context.''  Recall that we saw an output looking something like the following from third.cc:

  /NodeList/7/$ns3::MobilityModel/CourseChange x = 7.27897, y = 2.22677

The first part of the output is the context.  It is simply the path through which the config code located the trace source.  In the case we have been looking at there can be any number of trace sources in the system corresponding to any number of nodes with mobility models.  There needs to be some way to identify which trace source is actually the one that fired the Callback.  An easy way is to request a trace context when you Config::Connect.

How to Find and Connect Trace Sources, and Discover Callback Signatures
-----------------------------------------------------------------------

The first question that inevitably comes up for new users of the Tracing system is, ``okay, I know that there must be trace sources in the simulation core, but how do I find out what trace sources are available to me''?

The second question is, ``okay, I found a trace source, how do I figure out the config path to use when I connect to it''?

The third question is, ``okay, I found a trace source, how do I figure out what the return type and formal arguments of my callback function need to be''?

The fourth question is, ``okay, I typed that all in and got this incredibly bizarre error message, what in the world does it mean''?

What Trace Sources are Available?
---------------------------------

The answer to this question is found in the ns-3 Doxygen.  Go to the ns-3 web site ``here'' and select the ``Doxygen (stable)'' link ``Documentation'' on the navigation bar to the left side of the page.  Expand the ``Modules'' book in the NS-3 documentation tree a the upper left by clicking the ``+'' box.  Now, expand the ``Core'' book in the tree by clicking its ``+'' box.  You should now see three extremely useful links:

  * The list of all trace sources

  * The list of all attributes

  * The list of all global values

The list of interest to us here is ``the list of all trace sources.''  Go ahead and select that link.  You will see, perhaps not too surprisingly, a list of all of the trace sources available in the ns-3 core.

As an example, scroll down to ns3::MobilityModel.  You will find an entry for

  CourseChange: The value of the position and/or velocity vector changed

You should recognize this as the trace source we used in the third.cc example.  Perusing this list will be helpful.

What String do I use to Connect?
--------------------------------

The easiest way to do this is to grep around in the ns-3 codebase for someone who has already figured it out,  You should always try to copy someone else's working code before you start to write your own.  Try something like:

  find . -name '*.cc' | xargs grep CourseChange | grep Connect

and you may find your answer along with working code.  For example, in this case, ./ns-3-dev/examples/wireless/mixed-wireless.cc has something just waiting for you to use:

  Config::Connect (``/NodeList/*/$ns3::MobilityModel/CourseChange'',
    MakeCallback (&CourseChangeCallback));

If you cannot find any examples in the distribution, you can find this out from the ns-3 Doxygen.  It will probably be simplest just to walk through the ``CourseChanged'' example.

Let's assume that you have just found the ``CourseChanged'' trace source in ``The list of all trace sources'' and you want to figure out how to connect to it.  You know that you are using (again, from the third.cc example) an ns3::RandomWalk2dMobilityModel.  So open the ``Class List'' book in the NS-3 documentation tree by clicking its ``+'' box.  You will now see a list of all of the classes in ns-3.  Scroll down until you see the entry for ns3::RandomWalk2dMobilityModel and follow that link. You should now be looking at the ``ns3::RandomWalk2dMobilityModel Class Reference.''

If you now scroll down to the ``Member Function Documentation'' section, you will see documentation for the GetTypeId function.  You constructed one of these in the simple tracing example above:

    static TypeId GetTypeId (void)
    {
      static TypeId tid = TypeId ("MyObject")
        .SetParent (Object::GetTypeId ())
        .AddConstructor<MyObject> ()
        .AddTraceSource ("MyInteger",
                         "An integer value to trace.",
                         MakeTraceSourceAccessor (&MyObject::m_myInt))
        ;
      return tid;
    }

As mentioned above, this is the bit of code that connected the Config and Attribute systems to the underlying trace source.  This is also the place where you should start looking for information about the way to connect.

You are looking at the same information for the RandomWalk2dMobilityModel; and the information you want is now right there in front of you in the Doxygen:

  This object is accessible through the following paths with Config::Set and Config::Connect:

  /NodeList/[i]/$ns3::MobilityModel/$ns3::RandomWalk2dMobilityModel

The documentation tells you how to get to the RandomWalk2dMobilityModel Object.  Compare the string above with the string we actually used in the example code:

  "/NodeList/7/$ns3::MobilityModel"

The difference is due to the fact that two GetObject calls are implied in the string found in the documentation.  The first, for $ns3::MobilityModel will query the aggregation for the base class.  The second implied GetObject call, for $ns3::RandomWalk2dMobilityModel, is used to ``cast'' the base class to the concrete imlementation class.  The documentation shows both of these operations for you.  It turns out that the actual Attribute you are going to be looking for is found in the base class as we have seen.

Look further down in the GetTypeId doxygen.  You will find,

  No TraceSources defined for this type.
  TraceSources defined in parent class ns3::MobilityModel:

  CourseChange: The value of the position and/or velocity vector changed
  Reimplemented from ns3::MobilityModel

This is exactly what you need to know.  The trace source of interest is found in ns3::MobilityModel (which you knew anyway).  The interesting thing this bit of Doxygen tells you is that you don't need that extra cast in the config path above to get to the concrete class, since the trace source is actually in the base class.  Therefore the additional GetObject is not required and you simply use the path:

  /NodeList/[i]/$ns3::MobilityModel

which perfectly matches the example path:

  /NodeList/7/$ns3::MobilityModel

What Return Value and Formal Arguments?
---------------------------------------

The easiest way to do this is to grep around in the ns-3 codebase for someone who has already figured it out,  You should always try to copy someone else's working code.  Try something like:

  find . -name '*.cc' | xargs grep CourseChange | grep Connect

and you may find your answer along with working code.  For example, in this case, ./ns-3-dev/examples/wireless/mixed-wireless.cc has something just waiting for you to use.  You will find

  Config::Connect (``/NodeList/*/$ns3::MobilityModel/CourseChange'',
    MakeCallback (&CourseChangeCallback));

as a result of your grep.  The MakeCallback should indicate to you that there is a callback function there which you can use.  Sure enough, there is:

  static void
  CourseChangeCallback (std::string path, Ptr<const MobilityModel> model)
  {
    ...
  }

Take my Word for It
...................

If there are no examples to work from, this can be, well, challenging to actually figure out from the source code.

Before embarking on a walkthrough of the code, I'll be kind and just tell you a simple way to figure this out:  The return value of your callback will always be void.  The formal parameter list for a TracedCallback can be found from the template parameter list in the declaration.  Recall that for our current example, this is in mobility-model.h, where we have previously found:

  TracedCallback<Ptr<const MobilityModel> > m_courseChangeTrace;

There is a one-to-one correspondence between the template parameter list in the declaration and the formal arguments of the callback function.  Here, there is one template parameter, which is a Ptr<const MobilityModel>. This tells you that you need a function that returns void and takes a a Ptr<const MobilityModel>.  For example,

  void
  CourseChangeCallback (Ptr<const MobilityModel> model)
  {
    ...
  }

That's all you need if you want to Config::ConnectWithoutContext.  If you want a context, you need to Config::Connect and use a Callback function that takes a string context, then the required argument.

  void
  CourseChangeCallback (std::string path, Ptr<const MobilityModel> model)
  {
    ...
  }

If you want to ensure that your CourseChangeCallback is only visible in your local file, you can add the keyword static and come up with:

  static void
  CourseChangeCallback (std::string path, Ptr<const MobilityModel> model)
  {
    ...
  }

which is exactly what we used in the third.cc example.

The Hard Way
............

This section is entirely optional.  It is going to be a bumpy ride, especially for those unfamiliar with the details of templates.  However, if you get through this, you will have a very good handle on a lot of the ns-3 low level idioms.

So, again, let's figure out what signature of callback function is required for the ``CourseChange'' Attribute.  This is going to be painful, but you only need to do this once.  After you get through this, you will be able to just look at a TracedCallback and understand it.

The first thing we need to look at is the declaration of the trace source. Recall that this is in mobility-model.h, where we have previously found:

  TracedCallback<Ptr<const MobilityModel> > m_courseChangeTrace;

This declaration is for a template.  The template parameter is inside the angle-brackets, so we are really interested in finding out what that TracedCallback<> is.  If you have absolutely no idea where this might be found, grep is your friend.

We are probably going to be interested in some kind of declaration in the ns-3 source, so first change into the src directory.  Then, we know this declaration is going to have to be in some kind of header file, so just grep for it using:

  find . -name '*.h' | xargs grep TracedCallback

You'll see 124 lines fly by (I piped this through wc to see how bad it was). Although that may seem like it, that's not really a lot.  Just pipe the output through more and start scanning through it.  On the first page, you will see some very suspiciously template-looking stuff.

  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::TracedCallback ()
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::ConnectWithoutContext (c ...
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::Connect (const CallbackB ...
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::DisconnectWithoutContext ...
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::Disconnect (const Callba ...
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::operator() (void) const ...
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::operator() (T1 a1) const ...
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::operator() (T1 a1, T2 a2 ...
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::operator() (T1 a1, T2 a2 ...
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::operator() (T1 a1, T2 a2 ...
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::operator() (T1 a1, T2 a2 ...
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::operator() (T1 a1, T2 a2 ...
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::operator() (T1 a1, T2 a2 ...

It turns out that all of this comes from the header file traced-callback.h which sounds very promising.  You can then take a look at mobility-model.h and see that there is a line which confirms this hunch:

  #include "ns3/traced-callback.h"

Of course, you could have gone at this from the other direction and started by looking at the includes in mobility-model.h and noticing the include of traced-callback.h and inferring that this must be the file you want.

In either case, the next step is to take a look at src/core/traced-callback.h in your favorite editor to see what is happening.

You will see a comment at the top of the file that should be comforting:

  An ns3::TracedCallback has almost exactly the same API as a normal ns3::Callback but
  instead of forwarding calls to a single function (as an ns3::Callback normally does),
  it forwards calls to a chain of ns3::Callback.

This should sound very familiar and let you know you are on the right track.

Just after this comment, you will find,

  template<typename T1 = empty, typename T2 = empty,
           typename T3 = empty, typename T4 = empty,
           typename T5 = empty, typename T6 = empty,
           typename T7 = empty, typename T8 = empty>
  class TracedCallback
  {
    ...

This tells you that TracedCallback is a templated class.  It has eight possible type parameters with default values.  Go back and compare this with the declaration you are trying to understand:

  TracedCallback<Ptr<const MobilityModel> > m_courseChangeTrace;

The typename T1 in the templated class declaration corresponds to the Ptr<const MobilityModel> in the declaration above.  All of the other type parameters are left as defaults.  Looking at the constructor really doesn't tell you much.  The one place where you have seen a connection made between your Callback function and the tracing system is in the Connect and ConnectWithoutContext functions.  If you scroll down, you will see a ConnectWithoutContext method here:

  template<typename T1, typename T2,
           typename T3, typename T4,
           typename T5, typename T6,
           typename T7, typename T8>
  void
  TracedCallback<T1,T2,T3,T4,T5,T6,T7,T8>::ConnectWithoutContext ...
  {
    Callback<void,T1,T2,T3,T4,T5,T6,T7,T8> cb;
    cb.Assign (callback);
    m_callbackList.push_back (cb);
  }

You are now in the belly of the beast.  When the template is instantiated for the declaration above, the compiler will replace T1 with Ptr<const MobilityModel>.

  void
  TracedCallback<Ptr<const MobilityModel>::ConnectWithoutContext ... cb
  {
    Callback<void, Ptr<const MobilityModel> > cb;
    cb.Assign (callback);
    m_callbackList.push_back (cb);
  }

You can now see the implementation of everything we've been talking about.  The code creates a Callback of the right type and assigns your function to it.  This is the equivalent of the pfi = MyFunction we discussed at the start of this section.  The code then adds the Callback to the list of Callbacks for this source.  The only thing left is to look at the definition of Callback. Using the same grep trick as we used to find TracedCallback, you will be able to find that the file ./core/callback.h is the one we need to look at.

If you look down through the file, you will see a lot of probably almost incomprehensible template code.  You will eventually come to some Doxygen for the Callback template class, though.  Fortunately, there is some English:

  This class template implements the Functor Design Pattern.
  It is used to declare the type of a Callback:
   - the first non-optional template argument represents
     the return type of the callback.
   - the second optional template argument represents
     the type of the first argument to the callback.
   - the third optional template argument represents
     the type of the second argument to the callback.
   - the fourth optional template argument represents
     the type of the third argument to the callback.
   - the fifth optional template argument represents
     the type of the fourth argument to the callback.
   - the sixth optional template argument represents
     the type of the fifth argument to the callback.

We are trying to figure out what the

    Callback<void, Ptr<const MobilityModel> > cb;

declaration means.  Now we are in a position to understand that the first (non-optional) parameter, void, represents the return type of the Callback.  The second (non-optional) parameter, Ptr<const MobilityModel> represents the first argument to the callback.

The Callback in question is your function to receive the trace events.  From this you can infer that you need a function that returns void and takes a Ptr<const MobilityModel>.  For example,

  void
  CourseChangeCallback (Ptr<const MobilityModel> model)
  {
    ...
  }

That's all you need if you want to Config::ConnectWithoutContext.  If you want a context, you need to Config::Connect and use a Callback function that takes a string context.  This is because the Connect function will provide the context for you.  You'll need:

  void
  CourseChangeCallback (std::string path, Ptr<const MobilityModel> model)
  {
    ...
  }

If you want to ensure that your CourseChangeCallback is only visible in your local file, you can add the keyword static and come up with:

  static void
  CourseChangeCallback (std::string path, Ptr<const MobilityModel> model)
  {
    ...
  }

which is exactly what we used in the third.cc example.  Perhaps you should now go back and reread the previous section (Take My Word for It).

If you are interested in more details regarding the implementation of Callbacks, feel free to take a look at the ns-3 manual.  They are one of the most frequently used constructs in the low-level parts of ns-3. It is, in my opinion, a quite elegant thing.