Emulation
*********

ns-3 has been designed for integration into testbed and virtual machine environments.  We have addressed this need by providing two kinds of net devices.  The first kind, which we call an Emu NetDevice allows ns-3 simulations to send data on a ``real'' network.  The second kind, called a Tap NetDevice allows a ``real'' host to participate in an ns-3 simulation as if it were one of the simulated nodes.  An ns-3 simulation may be constructed with any combination of simulated, Emu, or Tap devices.

One of the use-cases we want to support is that of a testbed.  A concrete example of an environment of this kind is the ORBIT testbed.  ORBIT is a laboratory emulator/field trial network arranged as a two dimensional grid of 400 802.11 radio nodes.  We integrate with ORBIT by using their ``imaging'' process to load and run ns-3 simulations on the ORBIT array.  We use our Emu NetDevices to drive the hardware in the testbed and we can accumulate results either using the ns-3 tracing and logging functions, or the native ORBIT data gathering techniques.  See http://www.orbit-lab.org/ for details on the ORBIT testbed.

A simulation of this kind is shown in the following figure:

Figure: Example Implementation of Testbed Emulation.

You can see that there are separate hosts, each running a subset of a ``global'' simulation.  Instead of an ns-3 channel connecting the hosts, we use real hardware provided by the testbed.  This allows ns-3 applications and protocol stacks attached to a simulation node to communicate over real hardware.

We expect the primary use for this configuration will be to generate repeatable experimental results in a real-world network environment that includes all of the ns-3 tracing, loging, visualization and statistics gathering tools.

In what can be viewed as essentially an inverse configuration, we allow ``real'' machines running native applications and protocol stacks to integrate with an ns-3 simulation.  This allows for the simulation of large networks connected to a real mahince, and also enables virtualization.  A simulation of this kind is shown in the following figure:

Figure: Implementation overview of emulated channel.

Here, you will see that there is a single host with a number of virtual machines running on it.  An ns-3 simulation is shown running in the virtual machine shown in the center of the figure.  This simulation has a number of nodes with associated ns-3 applications and protocol stacks that are talking to an ns-3 channel through native simulated ns-3 net devices.

There are also two virtual machines shown at the far left and far right of the figure.  These VMs are running native (Linux) applications and protocol stacks. The VM is connected into the simulation by a Linux Tap net device.  The user-mode handler for the Tap device is instantiated in the simulation and attached to a proxy node that represents the native VM in the simulation.  These handlers allow the Tap devices on the native VMs to behave as if they were ns-3 net devices in the simulation VM.  This, in turn, allows the native software and protocol suites in the native VMs to believe that they are connected to the simulated ns-3 channel.

We expect the typical use case for this environment will be to analyze the behavior of native applications and protocol suites in the presence of large simulated ns-3 networks.

Behavior
========

Emu Net Device
--------------

The Emu net device allows a simulation node to send and receive packets over a real network.  The emulated net device relies on a specified interface being in promiscuous mode.  It opens a raw socket and binds to that interface. We perform MAC spoofing to separate simulation network traffic from other network traffic that may be flowing to and from the host.

Normally, the use case for emulated net devices is in collections of small simulations that connect to the outside world through specific interfaces. For example, one could construct a number of virtual machines and connect them via a host-only network.  To use the emulated net device, you would need to set all of the host-only interfaces in promiscuous mode and provide an appropriate device name, "eth1" for example.

One could also use the Emu net device in a testbed situation where the host on which the simulation is running has a specific interface of interest which  drives the testbed hardware.  You would also need to set this specific interface into promiscuous mode and provide an appropriate device name to the ns-3 emulated net device.  An example of this environment is the ORBIT testbed as described above.

The Emu net device only works if the underlying interface is up and in promiscuous mode.  Packets will be sent out over the device, but we use MAC spoofing.  The MAC addresses will be generated (by default) using the Organizationally Unique Identifier (OUI) 00:00:00 as a base.  This vendor code is not assigned to any organization and so should not conflict with any real hardware.

It is always up to the user to determine that using these MAC addresses is okay on your network and won't conflict with anything else (including another simulation using Emu devices) on your network.  If you are using the emulated net device in separate simulations you must consider global MAC address assignment issues and ensure that MAC addresses are unique across all simulations.  The emulated net device respects the MAC address provided in the SetAddress method so you can do this manually.  For larger simulations, you may want to set the OUI in the MAC address allocation function.

IP addresses corresponding to the emulated net devices are the addresses generated in the simulation, which are generated in the usual way via helper functions.  Since we are using MAC spoofing, there will not be a conflict between ns-3 network stacks and any native network stacks.

The emulated net device comes with a helper function as all ns-3 devices do. One unique aspect is that there is no channel associated with the underlying medium.  We really have no idea what this external medium is, and so have not made an effort to model it abstractly.  The primary thing to be aware of is the implication this has for static global routing.  The global router module attempts to walk the channels looking for adjacent networks.  Since there is no channel, the global router will be unable to do this and you must then use a dynamic routing protocol such as OLSR to include routing in Emu-based networks.

Tap Net Device
--------------

The Tap Net Device is scheduled for inclusion in ns-3.4 at the writing of this section.  We will include details as soon as the Tap device is merged.

Usage
=====

Emu Net Device
--------------

The usage of the Emu net device is straightforward once the network of simulations has been configured.  Since most of the work involved in working with this device is in network configuration before even starting a simulation, you may want to take a moment to review a couple of HOWTO pages on the ns-3 wiki that describe how to set up a virtual test network using VMware and how to run a set of example (client server) simulations that use Emu net devices.

http://www.nsnam.org/wiki/index.php/HOWTO_use_VMware_to_set_up_virtual_networks_(Windows) http://www.nsnam.org/wiki/index.php/HOWTO_use_ns-3_scripts_to_drive_real_hardware_(experimental)

Once you are over the configuration hurdle, the script changes required to use an Emu device are trivial.  The main structural difference is that you will need to create an ns-3 simulation script for each node.  In the case of the HOWTOs above, there is one client script and one server script.  The only ``challenge'' is to get the addresses set correctly.

Just as with all other ns-3 net devices, we provide a helper class for the Emu net device.  The following code snippet illustrates how one would declare an EmuHelper and use it to set the ``DeviceName'' attribute to ``eth1'' and install Emu devices on a group of nodes.  You would do this on both the client and server side in the case of the HOWTO seen above.

  EmuHelper emu;
  emu.SetAttribute ("DeviceName", StringValue ("eth1"));
  NetDeviceContainer d = emu.Install (n);

The only other change that may be required is to make sure that the address spaces (MAC and IP) on the client and server simulations are compatible.  First the MAC address is set to a unique well-known value in both places (illustrated here for one side).

  //
  // We've got the devices in place.  Since we're using MAC address
  // spoofing under the sheets, we need to make sure that the MAC addresses
  // we have assigned to our devices are unique.  Ns-3 will happily
  // automatically assign the same MAC addresses to the devices in both halves
  // of our two-script pair, so let's go ahead and just manually change them
  // to something we ensure is unique.
  //
  Ptr<NetDevice> nd = d.Get (0);
  Ptr<EmuNetDevice> ed = nd->GetObject<EmuNetDevice> ();
  ed->SetAddress ("00:00:00:00:00:02");

And then the IP address of the client or serveris set in the usual way using helpers.

  //
  // We've got the "hardware" in place.  Now we need to add IP addresses.
  // This is the server half of a two-script pair.  We need to make sure
  // that the addressing in both of these applications is consistent, so
  // we use provide an initial address in both cases.  Here, the client
  // will reside on one machine running ns-3 with one node having ns-3
  // with IP address "10.1.1.2" and talk to a server script running in
  // another ns-3 on another computer that has an ns-3 node with IP
  // address "10.1.1.3"
  //
  Ipv4AddressHelper ipv4;
  ipv4.SetBase ("10.1.1.0", "255.255.255.0", "0.0.0.2");
  Ipv4InterfaceContainer i = ipv4.Assign (d);

You will use application helpers to generate traffic exactly as you do in any ns-3 simulation script.  Note that the server address shown below in a snippet from the client, must correspond to the IP address assigned to the server node similarly to the snippet above.

  uint32_t packetSize = 1024;
  uint32_t maxPacketCount = 2000;
  Time interPacketInterval = Seconds (0.001);
  UdpEchoClientHelper client ("10.1.1.3", 9);
  client.SetAttribute ("MaxPackets", UintegerValue (maxPacketCount));
  client.SetAttribute ("Interval", TimeValue (interPacketInterval));
  client.SetAttribute ("PacketSize", UintegerValue (packetSize));
  ApplicationContainer apps = client.Install (n.Get (0));
  apps.Start (Seconds (1.0));
  apps.Stop (Seconds (2.0));

The Emu net device and helper provide access to ASCII and pcap tracing functionality just as other ns-3 net devices to.  You enable tracing similarly to these other net devices:

  EmuHelper::EnablePcapAll ("emu-udp-echo-client");

To see an example of a client script using the Emu net device, see examples/emu-udp-echo-client.cc and examples/emu-udp-echo-server.cc in the repository http://code.nsnam.org/craigdo/ns-3-emu/.

Tap Net Device
--------------

The Tap Net Device is scheduled for inclusion in ns-3.4 at the writing of this section.  We will include details as soon as the Tap device is merged.

Implementation
==============

Perhaps the most unusual part of the Emu and Tap device implementation relates to the requirement for executing some of the code with super-user permissions.  Rather than force the user to execute the entire simulation as root, we provide a small ``creator'' program that runs as root and does any required high-permission sockets work.

We do a similar thing for both the Emu and the Tap devices. The high-level view is that the CreateSocket method creates a local interprocess (Unix) socket, forks, and executes the small creation program. The small program, which runs as suid root, creates a raw socket and sends back the raw socket file descriptor over the Unix socket that is passed to it as a parameter.  The raw socket is passed as a control message (sometimes called ancillary data) of type SCM_RIGHTS.

Emu Net Device
--------------

The Emu net device uses the ns-3 threading and multithreaded real-time scheduler extensions.  The interesting work in the Emu device is done when the net device is started (EmuNetDevice::StartDevice ()).  An attribute (``Start'') provides a simulation time at which to spin up the net device.  At this specified time (which defaults to t=0), the socket creation function is called and executes as described above.  You may also specify a time at which to stop the device using the ``Stop'' attribute.

Once the (promiscuous mode) socket is created, we bind it to an interface name also provided as an attribute (``DeviceName'') that is stored internally as m_deviceName:

  struct ifreq ifr;
  bzero (&ifr, sizeof(ifr));
  strncpy ((char *)ifr.ifr_name, m_deviceName.c_str (), IFNAMSIZ);

  int32_t rc = ioctl (m_sock, SIOCGIFINDEX, &ifr);

  struct sockaddr_ll ll;
  bzero (&ll, sizeof(ll));

  ll.sll_family = AF_PACKET;
  ll.sll_ifindex = m_sll_ifindex;
  ll.sll_protocol = htons(ETH_P_ALL);

  rc = bind (m_sock, (struct sockaddr *)&ll, sizeof (ll));

After the promiscuous raw socket is set up, a separate thread is spawned to do reads from that socket and the link state is set to Up.

  m_readThread = Create<SystemThread> (
    MakeCallback (&EmuNetDevice::ReadThread, this));
  m_readThread->Start ();

  NotifyLinkUp ();

The EmuNetDevice::ReadThread function basically just sits in an infinite loop reading from the promiscuous mode raw socket and scheduling packet receptions using the real-time simulator extensions.

  for (;;)
    {
      ...

      len = recvfrom (m_sock, buf, bufferSize, 0, (struct sockaddr *)&addr,
        &addrSize);

      ...

      DynamicCast<RealtimeSimulatorImpl> (Simulator::GetImplementation ())->
        ScheduleRealtimeNow (
          MakeEvent (&EmuNetDevice::ForwardUp, this, buf, len));

      ...
    }

The line starting with our templated DynamicCast function probably deserves a comment.  It gains access to the simulator implementation object using the Simulator::GetImplementation method and then casts to the real-time simulator implementation to use the real-time schedule method ScheduleRealtimeNow.  This function will cause a handler for the  newly received packet to be scheduled for execution at the current real time clock value.  This will, in turn cause the simulation clock to be advanced to that real time value when the scheduled event (EmuNetDevice::ForwardUp) is fired.

The ForwardUp function operates as most other similar ns-3 net device methods do.  The packet is first filtered based on the destination address.  In the case of the Emu device, the MAC destination address will be the address of the Emu device and not the hardware address of the real device.  Headers are then stripped off and the trace hooks are hit.  Finally, the packet is passed up the ns-3 protocol stack using the receive callback function of the net device.

Sending a packet is equally straightforward as shown below.  The first thing we do is to add the ethernet header and trailer to the ns-3 Packet we are sending.  The source address corresponds to the address of the Emu device and not the underlying native device MAC address.  This is where the MAC address spoofing is done.  The trailer is added and we enqueue and dequeue the packet from the net device queue to hit the trace hooks.

  header.SetSource (source);
  header.SetDestination (destination);
  header.SetLengthType (packet->GetSize ());
  packet->AddHeader (header);

  EthernetTrailer trailer;
  trailer.CalcFcs (packet);
  packet->AddTrailer (trailer);

  m_queue->Enqueue (packet);
  packet = m_queue->Dequeue ();

  struct sockaddr_ll ll;
  bzero (&ll, sizeof (ll));

  ll.sll_family = AF_PACKET;
  ll.sll_ifindex = m_sll_ifindex;
  ll.sll_protocol = htons(ETH_P_ALL);

  rc = sendto (m_sock, packet->PeekData (), packet->GetSize (), 0,
    reinterpret_cast<struct sockaddr *> (&ll), sizeof (ll));

Finally, we simply send the packet to the raw socket which puts it out on the real network.

Tap Net Device
--------------

The Tap Net Device is scheduled for inclusion in ns-3.4 at the writing of this section.  We will include details as soon as the Tap device is merged.