diff --git a/doc/manual/emulation.texi b/doc/manual/emulation.texi index 8d37cc6a2..75b1d551c 100644 --- a/doc/manual/emulation.texi +++ b/doc/manual/emulation.texi @@ -3,75 +3,374 @@ @anchor{chap:Emulation} ns-3 has been designed for integration into testbed and virtual machine -environments. These two environments require distinct approaches. An -architecture for the first approach, used in testbeds, is shown in the -following figure. +environments. We have addressed this need by providing two kinds of +net devices. The first kind, which we call an @code{Emu} @code {NetDevice} +allows ns-3 simulations to send data on a ``real'' network. The second kind, +called a @code{Tap} @code{NetDevice} allows a ``real'' host to participate +in an ns-3 simulation as if it were one of the simulated nodes. An ns-3 +simulation may be constructed with any combination of simulated, @code{Emu}, +or @code{Tap} devices. + +One of the use-cases we want to support is that of a testbed. A concrete +example of an environment of this kind is the ORBIT testbed. ORBIT is a +laboratory emulator/field trial network arranged as a two dimensional grid of +400 802.11 radio nodes. We integrate with ORBIT by using their ``imaging'' +process to load and run ns-3 simulations on the ORBIT array. We use our +@code{Emu} @code{NetDevice}s to drive the hardware in the testbed and we can +accumulate results either using the ns-3 tracing and logging functions, or the +native ORBIT data gathering techniques. See @uref{http://www.orbit-lab.org/} +for details on the ORBIT testbed. + +A simulation of this kind is shown in the following figure: @float Figure,fig:testbed -@caption{Implementation overview of testbed.} -@image{figures/testbed} +@center @caption{Example Implementation of Testbed Emulation.} +@center @image{figures/testbed, 5in} @end float -You can see that there are separate hosts, each running a subset of a +You can see that there are separate hosts, each running a subset of a ``global'' simulation. Instead of an ns-3 channel connecting the hosts, we use real hardware provided by the testbed. This allows ns-3 applications and protocol stacks attached to a simulation node to communicate over real hardware. -We expect the primary use for this configuration will be to analyze experimental -applications and protocols in a real network environment that includes all of +We expect the primary use for this configuration will be to generate repeatable +experimental results in a real-world network environment that includes all of the ns-3 tracing, loging, visualization and statistics gathering tools. -A concrete example of a testbed environment is the ORBIT testbed which provides -a rectangular array of hosts with various types of WiFi cards and other radios. -We integrate with ORBIT in that we can load and run ns-3 simulations on the ORBIT -array and use our simulator to drive the hardware in the testbed and accumulate -results either using the ns-3 tracing and logging functions, or the native ORBIT -data gathering techniques. - -In what is essentially an inverse configuration, we allow virtual machines -running native applications and protocol stacks to communicate over an ns-3 -simulated channel. This type of scenario is shown in the following figure. +In what can be viewed as essentially an inverse configuration, we allow ``real'' +machines running native applications and protocol stacks to integrate with +an ns-3 simulation. This allows for the simulation of large networks connected +to a real mahince, and also enables virtualization. A simulation of this kind +is shown in the following figure: @float Figure,fig:emulated-channel @caption{Implementation overview of emulated channel.} -@image{figures/emulated-channel} +@image{figures/emulated-channel, 5in} @end float -In this figure, you will see that there is a single host with a number of -virtual machines running on it. An ns-3 simulation is shown running in the -center virtual machine. This simulation has a number of nodes with associated -ns-3 applications and protocol stacks that are talking to an ns-3 channel -through ns-3 net devices. +Here, you will see that there is a single host with a number of virtual machines +running on it. An ns-3 simulation is shown running in the virtual machine shown +in the center of the figure. This simulation has a number of nodes with associated +ns-3 applications and protocol stacks that are talking to an ns-3 channel through +native simulated ns-3 net devices. There are also two virtual machines shown at the far left and far right of the -figure. These VMs are running native applications and native protocol stacks, -and are connected into the simulation by a Tap net device. The user-mode handler -for the Tap device is instantiated in the simulation and attached to a proxy node -that represents the native VM in the simulation. - -The nodes at either end of the simulation VM have only a Tap handler and net -device associated with them. These handlers allow the Tap devices on the native -VMs to behave as an ns-3 net device in the simulation VM. This, in turn, allows -the native software and protocol suites in the native VMs to believe that they -are connected to a real network represented by the simulation. +figure. These VMs are running native (Linux) applications and protocol stacks. +The VM is connected into the simulation by a Linux Tap net device. The user-mode +handler for the Tap device is instantiated in the simulation and attached to a +proxy node that represents the native VM in the simulation. These handlers allow +the Tap devices on the native VMs to behave as if they were ns-3 net devices in +the simulation VM. This, in turn, allows the native software and protocol suites +in the native VMs to believe that they are connected to the simulated ns-3 +channel. We expect the typical use case for this environment will be to analyze the behavior of native applications and protocol suites in the presence of large -simulated ns-3 networks. For example, the channel in the simulation VM could -be an ns-3 Wifi channel and could also be connected to any ns-3 topology. This -would allow native applications or protocol suites to be analyzed in the presence -of very complex simulated networks. +simulated ns-3 networks. @section Behavior -TBD +@subsection Emu Net Device + +The @code{Emu} net device allows a simulation node to send and receive packets +over a real network. The emulated net device relies on a specified interface +being in promiscuous mode. It opens a raw socket and binds to that interface. +We perform MAC spoofing to separate simulation network traffic from other +network traffic that may be flowing to and from the host. + +Normally, the use case for emulated net devices is in collections of small +simulations that connect to the outside world through specific interfaces. +For example, one could construct a number of virtual machines and connect them +via a host-only network. To use the emulated net device, you would need to +set all of the host-only interfaces in promiscuous mode and provide an +appropriate device name, "eth1" for example. + +One could also use the @code{Emu} net device in a testbed situation where the +host on which the simulation is running has a specific interface of interest +which drives the testbed hardware. You would also need to set this specific +interface into promiscuous mode and provide an appropriate device name to the +ns-3 emulated net device. An example of this environment is the ORBIT testbed +as described above. + +The @code{Emu} net device only works if the underlying interface is up and in +promiscuous mode. Packets will be sent out over the device, but we use MAC +spoofing. The MAC addresses will be generated (by default) using the +Organizationally Unique Identifier (OUI) 00:00:00 as a base. This vendor code +is not assigned to any organization and so should not conflict with any real +hardware. + +It is always up to the user to determine that using these MAC addresses is +okay on your network and won't conflict with anything else (including another +simulation using @code{Emu} devices) on your network. If you are using the +emulated net device in separate simulations you must consider global MAC +address assignment issues and ensure that MAC addresses are unique across +all simulations. The emulated net device respects the MAC address provided +in the @code{SetAddress} method so you can do this manually. For larger +simulations, you may want to set the OUI in the MAC address allocation function. + +IP addresses corresponding to the emulated net devices are the addresses +generated in the simulation, which are generated in the usual way via helper +functions. Since we are using MAC spoofing, there will not be a conflict +between ns-3 network stacks and any native network stacks. + +The emulated net device comes with a helper function as all ns-3 devices do. +One unique aspect is that there is no channel associated with the underlying +medium. We really have no idea what this external medium is, and so have not +made an effort to model it abstractly. The primary thing to be aware of is the +implication this has for static global routing. The global router module +attempts to walk the channels looking for adjacent networks. Since there +is no channel, the global router will be unable to do this and you must then +use a dynamic routing protocol such as OLSR to include routing in +@code{Emu}-based networks. + +@subsection Tap Net Device + +The @code{Tap} Net Device is scheduled for inclusion in ns-3.4 at the writing +of this section. We will include details as soon as the @code{Tap} device is +merged. @section Usage -TBD +@subsection Emu Net Device + +The usage of the @code{Emu} net device is straightforward once the network of +simulations has been configured. Since most of the work involved in working +with this device is in network configuration before even starting a simulation, +you may want to take a moment to review a couple of HOWTO pages on the ns-3 wiki +that describe how to set up a virtual test network using VMware and how to run +a set of example (client server) simulations that use @code{Emu} net devices. + +@uref{http://www.nsnam.org/wiki/index.php/HOWTO_use_VMware_to_set_up_virtual_networks_(Windows)} +@uref{http://www.nsnam.org/wiki/index.php/HOWTO_use_ns-3_scripts_to_drive_real_hardware_(experimental)} + +Once you are over the configuration hurdle, the script changes required to use +an @code{Emu} device are trivial. The main structural difference is that you +will need to create an ns-3 simulation script for each node. In the case of +the HOWTOs above, there is one client script and one server script. The only +``challenge'' is to get the addresses set correctly. + +Just as with all other ns-3 net devices, we provide a helper class for the +@code{Emu} net device. The following code snippet illustrates how one would +declare an EmuHelper and use it to set the ``DeviceName'' attribute to ``eth1'' +and install @code{Emu} devices on a group of nodes. You would do this on both +the client and server side in the case of the HOWTO seen above. + +@verbatim + EmuHelper emu; + emu.SetAttribute ("DeviceName", StringValue ("eth1")); + NetDeviceContainer d = emu.Install (n); +@end verbatim + +The only other change that may be required is to make sure that the address +spaces (MAC and IP) on the client and server simulations are compatible. First +the MAC address is set to a unique well-known value in both places (illustrated +here for one side). + +@verbatim + // + // We've got the devices in place. Since we're using MAC address + // spoofing under the sheets, we need to make sure that the MAC addresses + // we have assigned to our devices are unique. Ns-3 will happily + // automatically assign the same MAC addresses to the devices in both halves + // of our two-script pair, so let's go ahead and just manually change them + // to something we ensure is unique. + // + Ptr nd = d.Get (0); + Ptr ed = nd->GetObject (); + ed->SetAddress ("00:00:00:00:00:02"); +@end verbatim + +And then the IP address of the client or serveris set in the usual way using +helpers. + +@verbatim + // + // We've got the "hardware" in place. Now we need to add IP addresses. + // This is the server half of a two-script pair. We need to make sure + // that the addressing in both of these applications is consistent, so + // we use provide an initial address in both cases. Here, the client + // will reside on one machine running ns-3 with one node having ns-3 + // with IP address "10.1.1.2" and talk to a server script running in + // another ns-3 on another computer that has an ns-3 node with IP + // address "10.1.1.3" + // + Ipv4AddressHelper ipv4; + ipv4.SetBase ("10.1.1.0", "255.255.255.0", "0.0.0.2"); + Ipv4InterfaceContainer i = ipv4.Assign (d); +@end verbatim + +You will use application helpers to generate traffic exactly as you do in any +ns-3 simulation script. Note that the server address shown below in a snippet +from the client, must correspond to the IP address assigned to the server node +similarly to the snippet above. + +@verbatim + uint32_t packetSize = 1024; + uint32_t maxPacketCount = 2000; + Time interPacketInterval = Seconds (0.001); + UdpEchoClientHelper client ("10.1.1.3", 9); + client.SetAttribute ("MaxPackets", UintegerValue (maxPacketCount)); + client.SetAttribute ("Interval", TimeValue (interPacketInterval)); + client.SetAttribute ("PacketSize", UintegerValue (packetSize)); + ApplicationContainer apps = client.Install (n.Get (0)); + apps.Start (Seconds (1.0)); + apps.Stop (Seconds (2.0)); +@end verbatim + +The @code{Emu} net device and helper provide access to ASCII and pcap tracing +functionality just as other ns-3 net devices to. You enable tracing similarly +to these other net devices: + +@verbatim + EmuHelper::EnablePcapAll ("emu-udp-echo-client"); +@end verbatim + +To see an example of a client script using the @code{Emu} net device, see +@code{examples/emu-udp-echo-client.cc} and @code{examples/emu-udp-echo-server.cc} +in the repository @uref{http://code.nsnam.org/craigdo/ns-3-emu/}. + +@subsection Tap Net Device + +The @code{Tap} Net Device is scheduled for inclusion in ns-3.4 at the writing +of this section. We will include details as soon as the @code{Tap} device is +merged. @section Implementation -TBD +Perhaps the most unusual part of the @code{Emu} and @code{Tap} device +implementation relates to the requirement for executing some of the code +with super-user permissions. Rather than force the user to execute the entire +simulation as root, we provide a small ``creator'' program that runs as root +and does any required high-permission sockets work. + +We do a similar thing for both the @code{Emu} and the @code{Tap} devices. +The high-level view is that the @code{CreateSocket} method creates a local +interprocess (Unix) socket, forks, and executes the small creation program. +The small program, which runs as suid root, creates a raw socket and sends +back the raw socket file descriptor over the Unix socket that is passed to +it as a parameter. The raw socket is passed as a control message (sometimes +called ancillary data) of type SCM_RIGHTS. + +@subsection Emu Net Device + +The @code{Emu} net device uses the ns-3 threading and multithreaded real-time +scheduler extensions. The interesting work in the @code{Emu} device is done +when the net device is started (@code{EmuNetDevice::StartDevice ()}). An +attribute (``Start'') provides a simulation time at which to spin up the +net device. At this specified time (which defaults to t=0), the socket +creation function is called and executes as described above. You may also +specify a time at which to stop the device using the ``Stop'' attribute. + +Once the (promiscuous mode) socket is created, we bind it to an interface name +also provided as an attribute (``DeviceName'') that is stored internally as +@code{m_deviceName}: + +@verbatim + struct ifreq ifr; + bzero (&ifr, sizeof(ifr)); + strncpy ((char *)ifr.ifr_name, m_deviceName.c_str (), IFNAMSIZ); + + int32_t rc = ioctl (m_sock, SIOCGIFINDEX, &ifr); + + struct sockaddr_ll ll; + bzero (&ll, sizeof(ll)); + + ll.sll_family = AF_PACKET; + ll.sll_ifindex = m_sll_ifindex; + ll.sll_protocol = htons(ETH_P_ALL); + + rc = bind (m_sock, (struct sockaddr *)&ll, sizeof (ll)); +@end verbatim + +After the promiscuous raw socket is set up, a separate thread is spawned to do +reads from that socket and the link state is set to @code{Up}. + +@verbatim + m_readThread = Create ( + MakeCallback (&EmuNetDevice::ReadThread, this)); + m_readThread->Start (); + + NotifyLinkUp (); +@end verbatim + +The @code{EmuNetDevice::ReadThread} function basically just sits in an infinite +loop reading from the promiscuous mode raw socket and scheduling packet +receptions using the real-time simulator extensions. + +@verbatim + for (;;) + { + ... + + len = recvfrom (m_sock, buf, bufferSize, 0, (struct sockaddr *)&addr, + &addrSize); + + ... + + DynamicCast (Simulator::GetImplementation ())-> + ScheduleRealtimeNow ( + MakeEvent (&EmuNetDevice::ForwardUp, this, buf, len)); + + ... + } +@end verbatim + +The line starting with our templated DynamicCast function probably deserves a +comment. It gains access to the simulator implementation object using +the @code{Simulator::GetImplementation} method and then casts to the real-time +simulator implementation to use the real-time schedule method +@code{ScheduleRealtimeNow}. This function will cause a handler for the newly +received packet to be scheduled for execution at the current real time clock +value. This will, in turn cause the simulation clock to be advanced to that +real time value when the scheduled event (@code{EmuNetDevice::ForwardUp}) is +fired. + +The @code{ForwardUp} function operates as most other similar ns-3 net device +methods do. The packet is first filtered based on the destination address. In +the case of the @code{Emu} device, the MAC destination address will be the +address of the @code{Emu} device and not the hardware address of the real +device. Headers are then stripped off and the trace hooks are hit. Finally, +the packet is passed up the ns-3 protocol stack using the receive callback +function of the net device. + +Sending a packet is equally straightforward as shown below. The first thing +we do is to add the ethernet header and trailer to the ns-3 @code{Packet} we +are sending. The source address corresponds to the address of the @code{Emu} +device and not the underlying native device MAC address. This is where the +MAC address spoofing is done. The trailer is added and we enqueue and dequeue +the packet from the net device queue to hit the trace hooks. + +@verbatim + header.SetSource (source); + header.SetDestination (destination); + header.SetLengthType (packet->GetSize ()); + packet->AddHeader (header); + + EthernetTrailer trailer; + trailer.CalcFcs (packet); + packet->AddTrailer (trailer); + + m_queue->Enqueue (packet); + packet = m_queue->Dequeue (); + + struct sockaddr_ll ll; + bzero (&ll, sizeof (ll)); + + ll.sll_family = AF_PACKET; + ll.sll_ifindex = m_sll_ifindex; + ll.sll_protocol = htons(ETH_P_ALL); + + rc = sendto (m_sock, packet->PeekData (), packet->GetSize (), 0, + reinterpret_cast (&ll), sizeof (ll)); +@end verbatim +Finally, we simply send the packet to the raw socket which puts it out on the +real network. + +@subsection Tap Net Device + +The @code{Tap} Net Device is scheduled for inclusion in ns-3.4 at the writing +of this section. We will include details as soon as the @code{Tap} device is +merged. +