diff --git a/doc/models/Makefile b/doc/models/Makefile index c846969bf..ae9e7263b 100644 --- a/doc/models/Makefile +++ b/doc/models/Makefile @@ -38,6 +38,7 @@ SOURCES = \ $(SRC)/mpi/doc/distributed.rst \ $(SRC)/energy/doc/energy.rst \ $(SRC)/fd-net-device/doc/fd-net-device.rst \ + $(SRC)/fd-net-device/doc/dpdk-net-device.rst \ $(SRC)/fd-net-device/doc/netmap-net-device.rst \ $(SRC)/tap-bridge/doc/tap.rst \ $(SRC)/mesh/doc/source/mesh.rst \ diff --git a/doc/models/source/emulation-overview.rst b/doc/models/source/emulation-overview.rst index 02d983578..56b7ad61b 100644 --- a/doc/models/source/emulation-overview.rst +++ b/doc/models/source/emulation-overview.rst @@ -80,8 +80,15 @@ We expect the typical use case for this environment will be to analyze the behavior of native applications and protocol suites in the presence of large simulated |ns3| networks. +The basic testbed mode of emulation uses raw sockets. Two other variants +(netmap-based and DPDK-based emulation) have been recently added; these +make use of more recent network interface cards that make use of +directly-mapped memory capabilities to improve packet processing efficiency. + For more details: * :doc:`FdNetDevice ` chapter. +* :doc:`NetmapNetDevice ` chapter. +* :doc:`DpdkNetDevice ` chapter. * :doc:`TapBridge ` chapter. diff --git a/doc/models/source/index.rst b/doc/models/source/index.rst index 2bbeaa19b..3c074351e 100644 --- a/doc/models/source/index.rst +++ b/doc/models/source/index.rst @@ -26,6 +26,7 @@ This document is written in `reStructuredText `_ for detailed instructions. -To allocate hugepages at runtime, edit ``/etc/default/grub``, and following to ``GRUB_CMDLINE_LINUX_DEFAULT``: +To allocate hugepages at runtime, write a value such as '256' to the following: + +.. sourcecode:: text + + echo 256 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages + +To allocate hugepages at boot time, edit ``/etc/default/grub``, and following to ``GRUB_CMDLINE_LINUX_DEFAULT``: .. sourcecode:: text @@ -204,7 +210,8 @@ To check allocation of hugepages, run: You will see the number of hugepages allocated, they should be equal to the number you used above. -Once the hugepage memory is reserved, to make the memory available for DPDK use, perform the following steps: +Once the hugepage memory is reserved (at either runtime or boot time), +to make the memory available for DPDK use, perform the following steps: .. sourcecode:: text @@ -221,7 +228,17 @@ The mount point can be made permanent across reboots, by adding the following li Usage ***** -``DpdkNetDevice`` uses the ``EmuFdNetDeviceHelper`` class provided in the ``fd-net-device/helper`` directory. The ``EmuFdNetDeviceHelper`` will allows us to set ``dpdkMode`` which will launch the ``DpdkNetDevice`` instead of the ``FdNetDevice``. +The status of DPDK support is shown in the output of ``./waf configure``. If +it is found, a user should see: + +.. sourcecode:: text + + DPDK NetDevice : enabled + +``DpdkNetDevice`` does not use a dedicated helper class, but reuses the +``EmuFdNetDeviceHelper`` class provided in the ``fd-net-device/helper`` +directory. The ``EmuFdNetDeviceHelper`` will allows us to set ``dpdkMode`` +which will launch the ``DpdkNetDevice`` instead of the ``FdNetDevice``. .. sourcecode:: text @@ -302,7 +319,6 @@ Output As ``DpdkNetDevice`` is inherited from ``FdNetDevice``, all the output methods provided by ``FdNetDevice`` can be used directly. - Examples ======== diff --git a/src/fd-net-device/doc/netmap-net-device.rst b/src/fd-net-device/doc/netmap-net-device.rst index f5d64f51b..68bde9bc3 100644 --- a/src/fd-net-device/doc/netmap-net-device.rst +++ b/src/fd-net-device/doc/netmap-net-device.rst @@ -1,5 +1,5 @@ Netmap NetDevice -------------------------- +---------------- .. include:: replace.txt .. highlight:: cpp @@ -14,44 +14,141 @@ from the ``FdNetDevice`` which is able to read and write traffic using a netmap This netmap file descriptor must be associated to a real ethernet device in the host machine. The ``NetmapNetDeviceHelper`` class supports the configuration of a ``NetmapNetDevice``. -netmap is a fast packets processing that bypasses the host networking stack and gains -direct access to network device. +netmap is a fast packet processing capability that bypasses the +host networking stack and gains direct access to network device. +netmap was developed by Luigi Rizzo [Rizzo2012]_ and is maintained as +an open source project on GitHub at https://github.com/luigirizzo/netmap. + +The NetmapNetDevice for ns-3 [Imputato2019]_ was developed by Pasquale Imputato in the 2017-19 timeframe. The use of NetmapNetDevice requires that the +host system has netmap support (and for best performance, the drivers +must support netmap and must be using a netmap-enabled device driver). Users +can expect that emulation support using Netmap will support higher packets +per second than emulation using FdNetDevice with raw sockets (which pass +through the Linux networking kernel). + +.. [Rizzo2012] Luigi Rizzo, "netmap: A Novel Framework for Fast Packet I/O", Proceedings of 2012 USENIX Annual Techincal Conference, June 2012. + +.. [Imputato2019] Pasquale Imputato, Stefano Avallone, Enhancing the fidelity of network emulation through direct access to device buffers, Journal of Network and Computer Applications, Volume 130, 2019, Pages 63-75, (http://www.sciencedirect.com/science/article/pii/S1084804519300220) Model Description ***************** - - Design ====== +Because netmap uses file descriptor based communication to interact with the +real device, the straightforward approach to design a new ``NetDevice`` around +netmap is to have it inherit from the existing ``FdNetDevice`` and implement +a specialized version of the operations specific to netmap. +The operations that require a specialized implementation are the +initialization, because the NIC has to be put in netmap mode, and the +read/write methods, which have to make use of the netmap API to coordinate +the exchange of packets with the netmap rings. +In the initialization stage, the network device is switched to netmap mode, +so that ns-3 is able to send/receive packets to/from the +real network device by writing/reading them to/from the netmap rings. +Following the design of the ``FdNetDevice``, a separate reading thread is +started during the initialization. The task of the reading thread is +to wait for new incoming packets in the netmap receiver rings, in order +to schedule the events of packet reception. In +the initialization of the ``NetmapNetDevice``, an additional thread, +the sync thread, is started. The sync thread is required because, in order +to reduce the cost of the system calls, netmap does not automatically +transfer a packet written to a slot of the netmap ring to the transmission +ring or to the installed qdisc. It is up to the user process to +periodically request a synchronization of the netmap ring. Therefore, +the purpose of the sync thread is to periodically make a TXSYNC ioctl +request, so that pending packets in the netmap ring are transferred to +the transmission ring, if in native mode, or to the installed qdisc, if in +generic mode. Also, as described further below, the +sync thread is exploited to perform flow control and notify the BQL library +about the +amount of bytes that have been transferred to the network device. + +The read method is called by the reading thread to retrieve new incoming +packets stored in the netmap receiver ring and pass them to the appropriate +ns-3 protocol handler for further processing within the simulator’s network +stack. After retrieving packets, the reading thread also synchronizes +the netmap receiver ring, so that the retrieved packets can be removed +from the netmap receiver ring. + +The ``NetmapNetDevice`` also specializes the write method, i.e., the method +used to transmit a packet received from the upper layer (the ns-3 traffic +control layer). The write method uses the netmap API to write the packet to a +free slot in the netmap +transmission ring. After writing a packet, the write method checks whether +there is enough room in the netmap transmission ring for another packet. +If not, the ``NetmapNetDevice`` stops its queue so that the ns-3 traffic +control layer does not attempt to send a packet that could not be stored in +the netmap transmission ring. + +A stopped ``NetmapNetDevice`` queue needs to be restarted as soon as some +room is made in the netmap transmission ring. The sync thread can be exploited +for this purpose, given that it periodically synchronizes the netmap +transmission ring. In particular, the sync thread also checks the number of +free slots in the netmap transmission ring in case the ``NetmapNetDevice`` +queue is stopped. If the number of free slots exceeds a configurable value, +the sync thread restarts the ``NetmapNetDevice`` +queue and wakes the associated ns-3 qdisc. The NetmapNetDevice also supports +BQL: the write method notifies the BQL library of the amount of bytes that +have been written to the netmap transmission ring, while the sync thread +notifies the BQL library of the amount of bytes that have been removed from +the netmap transmission ring and transferred to the NIC since the previous +notification. Scope and Limitations ===================== -The main scope of ``NetmapNetDevice`` is to support the flow-control between the physical device and -the upper layer and using at best the computational resources to process packets. - +The main scope of ``NetmapNetDevice`` is to support the flow-control between +the physical device and the upper layer and using at best the computational +resources to process packets. However, the (Linux) system and network +device must support netmap to make use of this feature. Usage ***** +The installation of netmap itself on a host machine is out of scope for +this document. Refer to the `netmap GitHub README `_ for instructions. +The ns-3 netmap code has only been tested on Linux; it is not clear whether +other operating systems can be supported. + +If ns-3 is able to detect the presence of netmap on the system, it will +report that: + +.. sourcecode:: text + + Netmap emulation FdNetDevice : not enabled + +If not, it will report: + +.. sourcecode:: text + + Netmap emulation FdNetDevice : not enabled (needs net/netmap_user.h) + +To run FdNetDevice-enabled simulations, one must pass the ``--enable-sudo`` +option to ``./waf configure``, or else run the simulations with root +privileges. Helpers ======= - +ns-3 netmap support uses a ``NetMapNetDeviceHelper`` helper object to +install the NetmapNetDevice. In other respects, the API and use is similar +to that of the ``EmuFdNetDeviceHelper``. Attributes ========== - +The ``NetmapNetDevice`` does not have any specialized attributes, but +supports the ``FdNetDevice`` attributes (see the chapter on ``FdNetDevice``). Output ====== - +The ``NetmapNetDevice`` does not provide any specialized output, but +supports the ``FdNetDevice`` output and traces (such as a promiscuous sniffer +trace). Examples ======== @@ -59,21 +156,19 @@ Examples Several examples are provided: * ``fd-emu-onoff.cc``: This example is aimed at measuring the throughput of the - NetmapNetDevice when using the NetmapNetDeviceHelper to attach the simulated - device to a real device in the host machine. This is achieved by saturating - the channel with TCP or UDP traffic. -* ``fd-emu-ping.cc``: This example uses the NetmapNetDevice to send ICMP + ``NetmapNetDevice`` when using the ``NetmapNetDeviceHelper`` to attach the + simulated device to a real device in the host machine. This is achieved + by saturating the channel with TCP or UDP traffic. +* ``fd-emu-ping.cc``: This example uses the ``NetmapNetDevice`` to send ICMP traffic over a real device. -* ``fd-emu-tc.cc``: This example configures a router on a machine with two interfaces - in emulated mode through netmap. The aim is to explore different qdiscs behaviours on the backlog of a - device emulated bottleneck side. -* ``fd-emu-send.cc``: This example builds a node with a device in emulation mode through netmap. - The aim is to measure the maximum tx rate in pps achievable with NetmapNetDevice on a specific machine. +* ``fd-emu-tc.cc``: This example configures a router on a machine with two + interfaces in emulated mode through netmap. The aim is to explore different + qdiscs behaviours on the backlog of a device emulated bottleneck side. +* ``fd-emu-send.cc``: This example builds a node with a device in + emulation mode through netmap. The aim is to measure the maximum transmit + rate in packets per second (pps) achievable with ``NetmapNetDevice`` on + a specific machine. -Note that all the examples run in emulation mode through netmap (with NetmapNetDevice) and raw socket (with FdNetDevice). +Note that all the examples run in emulation mode through netmap (with +``NetmapNetDevice``) and raw socket (with ``FdNetDevice``). -Acknowledgments -*************** -The NetmapNetDevice has been presented and evaluated in - -* Pasquale Imputato, Stefano Avallone, Enhancing the fidelity of network emulation through direct access to device buffers, Journal of Network and Computer Applications, Volume 130, 2019, Pages 63-75, (http://www.sciencedirect.com/science/article/pii/S1084804519300220) diff --git a/src/fd-net-device/examples/fd-emu-onoff.cc b/src/fd-net-device/examples/fd-emu-onoff.cc index d09c66170..185c306fc 100644 --- a/src/fd-net-device/examples/fd-emu-onoff.cc +++ b/src/fd-net-device/examples/fd-emu-onoff.cc @@ -221,7 +221,7 @@ main (int argc, char *argv[]) sinkApp.Start (Seconds (1.0)); sinkApp.Stop (Seconds (60.0)); - emu.EnablePcap ("fd-server", device); + helper->EnablePcap ("fd-server", device); } else { @@ -237,7 +237,7 @@ main (int argc, char *argv[]) clientApps.Start (Seconds (4.0)); clientApps.Stop (Seconds (58.0)); - emu.EnablePcap ("fd-client", device); + helper->EnablePcap ("fd-client", device); } Simulator::Stop (Seconds (61.0)); diff --git a/src/fd-net-device/model/fd-net-device.h b/src/fd-net-device/model/fd-net-device.h index 6b9e9920b..22a2136b2 100644 --- a/src/fd-net-device/model/fd-net-device.h +++ b/src/fd-net-device/model/fd-net-device.h @@ -218,6 +218,16 @@ protected: */ int GetFileDescriptor (void) const; + /** + * Mutex to increase pending read counter. + */ + SystemMutex m_pendingReadMutex; + + /** + * Number of packets that were received and scheduled for read but not yet read. + */ + std::queue< std::pair > m_pendingQueue; + private: /** * \brief Copy constructor @@ -253,11 +263,6 @@ private: */ virtual void DoFinishStoppingDevice (void); - /** - * Callback to invoke when a new frame is received - */ - void ReceiveCallback (uint8_t *buf, ssize_t len); - /** * Forward the frame to the appropriate callback for processing */ @@ -340,21 +345,11 @@ private: */ bool m_isMulticast; - /** - * Number of packets that were received and scheduled for read but not yet read. - */ - std::queue< std::pair > m_pendingQueue; - /** * Maximum number of packets that can be received and scheduled for read but not yet read. */ uint32_t m_maxPendingReads; - /** - * Mutex to increase pending read counter. - */ - SystemMutex m_pendingReadMutex; - /** * Time to start spinning up the device */ diff --git a/src/fd-net-device/wscript b/src/fd-net-device/wscript index b273662e7..b7a3d4c80 100644 --- a/src/fd-net-device/wscript +++ b/src/fd-net-device/wscript @@ -49,10 +49,16 @@ def configure(conf): dpdk_lib_mode = 'pkg' if have_dpdk_pkg else 'src' conf.env.append_value('DPDK_LIB_MODE', [dpdk_lib_mode]) conf.env.append_value('DEFINES', ['NS3_DPDK']) - conf.report_optional_feature("DpdkNetDevice", - "DPDK NetDevice (using {0} DPDK)".format("package" if have_dpdk_pkg else "source"), - True, - "DPDKNetDevice module enabled") + if have_dpdk_pkg: + conf.report_optional_feature("DpdkNetDevice", + "DPDK NetDevice", + True, + "DPDKNetDevice module enabled (from libdpdk)") + else: + conf.report_optional_feature("DpdkNetDevice", + "DPDK NetDevice", + True, + "DPDKNetDevice module enabled (from source)") else: conf.report_optional_feature("DpdkNetDevice", "DPDK NetDevice",