From 379f255d63b08f1821e61613f030948bd61460f5 Mon Sep 17 00:00:00 2001 From: F5 Date: Sat, 16 Sep 2023 22:49:09 +0800 Subject: [PATCH] Update README.md for UNISON --- README.md | 309 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 204 insertions(+), 105 deletions(-) diff --git a/README.md b/README.md index e37e17f63..424c45a8b 100644 --- a/README.md +++ b/README.md @@ -1,126 +1,225 @@ -The Network Simulator, Version 3 +UNISON for ns-3 ================================ +A fast and user-transparent parallel simulator implementation for ns-3. +More information about UNISON can be found in our EuroSys '24 paper (coming soon). -## Table of Contents: +## Getting Started -1) [An overview](#an-open-source-project) -2) [Building ns-3](#building-ns-3) -3) [Running ns-3](#running-ns-3) -4) [Getting access to the ns-3 documentation](#getting-access-to-the-ns-3-documentation) -5) [Working with the development version of ns-3](#working-with-the-development-version-of-ns-3) - -Note: Much more substantial information about ns-3 can be found at -https://www.nsnam.org - -## An Open Source project - -ns-3 is a free open source project aiming to build a discrete-event -network simulator targeted for simulation research and education. -This is a collaborative project; we hope that -the missing pieces of the models we have not yet implemented -will be contributed by the community in an open collaboration -process. - -The process of contributing to the ns-3 project varies with -the people involved, the amount of time they can invest -and the type of model they want to work on, but the current -process that the project tries to follow is described here: -https://www.nsnam.org/developers/contributing-code/ - -This README excerpts some details from a more extensive -tutorial that is maintained at: -https://www.nsnam.org/documentation/latest/ - -## Building ns-3 - -The code for the framework and the default models provided -by ns-3 is built as a set of libraries. User simulations -are expected to be written as simple programs that make -use of these ns-3 libraries. - -To build the set of default libraries and the example -programs included in this package, you need to use the -tool 'ns3'. Detailed information on how to use ns3 is -included in the file doc/build.txt - -However, the real quick and dirty way to get started is to -type the command -```shell -./ns3 configure --enable-examples -``` - -followed by +The quickest way to get started is to type the command ```shell -./ns3 +./ns3 configure --enable-mtp --enable-examples ``` -in the directory which contains this README file. The files -built will be copied in the build/ directory. +> The build profile is set to default (which uses `-O2 -g` compiler flags) in this case. +> If you want to get `-O3` optimized build and discard all log outputs, please add `-d optimized` arguments. -The current codebase is expected to build and run on the -set of platforms listed in the [release notes](RELEASE_NOTES.md) -file. +The `--enable-mtp` option will enable multi-threaded parallelization. +You can verify UNISON is enabled by checking whether `Multithreaded Simulation : ON` appears in the optional feature list. -Other platforms may or may not work: we welcome patches to -improve the portability of the code to these other platforms. - -## Running ns-3 - -On recent Linux systems, once you have built ns-3 (with examples -enabled), it should be easy to run the sample programs with the -following command, such as: +Now, let's build and run a DCTCP example with default sequential simulation and parallel simulation (using 4 threads) respectively: ```shell -./ns3 run simple-global-routing +./ns3 build dctcp-example dctcp-example-mtp +time ./ns3 run dctcp-example +time ./ns3 run dctcp-example-mtp ``` -That program should generate a `simple-global-routing.tr` text -trace file and a set of `simple-global-routing-xx-xx.pcap` binary -pcap trace files, which can be read by `tcpdump -tt -r filename.pcap` -The program source can be found in the examples/routing directory. +The simulation should finish in 4-5 minutes for `dctcp-example` and 1-2 minutes for `dctcp-example-mtp`, depending on your hardware and your build profile. +The output in `*.dat` should be in accordance with the comments in the source file. -## Getting access to the ns-3 documentation +The speedup of UNISON is more significant for larger topologies and traffic volumes. +If you are interested in using it to simulate topologies like fat-tree, BCube and 2D-torus, please refer to [Running Evaluations](#running-evaluations). -Once you have verified that your build of ns-3 works by running -the simple-point-to-point example as outlined in 3) above, it is -quite likely that you will want to get started on reading -some ns-3 documentation. +## Speedup Your Existing Code -All of that documentation should always be available from -the ns-3 website: https://www.nsnam.org/documentation/. +To understand how UNISON affects your model code, let's find the differences between two versions of the source files of the above example: -This documentation includes: - - - a tutorial - - - a reference manual - - - models in the ns-3 model library - - - a wiki for user-contributed tips: https://www.nsnam.org/wiki/ - - - API documentation generated using doxygen: this is - a reference manual, most likely not very well suited - as introductory text: - https://www.nsnam.org/doxygen/index.html - -## Working with the development version of ns-3 - -If you want to download and use the development version of ns-3, you -need to use the tool `git`. A quick and dirty cheat sheet is included -in the manual, but reading through the git -tutorials found in the Internet is usually a good idea if you are not -familiar with it. - -If you have successfully installed git, you can get -a copy of the development version with the following command: ```shell -git clone https://gitlab.com/nsnam/ns-3-dev.git +diff examples/tcp/dctcp-example.cc examples/mtp/dctcp-example-mtp.cc ``` -However, we recommend to follow the Gitlab guidelines for starters, -that includes creating a Gitlab account, forking the ns-3-dev project -under the new account's name, and then cloning the forked repository. -You can find more information in the [manual](https://www.nsnam.org/docs/manual/html/working-with-git.html). +It turns out that to bring UNISON to existing model code, all you need to do is to include the `ns3/mtp-interface.h` header file and add the following line at the beginning of the `main` function: + +```c++ +MtpInterface::Enable (numberOfThreads); +``` + +The parameter `numberOfThreads` is optional. +If it is omitted, the number of threads is automatically chosen and will not exceed the maximum number of available hardware threads on your system. If you want to enable UNISON for distributed simulation on existing MPI programs for further speedup, place the above line before MPI initialization. + +UNISON resolved a lot of thread-safety issues with ns-3's architecture. +You don't need to consider these issues on your own for most of the time, except if you have custom global statistics other than the built-in flow-monitor. +In the letter case, if multiple nodes can access your global statistics, you can replace them with atomic variables via `std::atomic<>`. +For complex custom data structures, you can create critical sections by adding + +```c++ +MtpInterface::CriticalSection cs; +``` + +at the beginning of your methods. + +## Running Evaluations + +To evaluate UNISON, please checkout to [unison-evaluations](https://github.com/NASA-NJU/UNISON-for-ns-3/tree/unison-evaluations) branch. +In this branch, you can find various topology models in the `scratch` folder. +There are a lot of parameters you can set for each topology. +We provided a utility script `exp.py` to compare these simulators and parameters. +We also provided `process.py` to convert these raw experiment data to CSV files suitable for ploting. +Please see the [README in that branch](https://github.com/NASA-NJU/UNISON-for-ns-3/tree/unison-evaluations) for more details. + +## Module Documentation + +### 1. Overview + +UNISON for ns-3 is mainly implemented in the `mtp` module, which stands for multi-threaded parallelization. +This module contains three parts: A parallel simulator implementation `multithreaded-simulator-impl`, an interface to users `mtp-interface`, and `logical-process` to represent LPs in terms of parallel simulation. + +All LPs and threads are stored in the `mtp-interface`. +It controls the simulation progress, schedules LPs to threads and manages the lifecycles of LPs and threads. +The interface also provides some methods and options for users to tweak the simulation. + +Each LP's logic is implemented in `logical-process`. It contains most of the methods of the default sequential simulator plus some auxiliary methods for parallel simulation. + +The simulator implementation `multithreaded-simulator-impl` is a derived class from the base simulator. +It converts calls to the base simulator into calls to logical processes based on the context of the current thread. +It also provides a partition method for automatic fine-grained topology partition. + +For distributed simulation with MPI, we added `hybrid-simulator-impl` in the `mpi` module. +This simulator uses both `mtp-interface` and `mpi-interface` to coordinate local LPs and global MPI communications. +We also modified the module to make it locally thread-safe. + +### 2. Modifications to ns-3 Architecture + +In addition to the `mtp` and `mpi` modules, we also modified the following part of the ns-3 architecture to make it thread-safe, also with some bug fixing for ns-3. + +Modifications to the build system to provide `--enable-mtp` option to enable/disable UNISON: + +``` +ns3 | 2 + +CMakeLists.txt | 1 + +build-support/custom-modules/ns3-configtable.cmake | 3 + +build-support/macros-and-definitions.cmake | 10 + +``` + +Modifications to the `core` module to make reference counting thread-safe: + +``` +src/core/CMakeLists.txt | 1 + +src/core/model/atomic-counter.h | 49 ++ +src/core/model/hash.h | 16 + +src/core/model/object.cc | 2 + +src/core/model/simple-ref-count.h | 11 +- +``` + +Modifications to the `network` module to make packets thread-safe: + +``` +src/network/model/buffer.cc | 15 +- +src/network/model/buffer.h | 7 + +src/network/model/byte-tag-list.cc | 14 +- +src/network/model/node.cc | 7 + +src/network/model/node.h | 7 + +src/network/model/packet-metadata.cc | 10 +- +src/network/model/packet-metadata.h | 11 +- +src/network/model/packet-tag-list.h | 11 +- +src/network/model/socket.cc | 6 + +src/network/utils/simple-net-device.cc | 4 + +``` + +Modifications to the `internet` module to make it thread-safe, plus adding per-flow ECMP routing and increasing # of available ports to support a large number of flows: + +``` +src/internet/model/global-route-manager-impl.cc | 2 + +src/internet/model/ipv4-end-point-demux.cc | 2 +- +src/internet/model/ipv4-global-routing.cc | 31 +- +src/internet/model/ipv4-global-routing.h | 5 +- +src/internet/model/ipv4-packet-info-tag.cc | 2 + +src/internet/model/ipv6-end-point-demux.cc | 4 +- +src/internet/model/ipv6-packet-info-tag.cc | 2 + +src/internet/model/tcp-option.cc | 2 +- +``` + +Modifications to the `flow-monitor` module to make it thread-safe: + +``` +src/flow-monitor/model/flow-monitor.cc | 42 + +src/flow-monitor/model/flow-monitor.h | 4 + +src/flow-monitor/model/ipv4-flow-classifier.cc | 12 + +src/flow-monitor/model/ipv4-flow-classifier.h | 4 + +src/flow-monitor/model/ipv4-flow-probe.cc | 2 + +src/flow-monitor/model/ipv6-flow-classifier.cc | 12 + +src/flow-monitor/model/ipv6-flow-classifier.h | 4 + +src/flow-monitor/model/ipv6-flow-probe.cc | 2 + +``` + +Modifications to the `nix-vector-routing` module to make it thread-safe: + +``` +src/nix-vector-routing/model/nix-vector-routing.cc | 92 ++ +src/nix-vector-routing/model/nix-vector-routing.h | 8 + +``` + +### 3. Logging + +The reason behind UNISON's fast speed is that it divides the network into multiple logical processes (LPs) with fine granularity and schedules them dynamically. +To get to know more details of such workflow, you can enable the following log component: + +```c++ +LogComponentEnable ("LogicalProcess", LOG_LEVEL_INFO); +LogComponentEnable ("MultithreadedSimulatorImpl", LOG_LEVEL_INFO); +``` + +### 4. Advanced Options + +These options can be modified at the beginning of the `main` function using the native config syntax of ns-3. + +You can also change the default maximum number of threads by setting + +```c++ +Config::SetDefault ("ns3::MultithreadedSimulatorImpl::MaxThreads", UintegerValue (8)); +Config::SetDefault ("ns3::HybridSimulatorImpl::MaxThreads", UintegerValue (8)); +``` + +The automatic partition will cut off stateless links whose delay is above the threshold. +The threshold is automatically calculated based on the delay of every link. +If you are not satisfied with the partition results, you can set a custom threshold by setting + +```c++ +Config::SetDefault ("ns3::MultithreadedSimulatorImpl::MinLookahead", TimeValue (NanoSeconds (500)); +Config::SetDefault ("ns3::HybridSimulatorImpl::MinLookahead", TimeValue (NanoSeconds (500)); +``` + +The scheduling method determines the priority (estimated completion time of the next round) of each logical process. +There are five available options: + +- `ByExecutionTime`: LPs with higher execution time of the last round will have higher priority. +- `ByPendingEventCount`: LPs with more pending events of this round will have higher priority. +- `ByEventCount`: LPs with more pending events of this round will have higher priority. +- `BySimulationTime`: LPs with larger current clock time will have higher priority. +- `None`: Do not schedule. The partition's priority is based on their ID. + +Many experiments show that the first one usually leads to better performance. +However, you can still choose one according to your taste by setting + +```c++ +GlobalValue::Bind ("PartitionSchedulingMethod", StringValue ("ByExecutionTime")); +``` + +By default, the scheduling period is 2 when the number of partitions is less than 16, 3 when it is less than 256, 4 when it is less than 4096, etc. +Since more partitions lead to more scheduling costs. +You can also set how frequently scheduling occurs by setting + +```c++ +GlobalValue::Bind ("PartitionSchedulingPeriod", UintegerValue (4)); +``` + +## Links + +If you find the code useful, please consider citing our paper (coming soon). +Below are some links that may also be helpful to you: + +- [ns-3 Tutorial](https://www.nsnam.org/docs/tutorial/html/index.html) +- [ns-3 Model Library](https://www.nsnam.org/docs/models/html/index.html) +- [ns-3 Manual](https://www.nsnam.org/docs/manual/html/index.html)