Anytest

This repository contains the open-source code for the SIGCOMM '26 paper "Anytest: Localizing the Root Cause of Hardware Transport Performance Anomalies."

Requirements

The primary development and test environment for this codebase is:

Hardware

• CPU: 2 × Intel(R) Xeon(R) Platinum 8469C
• Memory: 1 TiB
• NIC: NVIDIA Mellanox BlueField-3 DPU with 4 × 200 Gb/s ports, bonded pairwise into 2 × 400 Gb/s; firmware 32.45.1020

Software

• OS: Alibaba Cloud Linux 3 (Soaring Falcon), kernel 5.10.134 (5.10 series)
• DPDK: 24.11.2
• NIC driver: mlx5_core 25.04-0.6.1.3
• RDMA user-space libraries: rdma-core / libibverbs 2501mlnx56.2 (NVIDIA distribution, libibverbs ABI 1.14.56.0)
• Build toolchain: GCC 10.2.1, Meson 0.61.2, Ninja 1.10.2, CMake 3.26.5
• Python: 3.6.8

Repository Layout

Path	Contents
main.c	The DPDK application's main(): initializes EAL, the NIC port and mbuf pool, aligns the NIC clock with the CPU TSC, handshakes with peers, and dispatches to one of the experiment modes (SENDER / RECEIVER / ROCE / RECEIVER_CUSTOM_SEQ / RP_EXP_CAPABILITY_TEST).
src/	C source for the anytest DPDK application (see docs/Source_code.md for the breakdown).
include/	Public headers for the C sources. common.h defines the shared globals, roles (NP / CP / RP), and the exp_mode enum.
tools/	Python orchestration and analysis. run.py is the entry point; the analysis_*.py modules parse the per-run output (CNP samples, RoCE stats, detailed timestamps, WQE completion) into result.json and plots.
config/	YAML experiment configs, grouped by which DCQCN role is under test: config/opensource-np/ (Notification Point — CNP reply), config/opensource-rp/ (Reaction Point — rate reaction), and config/opensource-cp/ (Congestion Point — switch queueing).
docs/	Design notes and per-case reproduction guides (e.g. Configuration.md, State_machine.md, ICRC_fast_path.md).
figures/	Standalone plotting scripts that turn collected logs into the paper's figures.
libs/	Drop-in directory for residual dynamic libraries injected at runtime via LD_LIBRARY_PATH. Ships empty — the actual .so files are tightly coupled to a specific distribution/glibc/architecture and would almost certainly mismatch yours, so we do not provide them. Place the libraries your hosts need here and run.py distributes them to the test machines (see the build notes).
CMakeLists.txt, rebuild.sh	The build system; see Building below.

For a detailed breakdown of the C sources under src/ — the experiment modes, packet generation, RoCE sink, RDMA verbs path, and infrastructure files — see docs/Source_code.md.

Dependencies

Besides the build toolchain and libraries listed in Requirements (which are covered in more detail under Building below), the orchestration and analysis tooling needs a few extra packages.

System packages. run.py uses rsync to distribute files to the test hosts, and the analysis/plotting stack needs a handful of image libraries. On RPM-based distributions:

sudo yum install -y libjpeg-devel zlib-devel libtiff-devel freetype-devel lcms2-devel libwebp-devel rsync

(Use your distribution's package manager — e.g. apt — for the equivalent packages on other systems.)

Python packages. These are only needed on the machine that runs tools/run.py:

python3 -m pip install --user ruamel.yaml matplotlib numpy pandas scipy

Building

The build needs DPDK 24.11, libibverbs / libmlx5 (rdma-core), libyaml, libpcap, zlib, CMake ≥ 3.10, and a C11-capable GCC (see Requirements above for exact versions). The default semi-static build is produced by:

./rebuild.sh

This statically links the heavy application dependencies (most importantly DPDK) into a mostly self-contained binary that tools/run.py can distribute across the cluster, while glibc and a few unavoidably-dynamic plugins stay dynamic. See docs/Build_system.md for the STATIC_BUILD option and the two build modes (semi-static vs. dynamic), the -DDPDK_PREFIX handling, and why the binary is semi-static and what the libs/ directory is for.

Running Experiments

The Entry Point: run.py

tools/run.py is a cluster orchestrator: it distributes the prebuilt binary, the libs/ directory, and the chosen config to every test host, launches the DPDK application over SSH, then collects and analyzes the per-host results. Before first use, edit the HOST_CONFIGS / BASE_DIR block at the top of the script to match your cluster.

The main workflow is:

First, make a new directory (e.g. ~/anytest) on machines involved and distribute the binary:

./tools/run.py distribute anytest ~/anytest/anytest

Then, configure the DPDK environment on each host, including hugepages and jumbo frames.

Finall, run an experiment with a given config, for example:

./tools/run.py run --program anytest --config config/opensource-np/cnp-simple-test.yaml

which distributes the binary and config, launches the experiment, collects logs into logs/<timestamp>/, and runs the analysis. For the full configuration details and the complete subcommand reference, see docs/Run_script.md.

After an experiment finishes, the analysis results are stored under the logs/<timestamp>/ directory. For NP experiments, the number and arrival rate of CNPs are plotted automatically; for RP experiments, the per-QP throughput is plotted automatically.

Configuration

We ship a set of ready-to-use configs under config/. The meaning of each one is documented in Reproducing the Paper's Experimental Results below, which maps every config to the paper case it reproduces. To write your own config, see docs/Configuration.md.

Testing for the Ability to Receive RoCEv2 Packets

Before running any RP-testing experiments, it is important to verify that the test hosts can receive RoCEv2 packets in DPDK. This is done by running the RP_EXP_CAPABILITY_TEST mode on both hosts, which sends a few test packets and checks whether they are received correctly. To do so, run the following command:

./tools/run.py run --program ./anytest --config config/opensource-rp/rp-exp-capability-test.yaml

See docs/RP_capability_test.md for the detailed procedure.

Note: After applying the rte_flow configuration in the code, your NIC should be able to receive RoCEv2 packets. If your test environment cannot receive RoCEv2 packets, try disabling the NIC's RoCEv2 feature (mlxconfig -d YOUR_NIC_PCIE_ADDR -y s ROCE_CONTROL=1), which may work on some machines. In our experience, some NICs still cannot receive RoCEv2 packets even after the RoCEv2 feature is disabled; we found this may be related to the firmware and driver versions. We are currently investigating this issue. If you encounter a similar problem, you can try switching to a different firmware and driver version, or contact your NIC vendor for support. If you have any findings or solutions regarding this issue, you are also welcome to submit an issue or PR in our GitHub repository.

Debugging

For debugging the behavior of the DPDK application, we recommend the pdump tool, which can capture the application's TX/RX packets to help diagnose problems. See the official documentation for usage.

Reproducing the Paper's Experimental Results

This repository provides reproduction cases for the cases in the paper, to help researchers and engineers better understand and validate the methods and results. These cases cover nearly all of Anytest's functionality, so they also serve as a reference for learning and using Anytest. The mapping between the paper's cases and the reproduction cases is as follows:

Case	Config / Files	Description
#1.1	cnp-simple-test.yaml	Tests the NP's CNP reply capability using a continuous line-rate stream of ECN-marked packets.
#1.2	—	This test requires a specific experimental PCC algorithm whose release is out of scope, so no reproduction case is provided. See docs/Reproduce_case1_2.md for the reproduction approach.
#1.3	cnp-custom-seq-0.yaml, cnp-custom-seq-1.yaml, cnp-custom-seq-2.yaml	These three configs reproduce the results in Table 3 of the paper.
#2	—	Reproducing this result depends on hardware with a specific firmware bug and cannot be reproduced on devices without that bug.
#3	dcqcn-cnpintv-long.yaml, dcqcn-cnpintv-short.yaml	These two configs, together with the plotting script figures/case3_plot_throughput.py, reproduce Figure 13 of the paper.
#4	—	Reproducing this result depends on hardware with a specific firmware bug and cannot be reproduced on devices without that bug.
#5	ets-sche-test.yaml	Reproducing this result requires a specific experimental PCC algorithm whose release is out of scope, so no reproduction case is provided. See docs/Reproduce_case5_figure14.md for the reproduction approach.
#6	—	Reproducing this result depends on hardware with a specific firmware bug and cannot be reproduced on devices without that bug.

License

This project is licensed under the Apache License, Version 2.0. See the LICENSE and NOTICE files for details.

A few files (main.c, src/rte_flow_setup.c) are derived from the DPDK examples, which are licensed under BSD-3-Clause. Their original copyright notices are retained in the file headers and listed in NOTICE, as required; BSD-3-Clause is compatible with the Apache-2.0 license of this project.