Tutorial

This tutorial goes from git clone to a breakdown of where each I/O's time goes, on two machines, one command at a time. After every command comes the output it printed on our machines, so you can compare yours as you go: when they agree, you are on track.

The parts

Part	What you do	Time
1. Build	clone, build and test ntprof; copy it to the other machine; build SPDK with ntprof's probes for parts 4 and 5 (15 more minutes)	10 min
2. Linux NVMe/TCP	export a drive with the kernel's NVMe/TCP target, read it with fio from the other machine, record both machines, join the two records	20 min
3. Linux NVMe/RDMA	the same over RDMA, and what changes	10 min
4. SPDK NVMe/TCP	give the drive to SPDK, export it with SPDK's NVMe/TCP target, read it with spdk_nvme_perf, record both machines, join	20 min
5. SPDK NVMe/RDMA	the same over RDMA	10 min

Parts 2 to 5 are the four combinations of software and transport, each the same 4 KiB random read on the same drive. Our end-to-end means:

	NVMe/TCP	NVMe/RDMA
Linux kernel drivers	part 2: 86.4 µs	part 3: 72.8 µs
SPDK	part 4: 82.3 µs	part 5: 66.2 µs

Part 1 comes first. Parts 2 to 5 each leave both machines as they found them, so after part 1 you can go straight to the one you need.

What you need

• Two Linux machines (5.15 or 6.x) with root, and a network between them; RDMA NICs for parts 3 and 5.
• On one of them, the target, an NVMe drive you may overwrite: part 2 writes its first 64 GiB.
• On the machine you build on: git, make, clang 12 or later, and the libelf and zlib development files. bpftool v7 or later, or the tutorial shows how to build it.
• On the other machine, the host: fio and nvme-cli.

Our machines

	Host	Target
CPUs	2 × Xeon Gold 6526Y, 64 threads	the same
System	Ubuntu 22.04, Linux 5.15.143, MLNX OFED 24.10	Ubuntu 22.04, Linux 5.15.134 (a vendor build), MLNX OFED 24.10 from DKMS
NIC	ConnectX-7, 400 GbE RoCE, MTU 9000	the same
Address	10.101.0.4	10.101.0.5
Drive		Samsung PM9A3 1.92 TB, /dev/nvme1n1

The target's NVMe modules were built without BTF type information, which ntprof needs; part 2 shows what that looks like and what to do about it.

How to read the pages

Every step has the same three pieces: a command, its options, and the output we got.

Targetcommand

./ntprof --version

Output

ntprof 2.0.0 (3c4a27a)

• A dark panel is a command. Its label names the machine, Host or Target, and the shell when several run at once (terminal 2 is a second shell on the same machine, for a command that keeps running while you type others). The $ only marks where a command starts: the copy button at its top right copies the commands without it. A line ending in \ continues on the next line.
• A table under a command explains its options.
• A light panel marked Output is what the command printed for us, copied from our run. We left out lines that do not matter for the step and marked the gap with …, and our working directory shows as ~; nothing else is changed.
• A green Check box says what should match on your machine. Versions, the hooks ntprof finds, load: ok and the share of commands matched should be the same. Latencies depend on your hardware: compare which stage takes most of the time, not the digits.

Use your own address, drive and device names where ours appear: 10.101.0.5, /dev/nvme1n1, the PCI address 0000:d0:00.0, and the name the host gives the remote drive (nvme6n1 for us).