5. SPDK NVMe/RDMA

The same read as part 3, with SPDK in place of the kernel's drivers on both machines, and the same steps as part 4 over RDMA. With SPDK and RDMA together, no I/O passes through the kernel at all: SPDK polls the NICs and the drive from user space.

host

spdk_nvme_perfSPDK NVMe/RDMA host

ntprof → host.ntp

NVMe/RDMA, 10.101.0.5:4421

target

SPDK NVMe/RDMA targetSPDK bdev, drive

ntprof → target.ntp

ntprof records each machine on its own; ntprof report host.ntp target.ntp joins the two files afterwards.

In this part

You need: what part 4 needs, and RDMA NICs on both machines (RoCE here).
What changes from part 4: RDMA and rdma in the transport and the listener, -t rdma for ntprof, and trtype:RDMA in perf's address. Part 4 explains each step in more detail.
Time: about 10 minutes. Every command runs in ~/ntprof; SPDK is in ~/spdk.

1. Give the drive to SPDK

As in part 4, step 1: check the device, its driver and the hugepages, then move the drive from nvme to uio_pci_generic.

Targetcommand

lspci -s d0:00.0
basename $(readlink /sys/bus/pci/devices/0000:d0:00.0/driver)
grep -E 'HugePages_(Total|Free)' /proc/meminfo

Output

d0:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
nvme
HugePages_Total:   32768
HugePages_Free:    32768

The drive leaves Linux

Until step 6, /dev/nvme1n1 does not exist. Nothing may use the drive.

Targetcommand

echo uio_pci_generic | sudo tee /sys/bus/pci/devices/0000:d0:00.0/driver_override
echo 0000:d0:00.0 | sudo tee /sys/bus/pci/devices/0000:d0:00.0/driver/unbind
echo 0000:d0:00.0 | sudo tee /sys/bus/pci/drivers_probe
basename $(readlink /sys/bus/pci/devices/0000:d0:00.0/driver)

Output

uio_pci_generic
0000:d0:00.0
0000:d0:00.0
uio_pci_generic

Check

The last line says uio_pci_generic.

2. Start the SPDK target

The same nvmf_tgt as in part 4 (its options), in a second shell:

Targetterminal 2command

cd ~/ntprof
sudo ../spdk/build/bin/nvmf_tgt -m 0x10000 -s 2048

Outputso far

[2026-10-17 06:12:31.633098] Starting SPDK v26.05 git sha1 d519b163c / DPDK 26.03.0 initialization...
…
[2026-10-17 06:12:32.249019] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 16

From the first shell, the RDMA transport, the drive as bdev Nvme0n1 and its 64 GiB part Nvme0n1p0, and the export with an RDMA listener:

Targetcommand

sudo ../spdk/scripts/rpc.py nvmf_create_transport -t RDMA

Targetcommand

sudo ../spdk/scripts/rpc.py bdev_nvme_attach_controller \
    -b Nvme0 -t PCIe -a 0000:d0:00.0
sudo ../spdk/scripts/rpc.py bdev_split_create -s 65536 Nvme0n1 1

Output

Nvme0n1
Nvme0n1p0

Targetcommand

sudo ../spdk/scripts/rpc.py nvmf_create_subsystem \
    nqn.2026-10.edu.wisc.ntprof:tutorial -a -s NTPROF0000000001
sudo ../spdk/scripts/rpc.py nvmf_subsystem_add_ns \
    nqn.2026-10.edu.wisc.ntprof:tutorial Nvme0n1p0
sudo ../spdk/scripts/rpc.py nvmf_subsystem_add_listener \
    nqn.2026-10.edu.wisc.ntprof:tutorial -t rdma -a 10.101.0.5 -s 4421 -f ipv4

Outputtarget, terminal 2

[2026-10-17 06:12:32.737665] rdma.c:2757:create_ib_device: *NOTICE*: Create IB device mlx5_0(0x5650e2e49cf0/0x5650e2e4a420) succeed.
…
[2026-10-17 06:12:34.434186] rdma.c:3181:nvmf_rdma_listen: *NOTICE*: *** NVMe/RDMA Target Listening on 10.101.0.5 port 4421 ***

Create IB device mlx5_0 is SPDK opening the RDMA NIC when the transport is created; -a 10.101.0.5 must be an address of that NIC.

Check

NVMe/RDMA Target Listening on with your address and port.

3. Check and start recording on the target

Targetcommand

sudo ./ntprof probe -r target -t rdma --spdk-target ../spdk/build/bin/nvmf_tgt

Output

kernel BTF: yes
function hooks: fentry/fexit (they can attach now)

spdk_rdma_target (module BTF nvmet_rdma: absent)
  USDT probes in ../spdk/build/bin/nvmf_tgt:
  ok   nvmf_tcp_done          3 sites
  ok   nvmf_rdma_done         1 site
  ok   nvme_tcp_done          1 site
  ok   nvme_rdma_done         1 site
  running processes: 1521728 (sampling variable at 0x5650e1321e50)
  ok   ntp_u_nvmf_rdma_done   USDT ntprof:nvmf_rdma_done               (required)
  load: ok

Compare with part 4: over RDMA there is no kernel network stack in SPDK's path, so ntprof attaches only SPDK's own probe and no kernel hook at all.

Check

running processes: names a process, and load: ok.

Targetterminal 3command

cd ~/ntprof
sudo ./ntprof record -r target -t rdma -s 100 -i 10 \
    --spdk-target ../spdk/build/bin/nvmf_tgt -o target.ntp

Outputso far

ntprof: profiling as target, sampling 1 of 100 commands, saving records; Ctrl-C to stop

4. Run spdk_nvme_perf and record on the host

Hugepages for perf, on the NIC's NUMA node:

Hostcommand

echo 2048 | sudo tee /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
grep -E 'HugePages_(Total|Free)' /proc/meminfo

Output

2048
HugePages_Total:    2048
HugePages_Free:     2048

The workload in a second shell, the same as part 4's (its options) with trtype:RDMA:

Hostterminal 2command

cd ~/ntprof
sudo ../spdk/build/bin/spdk_nvme_perf -q 1 -o 4096 -w randread -t 30 -c 0x10000 \
    -r 'trtype:RDMA adrfam:IPv4 traddr:10.101.0.5 trsvcid:4421 subnqn:nqn.2026-10.edu.wisc.ntprof:tutorial'

While it runs, in the first shell:

Hostcommand

sudo ./ntprof probe -r host -t rdma --spdk-host ../spdk/build/bin/spdk_nvme_perf
sudo ./ntprof record -t rdma -s 100 -d 20 -i 10 \
    --spdk-host ../spdk/build/bin/spdk_nvme_perf -o host.ntp

Output

kernel BTF: yes
function hooks: fentry/fexit (they can attach now)

spdk_rdma_host (module BTF nvme_rdma: present)
  USDT probes in ../spdk/build/bin/spdk_nvme_perf:
  ok   nvme_tcp_done          1 site
  ok   nvme_rdma_done         1 site
  running processes: 2694637 (sampling variable at 0x564ca1403d70)
  ok   ntp_u_nvme_rdma_done   USDT ntprof:nvme_rdma_done               (required)
  load: ok
ntprof: profiling as host, sampling 1 of 100 commands, saving records; Ctrl-C to stop
every 10 s, per group: sampled commands, rate, latency, and each stage's share of the mean
(*: network and target together; the target's side is not recorded)
                                                                          host                   target
    time  group                       n   cmds/s   mean us    p99 us   block  driver network  driver storage
   10.0s  read 4KiB spdk-c1n1      1517    15.2k     66.08     73.53      0%      0%   100%*
ntprof: 2999 records written to host.ntp

=== 20.0 s, one command in 100 sampled ===
throughput (estimated from the sampled commands): read 15.0 kIOPS, 61.3 MB/s

== read 4KiB spdk-c1n1: 2999 commands sampled (from about 15.0k commands/s), host side only
  latency (us): mean 66.17, p50 62.98, p99 73.54, p99.9 74.13
  stage                                   mean us  share   avg QD
  host driver (SPDK)                         0.11     0%     0.00
  host receive path                          0.06     0%     0.00
  network and target (not recorded)         66.00   100%     0.99  ########################
  largest: network and target (not recorded), 100 % of the latency

records: 2999 host

SPDK's host driver takes 0.1 µs per read over RDMA, against 3.2 µs over TCP in part 4: posting a request to an RDMA NIC is a few memory writes, with no socket in between.

Outputhost, terminal 2

…
Initializing NVMe Controllers
Attached to NVMeoF Controller at RDMA (addr:10.101.0.5 subnqn:nqn.2026-10.edu.wisc.ntprof:tutorial)
Associating RDMA (addr:10.101.0.5 subnqn:nqn.2026-10.edu.wisc.ntprof:tutorial) NSID 1 with lcore 16
Initialization complete. Launching workers.
========================================================
                                                                                                                          Latency(us)
Device Information                                                                    :       IOPS      MiB/s    Average        min        max
RDMA (addr:10.101.0.5 subnqn:nqn.2026-10.edu.wisc.ntprof:tutorial) NSID 1 from core 16:   15122.27      59.07      66.11      61.89     341.63
========================================================
Total                                                                                 :   15122.27      59.07      66.11      61.89     341.63

Check

ntprof's mean (66.17 µs) and perf's Average (66.11 µs) agree within a fraction of a microsecond.

5. Stop the target's recording and join

Press Ctrl-C in the target's third shell:

Outputtarget, terminal 3

ntprof: profiling as target, sampling 1 of 100 commands, saving records; Ctrl-C to stop
ntprof: 4509 records written to target.ntp

records: 4509 target
report: ntprof report HOST.ntp target.ntp (the host's recording, same -s)

Hostcommand

scp target:ntprof/target.ntp .
./ntprof report --detail host.ntp target.ntp

Output

recorded on madracks-gnode4 (kernel 5.15.143), one command in 100 sampled

== read 4KiB spdk-c1n1: 2999 commands sampled (from about 15.0k commands/s), all with the target's side
  latency (us): mean 66.17, p50 62.98, p99 73.54, p99.9 74.13
  stage                                   mean us  share   avg QD
  host driver (SPDK)                         0.11     0%     0.00
  network (RDMA, NICs, wire)                 6.43    10%     0.10  ##
  target driver (SPDK)                       0.35     1%     0.01
  storage (SPDK bdev, drive)                59.27    90%     0.89  #####################
  largest: storage (SPDK bdev, drive), 90 % of the latency
  components, in path order (L: commands in each, by Little's law)
  component                                     n      mean       L       p50       p99   (us)
  host block layer, submission               2999      0.00    0.00      0.00      0.00
  host driver, submission                    2999      0.11    0.00      0.12      0.13
  network: wire and NICs (computed)          2999      6.27    0.09      6.25      6.37
    data transfer (RDMA, at the target)      2999      5.39    0.08      5.39      5.48
  network: target receive path               2999      0.10    0.00      0.10      0.12
  target driver, submission                  2999      0.09    0.00      0.09      0.11
  storage: target block layer and drive      2999     59.27    0.89     56.10     66.65
  target driver, completion                  2999      0.27    0.00      0.26      0.32
  network: host receive path                 2999      0.06    0.00      0.06      0.07
  host driver, completion                    2999      0.00    0.00      0.00      0.00
  host block layer, completion               2999      0.00    0.00      0.00      0.00
    handoff to nvme_complete_rq              2999      0.00    0.00      0.00      0.00
  end to end                                 2999     66.17    0.99     62.98     73.54

records: 2999 host, 4509 target; 2999 matched (100 % of the host's)

Check

All the host's commands matched.

Storage is 90 % of the read here: with SPDK over RDMA almost all that is left is the drive itself.

Linux and SPDK side by side

The same read over NVMe/RDMA, on the same machines and drive:

Stage (µs)	Linux NVMe/RDMA (part 3)	SPDK NVMe/RDMA
host block layer	0.50	(none)
host driver	1.27	0.11
network	10.00	6.43
target driver	1.01	0.35
storage	60.00	59.27
end to end	72.78	66.17

SPDK polls where the kernel drivers take interrupts. That shows in the two drivers and in the network stage, whose completions go through the drivers' interrupt handling on Linux.
The RDMA WRITE that carries the read's data takes 5.4 µs with SPDK's target, 7.1 µs with nvmet-rdma (the component "data transfer (RDMA, at the target)").
Storage is 0.7 µs lower through SPDK's own NVMe driver.

6. Clean up

Give back the host's hugepages:

Hostcommand

echo 0 | sudo tee /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages

Output

Stop the target with Ctrl-C in its second shell, then return the drive to Linux:

Targetcommand

echo 0000:d0:00.0 | sudo tee /sys/bus/pci/devices/0000:d0:00.0/driver/unbind
echo | sudo tee /sys/bus/pci/devices/0000:d0:00.0/driver_override
echo 0000:d0:00.0 | sudo tee /sys/bus/pci/drivers_probe
basename $(readlink /sys/bus/pci/devices/0000:d0:00.0/driver); lsblk -d -o NAME,SERIAL /dev/nvme1n1

Output

0000:d0:00.0

0000:d0:00.0
nvme
NAME    SERIAL
nvme1n1 S728NS0X304953P

Check

The drive is back on nvme with its own serial number.

Where next

Reading a report: every row and column.
Use cases: a busy drive, a busy network, queue depth, with ntprof compare.
SPDK: the four I/O classes, SPDK against Linux, and what the patch does.

1. Give the drive to SPDK​

2. Start the SPDK target​

3. Check and start recording on the target​

4. Run spdk_nvme_perf and record on the host​

5. Stop the target's recording and join​

Linux and SPDK side by side​

6. Clean up​

Where next​

1. Give the drive to SPDK

2. Start the SPDK target

3. Check and start recording on the target

4. Run spdk_nvme_perf and record on the host

5. Stop the target's recording and join

Linux and SPDK side by side

6. Clean up

Where next