4. SPDK NVMe/TCP
The same read as part 2, with SPDK in place of the kernel's drivers on both machines. SPDK runs
NVMe-oF in user space and polls instead of waiting for interrupts. On the target, nvmf_tgt serves the drive and
drives the SSD with SPDK's own NVMe driver; on the host, spdk_nvme_perf is the workload and the NVMe/TCP host
driver in one program.
The kernel's drivers of parts 2 and 3 needed no change: ntprof hooks them inside the kernel. SPDK's I/O never enters the kernel, so ntprof reads it through probes that a small patch adds to SPDK, built in part 1; why a patch for SPDK, and none for the kernel.
host.ntptarget.ntpntprof report host.ntp target.ntp joins the two files afterwards.- You need: ntprof on both machines and SPDK built with ntprof's probes (part 1); the drive free, not mounted and not exported by the kernel target of parts 2 and 3. If you skipped part 2, fill the drive first (part 2, step 1).
- You do: hand the drive to SPDK, start SPDK's target, record both sides while
spdk_nvme_perfreads, join, give the drive back to Linux. - Time: about 20 minutes. Every command runs in
~/ntprof; SPDK is in~/spdk.
1. Give the drive to SPDK
SPDK's NVMe driver runs in user space and needs the drive's PCI device for itself, so Linux's nvme driver has to
let go of it. First find the device, the driver that has it now, and the hugepages SPDK will allocate its memory
from:
lspci -s d0:00.0
basename $(readlink /sys/bus/pci/devices/0000:d0:00.0/driver)
grep -E 'HugePages_(Total|Free)' /proc/meminfo
d0:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
nvme
HugePages_Total: 32768
HugePages_Free: 32768
ls -l /sys/block/nvme1n1 shows the PCI address of yours (the last address in the path before nvme/). Our target
has hugepages reserved; if yours has none, echo 1024 | sudo tee /proc/sys/vm/nr_hugepages reserves 2 GiB, which
is what the target asks for in step 2.
From the next command until step 6, /dev/nvme1n1 does not exist. Nothing may use the drive: no file system
mounted on it, no kernel export from parts 2 and 3.
echo uio_pci_generic | sudo tee /sys/bus/pci/devices/0000:d0:00.0/driver_override
echo 0000:d0:00.0 | sudo tee /sys/bus/pci/devices/0000:d0:00.0/driver/unbind
echo 0000:d0:00.0 | sudo tee /sys/bus/pci/drivers_probe
basename $(readlink /sys/bus/pci/devices/0000:d0:00.0/driver)
| Step | What it does |
|---|---|
driver_override | tells the kernel that this device may only be bound to uio_pci_generic |
driver/unbind | detaches the device from the driver that has it, nvme |
drivers_probe | asks the kernel to bind the device again; because of the override it goes to uio_pci_generic |
uio_pci_generic
0000:d0:00.0
0000:d0:00.0
uio_pci_generic
uio_pci_generic is a minimal kernel driver that hands the device's registers and interrupts to a user-space
program, here SPDK.
Our target runs without an IOMMU, hence uio_pci_generic; with an IOMMU, use vfio-pci the same way. SPDK's
scripts/setup.sh does this step and the hugepages in one go, for every NVMe drive that is not in use; by hand, you
move only the one you mean.
The last line says uio_pci_generic.
2. Start the SPDK target
nvmf_tgt is SPDK's NVMe-oF target. Start it in a second shell on the target; it runs until Ctrl-C.
cd ~/ntprof
sudo ../spdk/build/bin/nvmf_tgt -m 0x10000 -s 2048
| Option | What it does |
|---|---|
-m 0x10000 | the CPUs SPDK may use, as a bit mask: bit 16 set, so CPU 16 alone, one on the NIC's and the drive's NUMA node. SPDK's thread polls without pause, so that CPU stays 100 % busy |
-s 2048 | 2048 MiB of hugepage memory for SPDK's buffers |
[2026-10-17 06:14:22.297099] Starting SPDK v26.05 git sha1 d519b163c / DPDK 26.03.0 initialization...
…
[2026-10-17 06:14:22.924750] reactor.c: 996:reactor_run: *NOTICE*: Reactor started on core 16
Everything else is set up through SPDK's RPC client, scripts/rpc.py, from the first shell. Each call changes the
running target; nothing is saved. First the transport:
sudo ../spdk/scripts/rpc.py nvmf_create_transport -t TCP
Then the drive. SPDK calls a block device a bdev: attaching the drive's controller over PCIe creates the bdev
Nvme0n1 (controller Nvme0, namespace 1), and a split makes a 64 GiB part of it, Nvme0n1p0: the region filled
in part 2.
sudo ../spdk/scripts/rpc.py bdev_nvme_attach_controller \
-b Nvme0 -t PCIe -a 0000:d0:00.0
sudo ../spdk/scripts/rpc.py bdev_split_create -s 65536 Nvme0n1 1
| Call | What it does |
|---|---|
bdev_nvme_attach_controller -b Nvme0 -t PCIe -a 0000:d0:00.0 | takes the drive at that PCI address with SPDK's NVMe driver, under the name Nvme0 |
bdev_split_create -s 65536 Nvme0n1 1 | cuts one part of 65536 MiB (64 GiB) from the start of Nvme0n1 |
Nvme0n1
Nvme0n1p0
Finally the export, as in part 2: a subsystem, its namespace, and the address it listens on.
sudo ../spdk/scripts/rpc.py nvmf_create_subsystem \
nqn.2026-10.edu.wisc.ntprof:tutorial -a -s NTPROF0000000001
sudo ../spdk/scripts/rpc.py nvmf_subsystem_add_ns \
nqn.2026-10.edu.wisc.ntprof:tutorial Nvme0n1p0
sudo ../spdk/scripts/rpc.py nvmf_subsystem_add_listener \
nqn.2026-10.edu.wisc.ntprof:tutorial -t tcp -a 10.101.0.5 -s 4421 -f ipv4
| Option | What it does |
|---|---|
-a | allows any host to connect, like attr_allow_any_host in part 2 |
-s NTPROF0000000001 | the serial number the hosts see |
Nvme0n1p0 | the bdev that becomes namespace 1 |
-t tcp -a 10.101.0.5 -s 4421 -f ipv4 | transport, address, port and address family of the listener |
These calls print nothing. The target's second shell confirms:
…
[2026-10-17 06:14:23.372962] tcp.c: 793:nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
…
[2026-10-17 06:14:25.076277] tcp.c:1117:nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 10.101.0.5 port 4421 ***
SPDK also logs notices and warnings about its own deprecated features while it starts; they do not matter here.
NVMe/TCP Target Listening on with your address and port.
3. Check and start recording on the target
With SPDK, ntprof needs to know which program to look at: --spdk-target names the nvmf_tgt binary.
sudo ./ntprof probe -r target -t tcp --spdk-target ../spdk/build/bin/nvmf_tgt
kernel BTF: yes
function hooks: fentry/fexit (they can attach now)
spdk_tcp_target (module BTF nvmet_tcp: absent)
USDT probes in ../spdk/build/bin/nvmf_tgt:
ok nvmf_tcp_done 3 sites
ok nvmf_rdma_done 1 site
ok nvme_tcp_done 1 site
ok nvme_rdma_done 1 site
running processes: 1523263 (sampling variable at 0x55db884a1e50)
ok ntp_s_recvmsg_fe tcp_recvmsg
ok ntp_s_recvmsg_kp tcp_recvmsg
ok ntp_s_copy_fe skb_copy_datagram_iter
ok ntp_s_copy_kp skb_copy_datagram_iter
ok ntp_s_sendmsg_fe tcp_sendmsg
ok ntp_s_sendmsg_kp tcp_sendmsg
ok ntp_u_nvmf_tcp_done USDT ntprof:nvmf_tcp_done (required)
load: ok
What the lines say:
- USDT probes: the probes the patch put in the binary (part 1).
- running processes: the
nvmf_tgtthat is running, and the variable in it through which ntprof tells SPDK how often to sample. Until ntprof writes it, SPDK samples nothing. tcp_recvmsg,tcp_sendmsg: SPDK's NVMe/TCP still sends and receives through the kernel's TCP stack, so ntprof also hooks it, for the time each command spends in the socket.module BTF nvmet_tcp: absentdoes not matter: the kernel's NVMe target driver plays no part now, so no--btf.
running processes: names a process (start nvmf_tgt first if it says none), and load: ok.
Start recording in a third shell:
cd ~/ntprof
sudo ./ntprof record -r target -t tcp -s 100 -i 10 \
--spdk-target ../spdk/build/bin/nvmf_tgt -o target.ntp
ntprof: profiling as target, sampling 1 of 100 commands, saving records; Ctrl-C to stop
4. Run spdk_nvme_perf and record on the host
spdk_nvme_perf needs hugepages too. We reserve 2048 pages of 2 MiB (4 GiB) on NUMA node 1, the node of the host's
NIC:
echo 2048 | sudo tee /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
grep -E 'HugePages_(Total|Free)' /proc/meminfo
2048
HugePages_Total: 2048
HugePages_Free: 2048
Start the workload in a second shell on the host. An SPDK host needs no nvme connect: perf connects by itself to
the address given with -r.
cd ~/ntprof
sudo ../spdk/build/bin/spdk_nvme_perf -q 1 -o 4096 -w randread -t 30 -c 0x10000 \
-r 'trtype:TCP adrfam:IPv4 traddr:10.101.0.5 trsvcid:4421 subnqn:nqn.2026-10.edu.wisc.ntprof:tutorial'
| Option | What it does |
|---|---|
-q 1 | one read at a time (queue depth 1), like fio's --iodepth=1 |
-o 4096 -w randread | 4 KiB reads at random offsets |
-t 30 | 30 seconds |
-c 0x10000 | CPU 16, a mask as for nvmf_tgt |
-r '…' | the target: transport, address family, address, port and subsystem NQN |
While it runs, check and record in the first shell. --spdk-host names the perf binary:
sudo ./ntprof probe -r host -t tcp --spdk-host ../spdk/build/bin/spdk_nvme_perf
sudo ./ntprof record -t tcp -s 100 -d 20 -i 10 \
--spdk-host ../spdk/build/bin/spdk_nvme_perf -o host.ntp
kernel BTF: yes
function hooks: fentry/fexit (they can attach now)
spdk_tcp_host (module BTF nvme_tcp: absent)
USDT probes in ../spdk/build/bin/spdk_nvme_perf:
ok nvme_tcp_done 1 site
ok nvme_rdma_done 1 site
running processes: 2696064 (sampling variable at 0x560e46308d70)
ok ntp_s_recvmsg_fe tcp_recvmsg
ok ntp_s_recvmsg_kp tcp_recvmsg
ok ntp_s_copy_fe skb_copy_datagram_iter
ok ntp_s_copy_kp skb_copy_datagram_iter
ok ntp_u_nvme_tcp_done USDT ntprof:nvme_tcp_done (required)
load: ok
ntprof: profiling as host, sampling 1 of 100 commands, saving records; Ctrl-C to stop
every 10 s, per group: sampled commands, rate, latency, and each stage's share of the mean
(*: network and target together; the target's side is not recorded)
host target
time group n cmds/s mean us p99 us block driver network driver storage
10.0s read 4KiB spdk-c1n1 1214 12.2k 82.39 91.36 0% 4% 96%*
ntprof: 2435 records written to host.ntp
=== 20.0 s, one command in 100 sampled ===
throughput (estimated from the sampled commands): read 12.2 kIOPS, 49.8 MB/s
== read 4KiB spdk-c1n1: 2435 commands sampled (from about 12.2k commands/s), host side only
latency (us): mean 82.34, p50 79.33, p99 91.25, p99.9 92.29
stage mean us share avg QD
host driver (SPDK) 3.22 4% 0.04 #
host receive path 4.35 5% 0.05 #
network and target (not recorded) 74.77 91% 0.91 ######################
largest: network and target (not recorded), 91 % of the latency
records: 2435 host
- An SPDK host has no block device and no block layer: ntprof names the drive by its controller and namespace
(
spdk-c1n1) and estimates the throughput from the sampled commands. module BTF nvme_tcp: absentonly means the kernel'snvme_tcpmodule is not loaded; SPDK's host does not use it.
perf ends ten seconds later with its own measurement:
…
Initializing NVMe Controllers
Attached to NVMeoF Controller at TCP (addr:10.101.0.5 subnqn:nqn.2026-10.edu.wisc.ntprof:tutorial)
Associating TCP (addr:10.101.0.5 subnqn:nqn.2026-10.edu.wisc.ntprof:tutorial) NSID 1 with lcore 16
Initialization complete. Launching workers.
========================================================
Latency(us)
Device Information : IOPS MiB/s Average min max
TCP (addr:10.101.0.5 subnqn:nqn.2026-10.edu.wisc.ntprof:tutorial) NSID 1 from core 16: 12181.13 47.58 82.08 76.64 328.60
========================================================
Total : 12181.13 47.58 82.08 76.64 328.60
EAL: No free 2048 kB hugepages reported on node 0 only says that node 0 has no hugepages; ours are on node 1. A
CQ transport error -6 (No such device or address) at the end comes from perf closing its connection when the run
is over; it is printed out of order because SPDK writes errors without buffering.
ntprof's mean (82.34 µs) and perf's Average (82.08 µs) agree within a fraction of a microsecond.
5. Stop the target's recording and join
Press Ctrl-C in the target's third shell:
ntprof: profiling as target, sampling 1 of 100 commands, saving records; Ctrl-C to stop
ntprof: 3616 records written to target.ntp
records: 3616 target
report: ntprof report HOST.ntp target.ntp (the host's recording, same -s)
scp target:ntprof/target.ntp .
./ntprof report --detail host.ntp target.ntp
recorded on madracks-gnode4 (kernel 5.15.143), one command in 100 sampled
== read 4KiB spdk-c1n1: 2435 commands sampled (from about 12.2k commands/s), all with the target's side
latency (us): mean 82.34, p50 79.33, p99 91.25, p99.9 92.29
stage mean us share avg QD
host driver (SPDK) 3.22 4% 0.04 #
network (TCP/IP, NICs, wire) 18.80 23% 0.23 #####
target driver (SPDK) 0.65 1% 0.01
storage (SPDK bdev, drive) 59.67 72% 0.73 #################
largest: storage (SPDK bdev, drive), 72 % of the latency
components, in path order (L: commands in each, by Little's law)
component n mean L p50 p99 (us)
host block layer, submission 2435 0.00 0.00 0.00 0.00
host driver, submission 2435 3.07 0.04 3.06 3.30
network: wire and NICs (computed) 2435 10.19 0.12 10.05 11.78
network: target receive path 2435 4.26 0.05 4.05 5.96
target driver, submission 2435 0.06 0.00 0.06 0.09
storage: target block layer and drive 2435 59.67 0.73 56.34 66.95
target driver, completion 2435 0.59 0.01 0.60 0.69
network: host receive path 2435 4.35 0.05 4.34 4.72
host driver, completion 2435 0.15 0.00 0.15 0.20
host block layer, completion 2435 0.00 0.00 0.00 0.00
handoff to nvme_complete_rq 2435 0.00 0.00 0.00 0.00
end to end 2435 82.34 1.00 79.33 91.25
records: 2435 host, 3616 target; 2435 matched (100 % of the host's)
The stage names say where SPDK's own code runs: "host driver (SPDK)", "target driver (SPDK)", and "storage (SPDK bdev, drive)", SPDK's block layer and the drive under its NVMe driver.
All the host's commands matched.
Linux and SPDK side by side
The same read over NVMe/TCP, on the same machines and drive:
| Stage (µs) | Linux NVMe/TCP (part 2) | SPDK NVMe/TCP |
|---|---|---|
| host block layer | 0.57 | (none) |
| host driver | 3.29 | 3.22 |
| network | 19.91 | 18.80 |
| target driver | 2.50 | 0.65 |
| storage | 60.13 | 59.67 |
| end to end | 86.39 | 82.34 |
- The network costs both about the same: SPDK's NVMe/TCP still goes through the kernel's TCP stack on both machines; its receive paths (4.3 µs on each side) are that stack's work.
- The target driver is where SPDK saves most, mostly when the read completes (1.85 µs with
nvmet-tcp, 0.59 µs with SPDK):nvmet-tcphands the response to a kernel worker thread that sends it, while SPDK's polling thread sends it at once. - Storage is 0.5 µs lower through SPDK's own NVMe driver.
6. Clean up
Give back the host's hugepages:
echo 0 | sudo tee /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
0
Stop the target with Ctrl-C in its second shell, then return the drive to Linux: the same three files as in step 1, with the override cleared.
echo 0000:d0:00.0 | sudo tee /sys/bus/pci/devices/0000:d0:00.0/driver/unbind
echo | sudo tee /sys/bus/pci/devices/0000:d0:00.0/driver_override
echo 0000:d0:00.0 | sudo tee /sys/bus/pci/drivers_probe
basename $(readlink /sys/bus/pci/devices/0000:d0:00.0/driver); lsblk -d -o NAME,SERIAL /dev/nvme1n1
0000:d0:00.0
0000:d0:00.0
nvme
NAME SERIAL
nvme1n1 S728NS0X304953P
The drive is back on nvme with its own serial number. Its name can change when it comes back; the serial tells you
it is the same drive.
Next
5. SPDK NVMe/RDMA: the same over RDMA.