3. Linux NVMe/RDMA
The same read as part 2, over RDMA: the target exports the drive with the kernel's NVMe/RDMA target
driver (nvmet-rdma) and the host connects with its NVMe/RDMA host driver (nvme-rdma). With RDMA the NICs move
the data between the two machines' memory themselves; neither CPU copies it through a network stack.
host.ntptarget.ntpntprof report host.ntp target.ntp joins the two files afterwards.- You need: ntprof on both machines (part 1), RDMA NICs on both (RoCE here), fio and nvme-cli on the host. If you skipped part 2, fill the drive first (part 2, step 1).
- What changes from part 2: the driver modules,
rdmaas the transport in configfs, in nvme-cli and in ntprof (-t rdma), and the port number. Everything else is the same. - Time: about 10 minutes. Every command runs in
~/ntprof.
1. Export the drive
Load the target driver for RDMA:
sudo modprobe nvmet-rdma
lsmod | grep '^nvmet'
nvmet_rdma 57344 1
…
Then the same subsystem and namespace as in part 2, and a port whose transport is rdma. We use port 4421; any port
that is free on the target works.
cd /sys/kernel/config/nvmet/subsystems
sudo mkdir nqn.2026-10.edu.wisc.ntprof:tutorial
cd nqn.2026-10.edu.wisc.ntprof:tutorial
echo 1 | sudo tee attr_allow_any_host
sudo mkdir namespaces/1
echo -n /dev/nvme1n1 | sudo tee namespaces/1/device_path; echo
echo 1 | sudo tee namespaces/1/enable
cd /sys/kernel/config/nvmet/ports
sudo mkdir 12 && cd 12
echo rdma | sudo tee addr_trtype
echo ipv4 | sudo tee addr_adrfam
echo 10.101.0.5 | sudo tee addr_traddr
echo 4421 | sudo tee addr_trsvcid
sudo ln -s \
/sys/kernel/config/nvmet/subsystems/nqn.2026-10.edu.wisc.ntprof:tutorial \
subsystems/
ls subsystems
cd ~/ntprof
1
/dev/nvme1n1
1
rdma
ipv4
10.101.0.5
4421
nqn.2026-10.edu.wisc.ntprof:tutorial
Part 2, step 2 explains each file. For RDMA, addr_traddr must be an address
of the RDMA NIC itself.
The last line names your subsystem.
2. Check and start recording on the target
Our target's nvmet_rdma module has no BTF either (see part 2, step 3),
so ntprof gets it with --btf:
sudo ./ntprof probe -r target -t rdma --btf ../ntprof-btf
kernel BTF: yes
function hooks: fentry/fexit (they can attach now)
nvme_rdma_target (module BTF nvmet_rdma: absent; kprobes)
ok ntp_t_req_init tracepoint nvmet_req_init (required)
…
ok ntp_tr_recv_done_fe nvmet_rdma_recv_done (required)
…
ok ntp_tr_read_done_fe nvmet_rdma_read_data_done
…
ok ntp_tr_send_done_fe nvmet_rdma_send_done (required)
…
ok ntp_tr_queue_resp_fe nvmet_rdma_queue_response (required)
…
load: ok
The RDMA hooks (ntp_tr_…) sit on the target driver's completion handlers: a command received, a write's data read
from the host, a response sent.
Every (required) line says ok, and load: ok.
Start recording in a second shell, with -t rdma:
cd ~/ntprof
sudo ./ntprof record -r target -t rdma -s 100 -i 10 \
--btf ../ntprof-btf -o target.ntp
ntprof: profiling as target, sampling 1 of 100 commands, saving records; Ctrl-C to stop
ntprof: warning: no NVMe-oF command seen in 5 s: is I/O running on an NVMe/RDMA device?
The warning only means the host has not started yet.
3. Connect from the host
sudo modprobe nvme-rdma
sudo nvme discover -t rdma -a 10.101.0.5 -s 4421
Discovery Log Number of Records 2, Generation counter 25
…
=====Discovery Log Entry 1======
trtype: rdma
adrfam: ipv4
subtype: nvme subsystem
treq: not specified, sq flow control disable supported
portid: 12
trsvcid: 4421
subnqn: nqn.2026-10.edu.wisc.ntprof:tutorial
traddr: 10.101.0.5
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms: rdma-cm
rdma_pkey: 0x0000
The rdma_… lines are RDMA's own connection parameters: a reliable connected queue pair (connected), set up
through the RDMA connection manager (rdma-cm).
sudo nvme connect -t rdma -a 10.101.0.5 -s 4421 \
-n nqn.2026-10.edu.wisc.ntprof:tutorial
sudo nvme list-subsys /dev/nvme6n1
lsblk -d -o NAME,MODEL,SERIAL,SIZE /dev/nvme6n1
nvme-subsys6 - NQN=nqn.2026-10.edu.wisc.ntprof:tutorial
\
+- nvme6 rdma traddr=10.101.0.5 trsvcid=4421 live optimized
NAME MODEL SERIAL SIZE
nvme6n1 Linux 0c3ffa91c873d43eba3b 1.7T
rdma and live. The serial number is made up anew at every export, so it differs from part 2's.
4. Run the workload and record on the host
sudo ./ntprof probe -r host -t rdma
kernel BTF: yes
function hooks: fentry/fexit (they can attach now)
nvme_rdma_host (module BTF nvme_rdma: present)
ok ntp_h_setup_cmd tracepoint nvme_setup_cmd (required)
…
ok ntp_hr_queue_rq_fe nvme_rdma_queue_rq (required)
…
ok ntp_hr_send_done_fe nvme_rdma_send_done (required)
…
ok ntp_hr_recv_done_fe nvme_rdma_recv_done (required)
…
load: ok
The same fio job as in part 2 (its options), in a second shell:
sudo fio --name=randread --filename=/dev/nvme6n1 \
--rw=randread --bs=4k --iodepth=1 --ioengine=io_uring --direct=1 \
--size=60G --time_based --runtime=30 --cpus_allowed=16
And the recording while it runs, with -t rdma:
sudo ./ntprof record -t rdma -s 100 -d 20 -i 10 --session nvme6n1 -o host.ntp
ntprof: profiling as host, sampling 1 of 100 commands, saving records; Ctrl-C to stop
every 10 s, per group: sampled commands, rate, latency, and each stage's share of the mean
(*: network and target together; the target's side is not recorded)
host target
time group n cmds/s mean us p99 us block driver network driver storage
10.0s read 4KiB nvme6n1 1317 13.2k 72.85 80.20 1% 2% 98%*
ntprof: 2646 records written to host.ntp
=== 20.0 s, one command in 100 sampled ===
throughput (all commands): read 13.1 kIOPS, 53.8 MB/s
== read 4KiB nvme6n1: 2646 commands sampled (from about 13.2k commands/s), host side only
latency (us): mean 72.78, p50 69.52, p99 80.22, p99.9 81.39
stage mean us share avg QD
host block layer 0.50 1% 0.01
host driver (nvme-rdma) 1.27 2% 0.02
host receive path 0.22 0% 0.00
network and target (not recorded) 70.79 97% 0.94 #######################
largest: network and target (not recorded), 97 % of the latency
records: 2646 host
No warnings this time: nothing ntprof hooks in nvme-rdma is inlined on our host. fio ends ten seconds later:
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
…
read: IOPS=13.2k, BW=51.4MiB/s (53.9MB/s)(1542MiB/30001msec)
slat (nsec): min=2240, max=230531, avg=2758.75, stdev=431.88
clat (nsec): min=410, max=1005.8k, avg=72957.43, stdev=5726.90
lat (usec): min=70, max=1008, avg=75.72, stdev= 5.74
…
ntprof's mean (72.78 µs) against fio's lat (75.72 µs): fio also counts the submission before the block layer
(slat, 2.8 µs).
5. Stop the target's recording and join
Press Ctrl-C in the target's second shell:
ntprof: profiling as target, sampling 1 of 100 commands, saving records; Ctrl-C to stop
ntprof: warning: no NVMe-oF command seen in 5 s: is I/O running on an NVMe/RDMA device?
ntprof: 3903 records written to target.ntp
records: 3903 target
report: ntprof report HOST.ntp target.ntp (the host's recording, same -s)
Copy the target's file to the host and report the two together, with every component:
scp target:ntprof/target.ntp .
./ntprof report --detail host.ntp target.ntp
recorded on madracks-gnode4 (kernel 5.15.143), one command in 100 sampled
== read 4KiB nvme6n1: 2646 commands sampled (from about 13.2k commands/s), all with the target's side
latency (us): mean 72.78, p50 69.52, p99 80.22, p99.9 81.39
stage mean us share avg QD
host block layer 0.50 1% 0.01
host driver (nvme-rdma) 1.27 2% 0.02
network (RDMA, NICs, wire) 10.00 14% 0.13 ###
target driver (nvmet-rdma) 1.01 1% 0.01
storage (target block layer, drive) 60.00 82% 0.79 ####################
largest: storage (target block layer, drive), 82 % of the latency
components, in path order (L: commands in each, by Little's law)
component n mean L p50 p99 (us)
host block layer, submission 2646 0.20 0.00 0.19 0.25
host driver, submission 2646 1.27 0.02 1.26 1.47
network: wire and NICs (computed) 2646 9.53 0.13 9.51 10.94
data transfer (RDMA, at the target) 2646 7.12 0.09 7.10 7.71
network: target receive path 2646 0.25 0.00 0.24 0.40
target driver, submission 2646 0.56 0.01 0.56 0.75
storage: target block layer and drive 2646 60.00 0.79 56.77 67.31
target block layer, submission 2646 0.50 0.01 0.50 0.58
drive 2646 59.41 0.79 56.17 66.72
target block layer, completion 2646 0.09 0.00 0.09 0.11
target driver, completion 2646 0.44 0.01 0.44 0.56
network: host receive path 2646 0.22 0.00 0.21 0.30
host driver, completion 2646 0.00 0.00 0.00 0.00
host block layer, completion 2646 0.30 0.00 0.32 0.42
handoff to nvme_complete_rq 2646 0.25 0.00 0.26 0.34
end to end 2646 72.78 0.96 69.52 80.22
records: 2646 host, 3903 target; 2646 matched (100 % of the host's)
All the host's commands matched, and storage is again about 60 µs: the same drive.
One component is new over RDMA: data transfer (RDMA, at the target), 7.1 µs. It is the RDMA WRITE that carries the read's data and its response to the host, from the moment the target posts it to the moment the send completes: each read's time in the network, measured apart from its time in storage. NVMe/RDMA explains it.
TCP and RDMA side by side
The same read on the same machines and drive:
| Stage (µs) | NVMe/TCP (part 2) | NVMe/RDMA |
|---|---|---|
| host block layer | 0.57 | 0.50 |
| host driver | 3.29 | 1.27 |
| network | 19.91 | 10.00 |
| target driver | 2.50 | 1.01 |
| storage | 60.13 | 60.00 |
| end to end | 86.39 | 72.78 |
RDMA saves 13.6 µs, all of it in the network and the two drivers, where no TCP stack runs; storage does not change.
6. Clean up
sudo nvme disconnect -n nqn.2026-10.edu.wisc.ntprof:tutorial
NQN:nqn.2026-10.edu.wisc.ntprof:tutorial disconnected 1 controller(s)
sudo rm \
/sys/kernel/config/nvmet/ports/12/subsystems/nqn.2026-10.edu.wisc.ntprof:tutorial
sudo rmdir /sys/kernel/config/nvmet/ports/12
cd /sys/kernel/config/nvmet/subsystems
echo 0 | sudo tee nqn.2026-10.edu.wisc.ntprof:tutorial/namespaces/1/enable
sudo rmdir nqn.2026-10.edu.wisc.ntprof:tutorial/namespaces/1 \
nqn.2026-10.edu.wisc.ntprof:tutorial
cd ~/ntprof
0
Unload them when nothing else uses them: sudo modprobe -r nvme-rdma on the host, sudo modprobe -r nvmet-rdma on
the target.
Next
4. SPDK NVMe/TCP: the same drive with SPDK on both machines, over TCP; then 5. SPDK NVMe/RDMA.