2. Linux NVMe/TCP
The target exports its drive with the kernel's NVMe/TCP target driver (nvmet-tcp); the host connects with the
kernel's NVMe/TCP host driver (nvme-tcp) and reads the drive with fio. ntprof records each machine on its own,
and at the end the two records are joined into one breakdown of where each read's time went.
host.ntptarget.ntpntprof report host.ntp target.ntp joins the two files afterwards.- You need: ntprof on both machines (part 1), fio and nvme-cli on the host, root on both.
- You do: prepare the drive, export it, connect, record both sides while fio reads, join the two records, clean up.
- Time: about 20 minutes. Every command runs in
~/ntprof.
1. Prepare the drive
First make sure which drive you are about to use. lsblk -d lists whole drives only (no partitions), and -o
picks the columns; the serial number is what identifies a drive for certain.
lsblk -d -o NAME,MODEL,SERIAL,SIZE /dev/nvme1n1
NAME MODEL SERIAL SIZE
nvme1n1 SAMSUNG MZQL21T9HCJR-00A07 S728NS0X304953P 1.7T
Next, write data to the part of the drive the reads will use. A block that was never written is not on the flash at all: the drive answers a read of it from its mapping table in a few microseconds, and the measurement would show a drive far faster than it is.
The next command overwrites the first 64 GiB of the drive. Use a drive that holds nothing you need.
sudo fio --name=fill --filename=/dev/nvme1n1 \
--rw=write --bs=1M --iodepth=32 --ioengine=io_uring \
--direct=1 --size=64G --refill_buffers=1
| Option | What it does |
|---|---|
--filename=/dev/nvme1n1 | writes to the raw drive, no file system |
--rw=write --bs=1M | sequential writes of 1 MiB each, the fastest way to fill a region |
--iodepth=32 --ioengine=io_uring | keeps 32 writes in flight through io_uring |
--direct=1 | bypasses the page cache, so every write goes to the drive |
--size=64G | the first 64 GiB, the region the reads use later |
--refill_buffers=1 | fresh data for every write, so nothing the drive could compress |
fill: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=32
…
WRITE: bw=2681MiB/s (2811MB/s), 2681MiB/s-2681MiB/s (2811MB/s-2811MB/s), io=64.0GiB (68.7GB), run=24448-24448msec
io=64.0GiB was written. It took us 24 seconds; you need to do this once per drive, not before every run.
2. Export the drive
The kernel's NVMe-oF target is configured through files in configfs, under /sys/kernel/config/nvmet. An export
has three parts:
- a subsystem, named by an NQN (NVMe Qualified Name, by convention
nqn.<year>-<month>.<reversed domain>:<name>), which is what a host connects to; - a namespace inside it, numbered from 1, backed by a block device;
- a port: an address, a TCP port and a transport, through which the subsystem is offered.
Load the target driver for TCP first:
sudo modprobe nvmet-tcp
lsmod | grep '^nvmet'
nvmet_tcp 28672 0
…
Create the subsystem and its namespace. Each tee prints back the value it wrote:
cd /sys/kernel/config/nvmet/subsystems
sudo mkdir nqn.2026-10.edu.wisc.ntprof:tutorial
cd nqn.2026-10.edu.wisc.ntprof:tutorial
echo 1 | sudo tee attr_allow_any_host
sudo mkdir namespaces/1
echo -n /dev/nvme1n1 | sudo tee namespaces/1/device_path; echo
echo 1 | sudo tee namespaces/1/enable
1
/dev/nvme1n1
1
| File | What it sets |
|---|---|
attr_allow_any_host | 1: any host may connect, without a list of allowed host NQNs |
namespaces/1/device_path | the block device behind namespace 1 |
namespaces/1/enable | 1: the namespace is served from now on |
attr_allow_any_host leaves the export open to every machine that can reach the port. That is fine on a private
test network; elsewhere, list the allowed hosts under the subsystem's allowed_hosts instead.
Then the port, and a link that offers the subsystem on it:
cd /sys/kernel/config/nvmet/ports
sudo mkdir 11 && cd 11
echo tcp | sudo tee addr_trtype
echo ipv4 | sudo tee addr_adrfam
echo 10.101.0.5 | sudo tee addr_traddr
echo 4420 | sudo tee addr_trsvcid
sudo ln -s \
/sys/kernel/config/nvmet/subsystems/nqn.2026-10.edu.wisc.ntprof:tutorial \
subsystems/
ls subsystems
cd ~/ntprof
tcp
ipv4
10.101.0.5
4420
nqn.2026-10.edu.wisc.ntprof:tutorial
| File | What it sets |
|---|---|
ports/11 | a port; the number is any one not yet taken (ls /sys/kernel/config/nvmet/ports) |
addr_trtype | the transport: tcp |
addr_adrfam, addr_traddr | the address family and the target's own address on the network the host uses |
addr_trsvcid | the TCP port; 4420 is the usual one for NVMe/TCP |
subsystems/<NQN> (link) | offers that subsystem on this port |
The last line names your subsystem: the drive is now exported.
3. Check that ntprof can hook the target
ntprof probe lists every hook ntprof needs for a role (-r target) and a transport (-t tcp), says whether
this kernel has it, and loads the programs without attaching them. Nothing is recorded.
sudo ./ntprof probe -r target -t tcp
ntprof: error: nvme_tcp_target: the nvmet_tcp module has no BTF (built out of tree?). Generate it from the DWARF of a debug build of the module with tools/gen-btf.sh and pass --btf DIR
kernel BTF: yes
function hooks: fentry/fexit (they can attach now)
nvme_tcp_target (module BTF nvmet_tcp: absent; kprobes)
cannot open object
This is what a module without BTF looks like. BTF is the type information ntprof uses to read the kernel's
data structures. The kernel itself has it (kernel BTF: yes), but our nvmet_tcp module comes from MLNX OFED's
DKMS packages, which install modules without it.
Your output then says module BTF nvmet_tcp: present and ends with load: ok: skip the next command and leave
out --btf in step 4.
The fix is to generate the BTF once from a debug build of the same module source, as
Kernels without BTF describes, and pass its directory with --btf. Ours is in ~/ntprof-btf:
sudo ./ntprof probe -r target -t tcp --btf ../ntprof-btf
kernel BTF: yes
function hooks: fentry/fexit (they can attach now)
nvme_tcp_target (module BTF nvmet_tcp: absent; kprobes)
ok ntp_t_req_init tracepoint nvmet_req_init (required)
ok ntp_t_exec_bdev_fe nvmet_bdev_execute_rw
ok ntp_t_exec_bdev_kp nvmet_bdev_execute_rw
…
ok ntp_t_queue_resp_fe nvmet_tcp_queue_response (required)
ok ntp_t_queue_resp_kp nvmet_tcp_queue_response (required)
…
load: ok
Each line is one hook: whether this kernel has it (ok), ntprof's program, and the kernel function or tracepoint
it attaches to. Most functions have two programs, an fentry/fexit one (_fe, _fx) and a kprobe one (_kp,
_kr); ntprof attaches one of them. Without the module's own BTF it uses kprobes, which cost slightly more per I/O.
Every (required) line says ok, and the last line says load: ok.
4. Start recording on the target
Open a second shell on the target and start recording there before the host starts. It runs until you stop it with Ctrl-C in step 8.
cd ~/ntprof
sudo ./ntprof record -r target -t tcp -s 100 -i 10 \
--btf ../ntprof-btf -o target.ntp
| Option | What it does |
|---|---|
-r target -t tcp | profiles the NVMe/TCP target |
-s 100 | records one command in 100. The host must use the same number: both sides pick the same commands from each command's ID and address, so the two records can be matched later |
-i 10 | a progress line every 10 seconds |
--btf ../ntprof-btf | the BTF made for our module (step 3) |
-o target.ntp | saves every record to this file |
ntprof: profiling as target, sampling 1 of 100 commands, saving records; Ctrl-C to stop
If it adds no NVMe-oF command seen in 5 s, that only means the host has not started yet.
5. Connect from the host
Load the host driver and ask the target what it offers. -t is the transport, -a the target's address, -s the
port:
sudo modprobe nvme-tcp
sudo nvme discover -t tcp -a 10.101.0.5 -s 4420
Discovery Log Number of Records 2, Generation counter 22
…
=====Discovery Log Entry 1======
trtype: tcp
adrfam: ipv4
subtype: nvme subsystem
treq: not specified, sq flow control disable supported
portid: 11
trsvcid: 4420
subnqn: nqn.2026-10.edu.wisc.ntprof:tutorial
traddr: 10.101.0.5
sectype: none
Entry 0, left out here, is the discovery service itself; entry 1 is the subsystem from step 2. Connect to it by its
NQN (-n):
sudo nvme connect -t tcp -a 10.101.0.5 -s 4420 \
-n nqn.2026-10.edu.wisc.ntprof:tutorial
nvme connect printed nothing for us. The remote drive appears as a new nvmeXnY device; sudo nvme list-subsys
shows which one carries your subsystem's NQN. On our host it is nvme6n1:
sudo nvme list-subsys /dev/nvme6n1
lsblk -d -o NAME,MODEL,SERIAL,SIZE /dev/nvme6n1
nvme-subsys6 - NQN=nqn.2026-10.edu.wisc.ntprof:tutorial
\
+- nvme6 tcp traddr=10.101.0.5 trsvcid=4420 src_addr=10.101.0.4 live optimized
NAME MODEL SERIAL SIZE
nvme6n1 Linux f21db5961bc730982302 1.7T
The model reads Linux because the kernel target answers for the drive, with a serial number it makes up; the size
is the real drive's.
The connection is live, and the size is your drive's.
6. Check that ntprof can hook the host
sudo ./ntprof probe -r host -t tcp
kernel BTF: yes
function hooks: fentry/fexit (they can attach now)
nvme_tcp_host (module BTF nvme_tcp: present)
ok ntp_h_setup_cmd tracepoint nvme_setup_cmd (required)
…
-- ntp_h_send_cmd_fx nvme_tcp_try_send_cmd_pdu
-- ntp_h_send_cmd_kp nvme_tcp_try_send_cmd_pdu
…
ok ntp_h_try_send_fe nvme_tcp_try_send (for nvme_tcp_try_send_cmd_pdu: inlined, used)
…
ok ntp_h_recv_skb_fe nvme_tcp_recv_skb (required)
…
load: ok
The host's nvme_tcp has its BTF (present), so no --btf here. A -- line is a function this kernel's build
has inlined into its caller: there is no function left to hook. ntprof then hooks the caller instead, and says so:
(for nvme_tcp_try_send_cmd_pdu: inlined, used). Which functions are inlined depends on how the kernel was built.
load: ok, and every (required) line says ok.
7. Run the workload and record
Start the workload in a second shell on the host: random 4 KiB reads, one at a time, for 30 seconds.
sudo fio --name=randread --filename=/dev/nvme6n1 \
--rw=randread --bs=4k --iodepth=1 --ioengine=io_uring --direct=1 \
--size=60G --time_based --runtime=30 --cpus_allowed=16
| Option | What it does |
|---|---|
--rw=randread --bs=4k | reads 4 KiB at random offsets |
--iodepth=1 | one read at a time: each read's latency is not mixed with queueing behind others |
--size=60G | the offsets stay inside the 64 GiB written in step 1 |
--time_based --runtime=30 | runs for 30 seconds, long enough for a 20-second recording to fit inside |
--cpus_allowed=16 | pins fio to CPU 16, on the same NUMA node as the NIC (cat /sys/class/net/<nic>/device/numa_node) |
While it runs, record 20 seconds in the first shell:
sudo ./ntprof record -t tcp -s 100 -d 20 -i 10 --session nvme6n1 -o host.ntp
| Option | What it does |
|---|---|
-t tcp | the host role is the default; the transport is TCP |
-s 100 | one command in 100, the same as the target |
-d 20 | stops after 20 seconds |
--session nvme6n1 | records only this drive, not any other NVMe-oF drive of the host |
-o host.ntp | saves the records |
ntprof: warning: nvme_tcp_host: nvme_tcp_try_send_cmd_pdu not available on this kernel; ntp_h_send_cmd_fx disabled
ntprof: warning: nvme_tcp_host: nvme_tcp_try_send_data not available on this kernel; ntp_h_send_data_fx disabled
ntprof: warning: nvme_tcp_host: nvme_tcp_try_send_data_pdu not available on this kernel; ntp_h_send_h2c_fe disabled
ntprof: profiling as host, sampling 1 of 100 commands, saving records; Ctrl-C to stop
every 10 s, per group: sampled commands, rate, latency, and each stage's share of the mean
(*: network and target together; the target's side is not recorded)
host target
time group n cmds/s mean us p99 us block driver network driver storage
10.0s read 4KiB nvme6n1 1140 11.4k 86.45 95.71 1% 4% 96%*
ntprof: 2298 records written to host.ntp
=== 20.0 s, one command in 100 sampled ===
throughput (all commands): read 11.3 kIOPS, 46.5 MB/s
== read 4KiB nvme6n1: 2298 commands sampled (from about 11.5k commands/s), host side only
latency (us): mean 86.39, p50 83.13, p99 95.60, p99.9 99.48
stage mean us share avg QD
host block layer 0.57 1% 0.01
host driver (nvme-tcp) 3.29 4% 0.04 #
host receive path 3.82 4% 0.04 #
rest of the data arriving 0.00 0% 0.00
network and target (not recorded) 78.72 91% 0.90 ######################
largest: network and target (not recorded), 91 % of the latency
records: 2298 host
- The three warnings are the inlined functions of step 6; nothing is missing.
- On its own, the host cannot tell the network from the target: their time is one row, "network and target (not
recorded)", and in the progress line its share carries a
*. Step 9 splits it.
fio ends ten seconds later in the second shell:
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
…
read: IOPS=11.4k, BW=44.4MiB/s (46.5MB/s)(1331MiB/30001msec)
slat (nsec): min=3435, max=255894, avg=4089.30, stdev=538.65
clat (nsec): min=440, max=6002.0k, avg=83622.31, stdev=13574.26
lat (usec): min=82, max=6006, avg=87.71, stdev=13.58
…
ntprof's mean (86.39 µs) is close to fio's lat average (87.71 µs). fio's lat also counts slat, the time to
submit a read before it reaches the block layer, where ntprof starts timing.
8. Stop the target's recording
Press Ctrl-C in the target's second shell. ntprof writes its file and says how to report it:
ntprof: profiling as target, sampling 1 of 100 commands, saving records; Ctrl-C to stop
ntprof: 3508 records written to target.ntp
records: 3508 target
report: ntprof report HOST.ntp target.ntp (the host's recording, same -s)
The target recorded all 30 seconds of fio and the host only 20, so the target has more records. That is expected: only the commands both sides recorded are joined.
9. Join the two records
Copy the target's file to the host (target stands for the target's host name or address) and report the two files
together:
scp target:ntprof/target.ntp .
./ntprof report host.ntp target.ntp
recorded on madracks-gnode4 (kernel 5.15.143), one command in 100 sampled
== read 4KiB nvme6n1: 2298 commands sampled (from about 11.5k commands/s), all with the target's side
latency (us): mean 86.39, p50 83.13, p99 95.60, p99.9 99.48
stage mean us share avg QD
host block layer 0.57 1% 0.01
host driver (nvme-tcp) 3.29 4% 0.04 #
network (TCP/IP, NICs, wire) 19.91 23% 0.23 ######
target driver (nvmet-tcp) 2.50 3% 0.03 #
storage (target block layer, drive) 60.13 70% 0.69 #################
largest: storage (target block layer, drive), 70 % of the latency
records: 2298 host, 3508 target; 2298 matched (100 % of the host's)
How to read it:
- The last line: each of the host's 2298 commands found its target side.
- The stages: the single row "network and target" of step 7 is now three rows: the network (19.9 µs), the target driver (2.5 µs) and storage (60.1 µs).
- share: each stage's part of the mean latency. Storage, the drive above all, takes 70 % of a 4 KiB read; the network comes next.
- avg QD: the mean number of commands in each stage at any moment. At queue depth 1 they add up to about 1.
matched (100 % of the host's), or close to it, and a breakdown of the same shape. If nothing matched, the two
sides used different -s, or the target was not recording while the host was.
--detail splits every stage into its parts, in the order a command passes them:
./ntprof report --detail host.ntp target.ntp
…
components, in path order (L: commands in each, by Little's law)
component n mean L p50 p99 (us)
host block layer, submission 2298 0.40 0.00 0.38 0.52
host driver, submission 2298 2.60 0.03 2.57 3.06
network: wire and NICs (computed) 2298 10.84 0.12 10.80 12.03
network: target receive path 2298 5.25 0.06 5.17 7.78
target driver, submission 2298 0.65 0.01 0.61 0.94
storage: target block layer and drive 2298 60.13 0.69 56.81 67.40
target block layer, submission 2298 0.53 0.01 0.52 0.62
drive 2298 59.50 0.68 56.18 66.75
target block layer, completion 2298 0.10 0.00 0.10 0.11
target driver, completion 2298 1.85 0.02 1.85 2.03
network: host receive path 2298 3.82 0.04 3.70 6.28
network: rest of the data arriving 2298 0.00 0.00 0.00 0.00
host driver, processing the data 2298 0.69 0.01 0.69 0.80
host driver, completion 2298 0.00 0.00 0.00 0.00
host block layer, completion 2298 0.17 0.00 0.17 0.21
handoff to nvme_complete_rq 2298 0.10 0.00 0.10 0.14
end to end 2298 86.39 0.99 83.13 95.60
…
The network's 19.9 µs are three parts: the wire and the two NICs (10.8 µs, which ntprof computes from the two sides' times), the target's receive path (5.3 µs) and the host's (3.8 µs). The drive itself is 59.5 µs of storage's 60.1. Reading a report explains every row.
10. Clean up
On the host, disconnect and unload the driver:
sudo nvme disconnect -n nqn.2026-10.edu.wisc.ntprof:tutorial
sudo modprobe -r nvme-tcp
NQN:nqn.2026-10.edu.wisc.ntprof:tutorial disconnected 1 controller(s)
On the target, undo step 2 in reverse order: the link, the port, the namespace (disabled first), the subsystem, then the driver.
sudo rm \
/sys/kernel/config/nvmet/ports/11/subsystems/nqn.2026-10.edu.wisc.ntprof:tutorial
sudo rmdir /sys/kernel/config/nvmet/ports/11
cd /sys/kernel/config/nvmet/subsystems
echo 0 | sudo tee nqn.2026-10.edu.wisc.ntprof:tutorial/namespaces/1/enable
sudo rmdir nqn.2026-10.edu.wisc.ntprof:tutorial/namespaces/1 \
nqn.2026-10.edu.wisc.ntprof:tutorial
cd ~/ntprof
sudo modprobe -r nvmet-tcp
0
ls /sys/kernel/config/nvmet/ports no longer lists port 11. modprobe -r refuses if something still uses the
module; that is safe to leave loaded.
Next
3. Linux NVMe/RDMA: the same over RDMA.