Overview
ntprof answers one question: where does the time of an NVMe-over-Fabrics I/O go? For a sample of the reads and writes an application sends to a remote NVMe namespace, it measures how long each one spends in every stage of its path: the host's block layer and NVMe-oF driver, the network, the target's NVMe-oF driver, and the target's storage (its block layer and drive). These pages call the machine that sends the I/O the host (the NVMe-oF initiator) and the machine with the drive the target. ntprof reports each stage's share of the latency and how many commands wait in it, and comparing two runs, it names the stage that amplifies latency under load.
The stages
ntprof follows the model of the NSDI '25 paper: the NVMe-oF path is a lossless network of software switches, and each stage is a queue.
| Stage | Where | What it covers |
|---|---|---|
| host block layer | host | blk-mq, on the way down and back up |
| host driver | host | the NVMe-oF driver: nvme-tcp or nvme-rdma (or SPDK's NVMe/TCP host) |
| network | between the machines | the NICs, the wire, and the TCP/IP or RDMA receive paths on both machines |
| target driver | target | the NVMe target's driver: nvmet-tcp or nvmet-rdma (or SPDK's nvmf_tgt) |
| storage | target | the target's block layer and the drive |
A command's latency is the sum of its stages. With --detail, each stage splits further into its components:
the two receive paths, the drive itself, and over NVMe/RDMA, each request's data transfer.
What it supports
| Host | Target | |
|---|---|---|
| Linux NVMe/TCP | nvme-tcp | nvmet-tcp |
| Linux NVMe/RDMA | nvme-rdma | nvmet-rdma |
| SPDK NVMe/TCP | lib/nvme (for example spdk_nvme_perf) | nvmf_tgt |
- Kernels: unmodified Linux, tested on 5.15 (with MLNX OFED's NVMe modules) and on 6.8 and 6.11. ntprof attaches eBPF programs to tracepoints and functions the kernel already has; nothing is patched, built into the kernel or rebooted.
- Mixed setups: a kernel host against an SPDK target and the reverse join in one report.
- Reads and writes of any size, including writes that the target fetches with R2T (TCP) or RDMA READ.
What a report looks like
4 KiB random reads over NVMe/RDMA at queue depth 1, from one server to another (ConnectX-7 400 GbE, a Samsung PM9A3 behind the kernel's NVMe target), both sides recorded and joined:
== read 4KiB nvme6n1: 3066 commands sampled (from about 13.3k commands/s), all with the target's side
latency (us): mean 72.77, p50 69.54, p99 80.30, p99.9 81.83
stage mean us share avg QD
host block layer 0.49 1% 0.01
host driver (nvme-rdma) 1.26 2% 0.02
network (RDMA, NICs, wire) 10.03 14% 0.13 ###
target driver (nvmet-rdma) 1.02 1% 0.01
storage (target block layer, drive) 59.97 82% 0.80 ####################
largest: storage (target block layer, drive), 82 % of the latency
Storage, the target's block layer and drive, takes 82 % of this read. Over NVMe/TCP the same read spends twice as long in the network (19.7 µs instead of 10.0 µs). Reading a report explains every column.
Where to go next
- Getting started: build, check the kernel, profile a workload.
- Use cases: a busy drive, a busy network, queue depth, network vs storage per request.
- How it works: the hooks, the sampling, the join and the stage model.