Skip to main content
NSDI ’25Understanding and profiling NVMe-over-TCP

See where every NVMe-over-Fabrics I/O spends its time.

Stage-by-stage latency of every I/O, from the host to the drive, with eBPF on unmodified Linux.

Get startedHow it works
No kernel patchNo rebootHost and target joined
host — NVMe/RDMA
$ ntprof report host.ntp target.ntp
== read 4KiB nvme6n1: 3066 commands sampled (from about 13.3k commands/s), all with the target's side
latency (us): mean 72.77, p50 69.54, p99 80.30, p99.9 81.83
host block layer0.491%
host driver (nvme-rdma)1.262%
network (RDMA, NICs, wire)10.0314%
target driver (nvmet-rdma)1.021%
storage (target block layer, drive)59.9782%
largest: storage (target block layer, drive), 82 % of the latency
$
0kernel patches

eBPF on existing hooks.

2transports

NVMe/TCP and NVMe/RDMA; TCP in SPDK too.

+0.8µs per I/O

4 KiB read, both sides on.

100% joined

Host and target, no clock sync.

The model

One I/O, every stage of its path

Click a stage to see what it covers.

Host
Network
Target
Storage: the drive

The SSD itself: from the request’s issue to the drive until its completion.

4 KiB read, queue depth 159.5 µsNVMe/TCP59.4 µsNVMe/RDMA

What it answers

Find the stage that breaks, not just the latency that grew

Compare a baseline with a loaded run: ntprof compare names the stage that slows down.

Is it the SSD?

A busy drive shows up in storage

A second writer on the SSD: storage 2.7×, the rest unchanged.

alone
72.9 µs
busy drive
173.3 µs
Read the use case →Is it the fabric?

A busy link shows up in the network

A 392 Gb/s RDMA flow: network 1.9×, storage unchanged.

alone
72.8 µs
busy link
81.5 µs
Read the use case →Why does latency grow with load?

Queue depth 1 → 32 over NVMe/TCP

Network 9.5×, SSD 1.1×: one TCP queue is the limit.

qd 1
86.7 µs
qd 32
275.4 µs
Read the use case →Network or storage, per request?

NVMe/RDMA: data transfer vs drive

Each request’s RDMA transfer and drive time, from one run.

data transfer (network) drive (storage)µs
4 KiB read
7.1 / 60.0
128 KiB read
12.3 / 93.7
128 KiB write
15.8 / 39.0
Read the use case →

Measured on real hardware

Every class of I/O, split into its stages

Same drive, two transports. Hover a segment for its time.

Host block layerHost driverNetworkTarget driverStorage
4 KiB read
NVMe/TCP
86.7 µs
NVMe/RDMA
72.8 µs
4 KiB write
NVMe/TCP
33.5 µs
NVMe/RDMA
19.7 µs
128 KiB read
NVMe/TCP
182.6 µs
NVMe/RDMA
121.2 µs
128 KiB write
NVMe/TCP
164.0 µs
NVMe/RDMA
76.8 µs

Means at queue depth 1 · two servers, 400 GbE RoCE, Samsung PM9A3 · NVMe/TCP’s 128 KiB numbers vary by connection.

How it works

Three steps, no kernel changes

Step 1

Hook what the kernel already has

eBPF on existing tracepoints and driver functions. No patch, no reboot.

Step 2

Sample the same commands on both sides

Host and target keep the same one command in N.

Step 3

Join, then model the stages

Both halves matched offline; no clock sync.

The hooks, the stage model and its limits →

Get started

Profile a running workload in minutes

One binary. Needs root and BTF.

Installation guide →
install
git clone --recurse-submodules \
    https://github.com/netlab-wisconsin/ntprof.git
cd ntprof
make && sudo make install          # /usr/local/sbin/ntprof
sudo ntprof probe -r both          # which hooks this kernel offers
# on the target: start first, stop last
sudo ntprof record -r target -t rdma -o target.ntp
# on the host, while the workload runs
sudo ntprof record -t rdma -o host.ntp -d 30
# anywhere, with both files
ntprof report host.ntp target.ntp

Research

Understanding and Profiling NVMe-over-TCP Using ntprof

Yuyuan Kang and Ming Liu · University of Wisconsin–Madison
22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI ’25)

Paper at USENIX →Citing ntprof

Where does your NVMe-oF latency go?

Two commands on two machines tell you, stage by stage.

Get startedView on GitHub