Paper

Understanding and Profiling NVMe-over-TCP Using ntprof. Yuyuan Kang and Ming Liu, University of Wisconsin–Madison. 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI '25), pages 1117–1136. Paper and talk at USENIX.

The paper treats the NVMe/TCP storage path as a lossless switched network: each kernel module on the path is a software switch with a profiling agent, and a query interface collects and analyzes their statistics. It uses the profiles to explain where NVMe/TCP spends its time and in case studies of software bottlenecks (too few cores or connections), hardware bottlenecks (a busy SSD, a busy NIC), interference between workloads, and applications.

The paper's stage names

The paper's Figure 4 numbers the stages S1–S9; ntprof reports them under five names:

ntprof's stage	The paper's stages
host block layer	S1
host driver	S2
network	S3–S5
target driver	S6
storage	S7–S9 (the drive alone: S8–S9)

The paper's suffixes (S) and (C), submission and completion, appear in --detail as the components named "submission" and "completion" (in the network: the target's and the host's receive path).

Citing ntprof

@inproceedings{kang2025ntprof,
  author    = {Yuyuan Kang and Ming Liu},
  title     = {Understanding and Profiling {NVMe-over-TCP} Using ntprof},
  booktitle = {22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25)},
  year      = {2025},
  pages     = {1117--1136},
  publisher = {USENIX Association},
}