Skip to main content

SPDK

SPDK runs NVMe/TCP in user space, where kernel hooks see nothing of it. ntprof reads SPDK through USDT probes that a small patch adds to SPDK's NVMe/TCP host, target and socket layer.

cd spdk && git checkout v26.05
git apply /path/to/ntprof/spdk/ntprof-usdt-v26.05.patch
./configure --with-usdt && make

Start the SPDK applications first, then ntprof, naming the binary of each side that runs in SPDK:

sudo ntprof probe -r both --spdk-host spdk/build/bin/spdk_nvme_perf --spdk-target spdk/build/bin/nvmf_tgt
sudo ntprof record -r both -s 1 --spdk-host spdk/build/bin/spdk_nvme_perf --spdk-target spdk/build/bin/nvmf_tgt
sudo ntprof record -r both --spdk-target spdk/build/bin/nvmf_tgt      # kernel host, SPDK target
  • ntprof profiles every running process that maps the named file. With SPDK's shared libraries, name the transport library first and add the socket module, for example --spdk-target build/lib/libspdk_nvmf.so,build/lib/libspdk_sock_posix.so.
  • SPDK samples nothing until ntprof, once attached, writes its sampling period into the processes, and ntprof writes 0 back when it stops. An unsampled command costs SPDK one load and one compare. For a sampled command, SPDK writes the stage times itself and one probe per side fires when the command is done, so the probe's cost stays out of the times.
  • An SPDK host has no block layer: the host block layer is not shown, and the throughput is estimated from the sampled commands.
  • The driver rows of an SPDK side read "host driver (SPDK)" and "target driver (SPDK)". With an SPDK target, storage is "storage (SPDK bdev, drive)", the bdev layer and the device, without the block-layer split.
  • tests/demo-spdk.sh runs the steps on one machine in about a minute, with ntprof's mean latency next to spdk_nvme_perf's own.

NVMe/RDMA in SPDK is not covered yet.