1. Build
ntprof is one binary with its eBPF programs built in. You build it on one machine and copy it to the other: at run time it needs only libelf and zlib. We build on the target, the machine that has clang.
- You need: git, make, clang 12 or later, and the libelf and zlib development files
(
sudo apt install git make clang libelf-dev zlib1g-devon Ubuntu). No root is needed to build. - You do: clone, build and test ntprof, copy it to the host. Parts 4 and 5 also need SPDK built with ntprof's probes: the last section.
- Time: about 10 minutes; SPDK takes about 15 more.
1. Clone
--recurse-submodules also fetches libbpf, the library ntprof's loader is built on, into libbpf/.
cd ~
git clone --recurse-submodules https://github.com/netlab-wisconsin/ntprof.git
cd ntprof && git log -1 --format='%h %s'
Cloning into 'ntprof'...
Submodule 'libbpf' (https://github.com/libbpf/libbpf.git) registered for path 'libbpf'
Cloning into '~/ntprof/libbpf'...
Submodule path 'libbpf': checked out '09b9e83102eb8ab9e540d36b4559c55f3bcdb95d'
3c4a27a spdk: the patch builds SPDK's unit tests
The libbpf submodule is checked out. Your last commit may be newer than ours.
2. Make sure bpftool is v7 or later
The build turns each eBPF program into a C header with bpftool gen skeleton, which needs bpftool v7 or later.
bpftool version
WARNING: bpftool not found for kernel 5.15.134+release+2.10.0r8
You may need to install the following packages for this specific kernel:
linux-tools-5.15.134+release+2.10.0r8-amd64
…
On Ubuntu, bpftool is a small wrapper that looks for the package built for the running kernel, and there is none
for a vendor kernel like ours. The real binaries are under /usr/lib/linux-tools-*:
/usr/lib/linux-tools-5.15.0-198/bpftool version
/usr/lib/linux-tools-5.15.0-198/bpftool v5.15.216
features:
That one is v5.15, too old: ntprof's build would stop with Error finalizing .BTF.
Skip to step 3 and leave out BPFTOOL=… there.
Otherwise build bpftool v7.4.0 in your home directory. Nothing is installed.
cd ~
git clone --recurse-submodules --depth 1 --branch v7.4.0 \
https://github.com/libbpf/bpftool.git
ls /usr/lib/llvm-*/bin/llvm-strip
PATH=/usr/lib/llvm-20/bin:$PATH make -C bpftool/src -j8
bpftool/src/bpftool version
| Part | What it does |
|---|---|
--depth 1 --branch v7.4.0 | fetches only the v7.4.0 release, not the whole history |
ls /usr/lib/llvm-*/bin/llvm-strip | finds the LLVM versions installed; bpftool's build calls llvm-strip by that name, and Ubuntu installs it only under these directories |
PATH=/usr/lib/llvm-20/bin:$PATH | puts one of them first in PATH for this one command |
make -C bpftool/src -j8 | builds in bpftool/src, eight jobs at a time |
Cloning into 'bpftool'...
…
/usr/lib/llvm-14/bin/llvm-strip
/usr/lib/llvm-20/bin/llvm-strip
…
LINK bpftool
make: Leaving directory '~/bpftool/src'
bpftool v7.4.0
using libbpf v1.4
features: llvm, skeletons
bpftool v7.4.0 with features: llvm, skeletons. Without the PATH part, make stops with
llvm-strip: No such file or directory.
3. Build ntprof
cd ~/ntprof
make BPFTOOL=$HOME/bpftool/src/bpftool -j8
BPFTOOL= names the bpftool to use; leave it out if your own is v7 or later. The build compiles libbpf, then each
eBPF program (BPF lines), turns each into a skeleton header (GEN-SKEL), and links one binary.
LIB ~/ntprof/build/libbpf.a
…
BPF build/nvme_tcp_host.bpf.o
BPF build/nvme_tcp_target.bpf.o
BPF build/nvme_rdma_host.bpf.o
BPF build/nvme_rdma_target.bpf.o
BPF build/spdk_tcp_host.bpf.o
BPF build/spdk_tcp_target.bpf.o
BPF build/spdk_rdma_host.bpf.o
BPF build/spdk_rdma_target.bpf.o
…
GEN-SKEL build/nvme_tcp_target.skel.h
…
CC build/loader.o
LINK ntprof
…
Eight BPF lines (host and target, for the kernel's NVMe/TCP and NVMe/RDMA drivers and for SPDK's), and LINK ntprof near the end. With -j8 the lines can come in another order.
4. Run the unit tests
The unit tests check the stage model, the join of host and target records, the statistics and the record files. They need no root and load no eBPF.
make test
…
== build/test_model
test_names_fit ok
…
219 checks, 0 failed
== build/test_join
…
39 checks, 0 failed
== build/test_stats
…
27 checks, 0 failed
== build/test_trace_file
ntprof: error: /tmp/ntp_test_nXSynp: written by another ntprof version (file v4)
ntprof: error: /tmp/ntp_test_jaE8k4: written by another ntprof version (file v6)
…
40 checks, 0 failed
Four checks, 0 failed lines. The two error lines belong to the test: it checks that record files of other
versions are refused.
5. Run it
./ntprof --version
./ntprof --help | head -n 20
ntprof 2.0.0 (3c4a27a)
ntprof 2.0.0: where the time of each NVMe-over-Fabrics I/O goes, stage by
stage (NVMe/TCP, NVMe/RDMA, SPDK), with eBPF on an unmodified kernel
usage: ntprof <command> [options] [files]
commands:
record profile the running kernel or SPDK (needs root)
report FILE... report recordings: the host's, and the target's
compare BASE LOAD latency amplification (LAD) of each stage between two
runs; a run is host.ntp or host.ntp,target.ntp
probe check which hooks this kernel offers
dump FILE... print the records of recordings as CSV
version print the version
…
The version line names the commit you built. Parts 2 to 5 use probe, record and report.
6. Copy it to the host
Our host has no clang, so it gets the binary. Here and in the later parts, target stands for the target's host
name or address.
mkdir -p ~/ntprof && cd ~/ntprof
scp target:ntprof/ntprof .
./ntprof --version
ldd ntprof | grep -E 'libelf|libz'
ntprof 2.0.0 (3c4a27a)
libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007f2e85f84000)
libz.so.1 => /usr/local/lib/libz.so.1 (0x00007f2e85f5f000)
The same version on both machines, and both libraries found (the addresses differ on every run). From now on every
command runs in ~/ntprof on both machines.
Build SPDK (for parts 4 and 5)
Skip this until you want parts 4 and 5. SPDK's I/O never passes the kernel's NVMe drivers, so ntprof reads SPDK
through probes that a small patch in the ntprof repository adds to it (why a patch, when the kernel needs
none). Build SPDK
on the target, next to ~/ntprof.
SPDK's own script installs the packages its build needs; run it once from the SPDK directory after cloning:
sudo scripts/pkgdep.sh --rdma.
Clone SPDK v26.05, the release the patch is made for:
cd ~
git clone --branch v26.05 --recurse-submodules \
https://github.com/spdk/spdk.git
cd spdk && git describe --tags
…
Submodule path 'ocf': checked out '6ad1007e6fa47ff6468eaba0b304d77334c34d4a'
Submodule path 'xnvme': checked out 'be52a634c139647b14940ba8a3ff254d6b1ca8c4'
v26.05
Apply ntprof's patch. git apply --stat first shows what it changes without changing anything:
git apply --stat ../ntprof/spdk/ntprof-usdt-v26.05.patch
git apply ../ntprof/spdk/ntprof-usdt-v26.05.patch && git status --short
include/spdk/sock.h | 7 +
include/spdk_internal/ntprof.h | 317 +++++++++++++++++++++++++++++++++++
include/spdk_internal/sock_module.h | 6 +
lib/nvme/nvme_rdma.c | 48 +++++
lib/nvme/nvme_tcp.c | 35 ++++
lib/nvmf/rdma.c | 49 +++++
lib/nvmf/tcp.c | 34 ++++
lib/sock/sock.c | 3
lib/sock/spdk_sock.map | 3
test/common/lib/test_rdma.c | 3
10 files changed, 505 insertions(+)
M include/spdk/sock.h
M include/spdk_internal/sock_module.h
M lib/nvme/nvme_rdma.c
M lib/nvme/nvme_tcp.c
M lib/nvmf/rdma.c
M lib/nvmf/tcp.c
M lib/sock/sock.c
M lib/sock/spdk_sock.map
M test/common/lib/test_rdma.c
?? include/spdk_internal/ntprof.h
The patch adds one header (ntprof.h) and a few lines to SPDK's NVMe/TCP and NVMe/RDMA host and target code, its
socket layer, and one test helper.
Configure and build:
./configure --with-usdt --with-rdma
make -j32
| Option | What it does |
|---|---|
--with-usdt | builds the probes in; without it SPDK never samples anything |
--with-rdma | adds NVMe/RDMA (part 5); NVMe/TCP (part 4) is always built |
make -j32 | 32 jobs at a time; use about as many as your CPUs |
Using default SPDK env in ~/spdk/lib/env_dpdk
Using default DPDK in ~/spdk/dpdk/build
Using 'verbs' RDMA provider
Configuring ISA-L (logfile: ~/spdk/.spdk-isal.log)...done.
Configuring ISA-L-crypto (logfile: ~/spdk/.spdk-isal-crypto.log)...done.
Creating mk/config.mk...done.
Creating mk/cc.flags.mk...done.
Type 'make' to build.
…
LINK tcp_ut
LINK subsystem_ut
LINK vhost_ut
LINK iscsi_ut
The build takes a few minutes and should end without an error; its last lines vary with -j.
SPDK's build runs Python scripts that need jinja2, pyelftools and tabulate. On our target the system's Python
packages were broken, so we ran make with a Python virtual environment holding those three first in PATH.
Check that the two programs the next parts use carry ntprof's probes:
readelf -n build/bin/nvmf_tgt | grep -A1 'Provider: ntprof' | grep Name
readelf -n build/bin/spdk_nvme_perf | grep -A1 'Provider: ntprof' | grep Name
Name: nvmf_tcp_done
Name: nvmf_tcp_done
Name: nvmf_tcp_done
Name: nvmf_rdma_done
Name: nvme_tcp_done
Name: nvme_rdma_done
Name: nvme_tcp_done
Name: nvme_rdma_done
Six probe sites in nvmf_tgt, the SPDK target (it also contains SPDK's host code, hence nvme_tcp_done and
nvme_rdma_done), and two in spdk_nvme_perf, the workload that runs on the host.
spdk_nvme_perf runs on the host. Ours has no build tools, so it gets a copy; you can also build SPDK there the
same way.
mkdir -p ~/spdk/build/bin
scp target:spdk/build/bin/spdk_nvme_perf ~/spdk/build/bin/
Next
2. Linux NVMe/TCP: profile the kernel's NVMe/TCP between the two machines.