Skip to main content

Getting started

ntprof is one binary with its eBPF programs inside. Build it once, copy it to the host and the target, and profile a workload that is already running. This page is the short version; the tutorial goes through it on two machines, one command at a time, with the output to expect.

Requirements​

  • To run: Linux 5.15 or 6.x with eBPF, root (or CAP_BPF and CAP_PERFMON), the NVMe-oF modules loaded (on the host: connected with nvme connect), and BTF: /sys/kernel/btf/vmlinux and the modules' own BTF. Modules built out of tree, such as MLNX OFED's DKMS packages, often lack it; see Kernels without BTF.
  • To build: clang 12 or later, bpftool, and the libelf and zlib development files. libbpf comes with the repository as a submodule.
Distribution bpftool

Ubuntu's /usr/sbin/bpftool refuses to run on a kernel it has no package for; use /usr/lib/linux-tools/<any version>/bpftool. The bpftool of Ubuntu 22.04 (v5.15) is too old ("Error finalizing .BTF"): build v7.4.0 or later from libbpf/bpftool and pass make BPFTOOL=/path/to/bpftool.

Build​

git clone --recurse-submodules https://github.com/netlab-wisconsin/ntprof.git
cd ntprof
make                    # builds ./ntprof
make test               # unit tests (no root, no BPF)
sudo make install       # /usr/local/sbin/ntprof (PREFIX=, DESTDIR= as usual)
ntprof --version

At run time the binary needs only libelf and zlib, so a build from one machine runs on another with the same libraries.

Check the kernel​

ntprof probe lists every hook for the role and transport you name, whether this kernel has it, and loads the programs without attaching them:

sudo ntprof probe -r both -t tcp      # host and target, NVMe/TCP
sudo ntprof probe -r host -t rdma     # the host, NVMe/RDMA

Each object line says whether its module has BTF and which hook flavour it will use (fentry/fexit, or kprobes), and the last line says load: ok. A hook the kernel's build has inlined shows as --; where ntprof has a stand-in for it, the line says so.

Try it on one machine​

With the stock nvme-tcp, nvmet-tcp and null_blk modules, fio and nvme-cli, the demo needs no SSD and no NIC:

tests/demo.sh          # about two minutes; tests/demo.sh down removes the test bed

It exports a null_blk device through the kernel target on 127.0.0.1, connects to it, and shows a live breakdown, the two sides recorded separately and joined, and the latency amplification between queue depth 1 and 32.

Profile two machines​

On the target, start recording first and stop last; on the host, record while the workload runs. Use the same sampling rate (-s) on both: both sides then pick the same commands.

# target
sudo ntprof record -r target -t rdma -o target.ntp
# host
sudo ntprof record -t rdma -o host.ntp -d 30
# anywhere, with both files
ntprof report host.ntp target.ntp

ntprof record alone, on the host, reports the network and the target together, as one row ("network and target (not recorded)"). -r both profiles a host and a target on the same machine in one command.

Sampling

ntprof records one command in 1000 by default. At low rates (queue depth 1 does about 10,000 I/Os per second) that leaves a few hundred commands a minute; use -s 100 for a short run. Every command passes the hooks either way, so the overhead depends little on the rate.

Next​