Skip to main content

How it works

ntprof turns timestamps taken on two machines into stage times, without changing the kernel and without synchronizing clocks. Four pieces do it.

1. Hooks on what the kernel already has​

ntprof attaches eBPF programs to tracepoints and to functions the compiler kept out of line, on both sides:

  • Block layer and NVMe core: the nvme_setup_cmd, nvme_complete_rq, block_rq_issue, block_rq_complete, nvmet_req_init and nvmet_req_complete tracepoints.
  • NVMe/TCP: the host's send functions and nvme_tcp_recv_skb, with the receive time of each packet; the target's queue-response and socket-send paths. Where a kernel build inlines the host's send functions, ntprof watches their caller and the socket sends it makes instead.
  • NVMe/RDMA: posting and completion handlers on both sides (nvme_rdma_post_send, nvme_rdma_recv_done, nvmet_rdma_recv_done, nvmet_rdma_read_data_done, nvmet_rdma_queue_response, nvmet_rdma_send_done).
  • SPDK: a few USDT probes added by a small patch; SPDK keeps the stage times itself and reports them once per command and side.

Every function hook exists as fentry/fexit and as kprobe/kretprobe. fentry costs less per call and is preferred; kprobes are used where a module has no BTF of its own, or where a busy CPU would make attaching fentry wait.

2. The same commands on both sides​

A command is sampled when a hash of its command ID, starting block and length falls in one bucket out of N. The host and the target compute the same hash, so they sample the same commands without sending anything to each other, and the wire format stays untouched.

3. The join​

Each side writes one record per sampled command when the command leaves that side. ntprof matches a host record with the target record of the same controller, queue and command ID, and checks the opcode, namespace, block address and length. Only differences between timestamps of one machine are ever taken, so the two machines' clocks need no synchronization.

4. The stage model​

For each round trip, the host waits from handing its message to the network until the answer reaches its network stack; the target is busy from the message's arrival until it hands its answer back. The host's wait minus the target's residence is the time in the wire, the NICs and the network-stack work that neither driver sees. With the two receive paths (from a packet's arrival to the driver handling it), it forms the network stage. The other components come from one side's timestamps each, and all of them add up exactly to the end-to-end latency. ntprof sums them into five stages: host block layer, host driver, network, target driver and storage.

Each stage's occupancy (avg QD) is its mean time times the command rate, by Little's law. With --exact-queues, ntprof instead counts the commands in each queue on every I/O, which costs map operations on every command.

Accuracy and its limits​

  • Validated on real hardware: on two servers over 400 GbE RoCE, every sampled command joined across host and target; the storage stage matched the drive measured alone on the target (4 KiB read: 60.0 vs 61.2 µs; 128 KiB read: 93.7 vs 93.7 µs); a busy drive appeared only in storage and a saturated link only in the network stage.
  • Sampling follows commands, not time: a run needs enough sampled commands per group for its percentiles.
  • The hooks run for every command, sampled or not: on the two servers they added 0.8 µs (NVMe/RDMA) and 1.3 µs (NVMe/TCP) to a 4 KiB read at queue depth 1 (Overhead). A sampled command does a little more work for its own timestamps, which its stage times include.
  • A kprobe can miss a command under heavy load: on an NVMe/RDMA target running kprobes, about 10 % of the sampled commands at queue depth 32 lacked the time their response was posted. ntprof closes such a record instead of letting the next command on the same request structure write into it, uses the time the command completed in the NVMe target instead (a difference under a microsecond when both are seen), and the report says how many records this concerns.
  • Over loopback the network share is close to 0 and can be slightly negative.

The paper describes the model; the repository's DESIGN.md documents every hook.