Comparing two runs

A workload slows down. Is it the SSD, the fabric, or the software in between? Record a baseline and a run under the condition you suspect (higher load, an interfering job), then:

ntprof compare base/host.ntp,base/target.ntp loaded/host.ntp,loaded/target.ntp

A run is host.ntp alone, or host.ntp,target.ntp. For each group present in both runs, compare prints each stage's mean in the baseline and under load and its latency amplification degree (LAD): the loaded mean divided by the baseline mean.

== read 4KiB nvme6n1: baseline 2694 commands sampled, loaded 2299
  stage                                    base us   load us     LAD
  host block layer                            0.53      0.60    1.14
  host driver (nvme-tcp)                      3.82      4.57    1.20
  network (TCP/IP, NICs, wire)               19.68    186.33    9.47
  target driver (nvmet-tcp)                   2.54     16.13    6.34
  storage (target block layer, drive)        60.08     67.77    1.13
  end to end                                 86.65    275.40    3.18
  largest LAD: network (TCP/IP, NICs, wire), 9.47

This is a 4 KiB random read over NVMe/TCP at queue depth 1 and 32, one fio job: the latency grows 3.2×, and the network is the stage that amplifies it (9.5×), not storage (1.1×). See Queue depth. A stage whose mean is under 0.5 µs at the baseline gets n/a instead of a LAD: it is too small for a ratio.

The stage with the largest LAD is the one to look at first. A stage whose LAD stays near 1 is not the bottleneck, however large its share.