Queue depth

Question: latency grows when the queue depth goes up. Is the SSD saturating, or something on the way?

Run: 4 KiB random reads from one fio job, at queue depth 1 and at 32, over NVMe/TCP and NVMe/RDMA.

TCP, qd 1

86.7 µs

TCP, qd 32

275.4 µs

RDMA, qd 1

72.8 µs

RDMA, qd 32

112.4 µs

Host block layerHost driverNetworkTarget driverStorage

Mean stage times of the sampled commands; the network highlighted.

Over NVMe/TCP:

== read 4KiB nvme6n1: baseline 2694 commands sampled, loaded 2299
  stage                                    base us   load us     LAD
  host block layer                            0.53      0.60    1.14
  host driver (nvme-tcp)                      3.82      4.57    1.20
  network (TCP/IP, NICs, wire)               19.68    186.33    9.47
  target driver (nvmet-tcp)                   2.54     16.13    6.34
  storage (target block layer, drive)        60.08     67.77    1.13
  end to end                                 86.65    275.40    3.18
  largest LAD: network (TCP/IP, NICs, wire), 9.47

Over NVMe/RDMA:

== read 4KiB nvme6n1: baseline 3066 commands sampled, loaded 5901
  stage                                    base us   load us     LAD
  host block layer                            0.49      0.39     n/a
  host driver (nvme-rdma)                     1.26      1.08    0.86
  network (RDMA, NICs, wire)                 10.03     41.38    4.13
  target driver (nvmet-rdma)                  1.02      1.25    1.22
  storage (target block layer, drive)        59.97     68.32    1.14
  end to end                                 72.77    112.42    1.54
  largest LAD: network (RDMA, NICs, wire), 4.13

(n/a: a stage under 0.5 µs at the baseline is too small for a ratio.)

Answer: not the SSD. Over NVMe/TCP the read gets 3.2× slower, and the stage that amplifies it is the network (9.5×), followed by the target driver (6.3×); storage grows only 1.1×. At queue depth 32 the network is 68 % of the latency: one fio job submits every command on one CPU, so they share one NVMe/TCP queue and one TCP connection, which carried about 94,000 I/Os per second (four jobs: 331,000). Spreading the load over more CPUs and queues is the thing to try; a faster SSD would not help. The NSDI '25 paper found the same shape on its testbed (the network 92 % of the latency at queue depth 32).

Over NVMe/RDMA the same ladder costs much less: 72.8 → 112.4 µs at 255,600 I/Os per second, with the network the largest LAD again (4.1×, most of it the RDMA data transfer waiting for the target to handle its send completions) and storage 1.1×.

The avg QD column of each report shows where the 32 commands wait. ntprof counted 27.5 of them in the kernel over NVMe/TCP; the rest had completed and waited in io_uring's completion queue for the fio thread, which is why fio's mean latency (339 µs) is above ntprof's end to end (275 µs).