A busy network

Question: a remote read got slower while other traffic shares the link. Is the network the reason?

Run: 4 KiB random reads at queue depth 1, first alone, then beside a bulk flow from the target to the host, the direction of the read data: ib_write_bw with 4 queue pairs and 1 MiB messages for NVMe/RDMA (391.8 Gb/s, the link saturated), and iperf3 with 8 streams for NVMe/TCP (24.4 Gb/s, one iperf3 process).

TCP, alone

86.5 µs

TCP, busy link

87.8 µs

RDMA, alone

72.8 µs

RDMA, busy link

81.5 µs

Host block layerHost driverNetworkTarget driverStorage

Mean stage times of the sampled commands; the network highlighted.

Over NVMe/RDMA, with the link saturated:

== read 4KiB nvme6n1: baseline 3057 commands sampled, loaded 2673
  stage                                    base us   load us     LAD
  host block layer                            0.50      0.48     n/a
  host driver (nvme-rdma)                     1.24      1.20    0.97
  network (RDMA, NICs, wire)                 10.01     18.91    1.89
  target driver (nvmet-rdma)                  1.02      1.02    1.00
  storage (target block layer, drive)        59.98     59.92    1.00
  end to end                                 72.75     81.53    1.12
  largest LAD: network (RDMA, NICs, wire), 1.89

(n/a: a stage under 0.5 µs at the baseline is too small for a ratio.)

Answer: over NVMe/RDMA, yes, and only the network. The network stage grew 1.9× (10.01 → 18.91 µs), and within it the read's data transfer (its RDMA WRITE and response, from posting to the send completion, in --detail) doubled, 7.1 → 15.3 µs; storage did not move (LAD 1.00).

Over NVMe/TCP:

== read 4KiB nvme6n1: baseline 2647 commands sampled, loaded 2586
  stage                                    base us   load us     LAD
  host block layer                            0.52      0.53    1.02
  host driver (nvme-tcp)                      3.83      4.17    1.09
  network (TCP/IP, NICs, wire)               19.79     20.39    1.03
  target driver (nvmet-tcp)                   2.47      2.62    1.06
  storage (target block layer, drive)        59.94     60.11    1.00
  end to end                                 86.55     87.83    1.01
  largest LAD: host driver (nvme-tcp), 1.09

Here the 24.4 Gb/s flow left the NVMe/TCP read almost unchanged: the network stage went from 19.79 to 20.39 µs (LAD 1.03), the end to end from 86.55 to 87.83 µs, and no stage grew by more than 9 %. An earlier run on the same servers, with the same iperf3 flow, doubled the network stage (19.42 → 38.82 µs). The likely reason, not a proven one: a TCP flow slows NVMe/TCP only when their receive processing shares CPUs, and the NIC's receive hash picks those CPUs per connection, so two runs on different connections can differ. Both times ntprof's report agreed with fio's own latency (here 88.5 → 89.1 µs).