Skip to main content

NVMe/RDMA

Over NVMe/RDMA the host CPU sees one round trip per command: the command is posted, and the response's completion is polled. The data moves without it: the target pulls a write's data with RDMA READ and pushes a read's data with RDMA WRITE. ntprof hooks nvme-rdma on the host and nvmet-rdma on the target:

sudo ntprof record -r target -t rdma -o target.ntp   # target
sudo ntprof record -t rdma -o host.ntp -d 30         # host
ntprof report --detail host.ntp target.ntp

The network and storage, per request​

With --detail, the network stage shows a component of its own, data transfer (RDMA, at the target):

  • for a write whose data is not in the command, the RDMA READ: from posting it to its completion, as the target sees it;
  • for a read, the RDMA WRITE of the data and the response: from posting them to the response's send completion.

That is each request's time in the network, measured apart from its time in storage, the same split a staged pipeline would time as a NIC stage and an SSD stage. Random reads and writes at queue depth 1, on two servers over 400 GbE RoCE:

Request (queue depth 1)Data transferStorageEnd to end
4 KiB read7.1 µs60.0 µs72.8 µs
128 KiB read12.3 µs93.7 µs121.2 µs
128 KiB write15.8 µs39.0 µs76.8 µs

See Network vs storage for the experiment.

What differs from TCP​

  • The network's computed part is still the host's wait minus the target's residence; the RDMA READ of a write counts as network, not as target time.
  • A write whose data fits in the command (inline, 4 KiB by default) has no data transfer component.
  • A read's transfer includes the host NIC's acknowledgement on its way back, which the host's wait does not, so it can exceed the read's share of the network. Under load it also includes the time until the target handles its send completions.