Sender Queueing and Burst Pacing

There is a class of latency spike that survives every receiver-side fix. You size the receive-path buffers correctly, you see zero UDP drops, zero NAK recovery, and zero flow-control back-pressure — and yet a multi-millisecond tail remains. It is not on the wire and not at the receiver. It is the message waiting in the sender’s own term buffer for the sender’s media driver to put it on the wire.

This page is the sender-side complement to the receiver-focused micro-burst model. Same trigger (an unpaced burst), different failure surface.

The mechanism

A publisher has two rates that are easy to conflate:

• Dispatch rate P — how fast the application calls offer(). An unpaced loop dispatches into the term buffer at memory speed — tens of millions of messages per second.
• Drain rate D — how fast the media driver’s single Sender thread serializes the term buffer onto the wire with sendmmsg(). This is bounded by per-packet CPU/syscall cost on one thread, not by NIC bandwidth (until messages get large — see caveats).

When P > D, the application gets far ahead of the driver and messages pile up in the term buffer. Each message’s wait there is pure sender-side latency — invisible to the receiver’s loss and back-pressure counters because nothing is lost and the flow-control window is never the gate.

Why this hides from your dashboards

The usual burst alarms — UdpRcvbufErrors, ethtool -S drop counters, Aeron loss-gap-fill, sender flow-control / back-pressure events — all stay at zero. The driver handles every message; it just cannot serialize them instantaneously. If you only watch loss and back-pressure, this tail is invisible.

The closed-form model

Treat the driver as a constant-rate server draining a queue. For a burst of N messages dispatched at rate P, the backlog peaks at the end of dispatch, and the last message waits the longest:

peak backlog        Q = N × (1 − D/P)      (0 if P ≤ D)
max sender-queue latency = Q / D = (N / D) × (1 − D/P)

The clean way to read this: a latency target T permits a backlog of at most B = D × T messages. Everything else follows.

• Unpaced (tight-loop) burst, P → ∞: the latency floor is N / D. A tight-loop burst of N meets target T only if N ≤ D × T.
• Paced burst: to hold N under T, pace the dispatch to at most

P_max = D / (1 − T × D / N)

• Sustained stream (not a finite burst): the queue is bounded only if P ≤ D. Any sustained P > D grows the backlog without limit and no target can hold it. The maximum sustainable dispatch rate is simply D.

Measuring your drain rate D

D is the one empirical constant the model needs. Measure it directly — don’t guess:

1. Run a single publisher with an unpaced burst of known size N (e.g. 90,000 messages).
2. Capture the burst on the sender’s NIC and, for each message, compute its sender-side queue time = driverSendTimestamp − applicationOfferTimestamp. (Aeron’s channel-snd-ts-offset stamps the driver send time into the payload; stamp the offer time yourself in an adjacent field.)
3. The queue time rises linearly with position in the burst and tops out at max ≈ N / D. Read the maximum and invert:

D ≈ N / (max sender-queue latency)

A representative single-thread Sender measurement, 160-byte messages, modern x86 in AWS:

Quantity	Value
Burst size N	90,000 messages
Max sender queue latency (unpaced)	≈ 55 ms (worst observed ≈ 67 ms)
Derived drain rate D	≈ 1.6 M msg/s typical; ≈ 1.3 M msg/s conservative
Equivalent wire rate	≈ 3 Gbps instantaneous (well under the NIC’s ~10+ Gbps)

Use the conservative D for a hard SLA bound — it leaves margin for GC pauses and scheduler jitter. The typical figure is what you see on a quiet, pinned core.

Pacing against a target

Plugging the typical D ≈ 1.6 M msg/s and N = 90,000:

Target T	Max instant (tight-loop) burst D·T	Pace N=90k burst at P_max
0.5 ms	≈ 800 msgs	≈ 1.61 M msg/s
1 ms	≈ 1,600 msgs	≈ 1.63 M msg/s
5 ms	≈ 8,000 msgs	≈ 1.76 M msg/s
10 ms	≈ 16,000 msgs	≈ 1.95 M msg/s
25 ms	≈ 40,000 msgs	≈ 2.88 M msg/s

Two ways to read the table:

• If your burst is small enough (N ≤ D·T), a tight loop already meets the target — no pacing needed. At T = 1 ms that ceiling is only ~1,600 messages, which is why large bursts always need pacing.
• If your burst is larger, pace dispatch to P_max. P_max can exceed D because the backlog only builds during the finite injection window; the binding constraint is the backlog at injection-end, D·T. As T approaches the unpaced floor N/D (~56 ms here), P_max → ∞ — pacing buys nothing because the tight loop already meets that loose target.

Practical guidance

• Pace bursts at the source. The cheapest fix is to spread the burst so dispatch never outruns the driver. Most burst-style publishers expose a rate knob; set it from P_max above.
• This is silicon-blind. D is set by the single Sender thread’s per-message cost, so the same publisher shows the same sender-queue tail on different CPU vendors. Don’t chase it as a hardware problem.
• It compounds with the receiver story. The same unpaced burst that builds a sender-side queue also delivers the steepest micro-burst at the receiver. Pacing the sender relieves both ends at once.
• The idle strategy sets D. A busy-spin Sender drains faster (higher D) than a sleeping one; see Smart Batching and Idle Strategy. Don’t add application-level batching to “help” — the driver already coalesces via sendmmsg(), and batching only adds wait time.

Caveats

• D is measured per publisher at a fixed payload size. It is packets-per-second / CPU-bound on one thread, so D in msg/s is roughly payload-independent — until messages grow large enough that NIC bandwidth becomes the gate, at which point D_eff = min(D_pps, bandwidth / message_size).
• The model assumes the driver Sender is continuously busy during the burst (true for any P > D). For P ≤ D there is no backlog and sender-queue latency collapses to the steady floor.

Summary

	Receiver micro-burst	Sender queueing
Where the time goes	OS socket / RX ring overflow → NAK recovery	Term buffer wait for the driver Sender
Smoking gun	UdpRcvbufErrors, NAK spikes climb	All loss/back-pressure counters stay zero
Bound by	SO_RCVBUF, rmem_max, ENA ring, window	Single Sender thread drain rate D
Fix	Size the receive-path buffers	Pace dispatch to P_max = D / (1 − T·D/N)

For how the media driver’s Sender thread coalesces and serializes sends internally, see The Aeron Files. This page stays on the operational model and how to pace against a latency budget.