Skip to content

What happens at a full queue

Three parts, and the split between them is ownership.

The bound is ours

Every subscription is bounded, and it is not configurable.

An unbounded queue is not a policy. It is a decision to fail once memory is gone rather than at a number somebody chose, and it converts a countable discard into invisible latency followed by a process that dies for a reason nobody can attribute.

That is not hypothetical. A spike against an early, since-deleted worker-pool prototype gave its queue a generous 1024 slots and reported zero loss, while 22 of 60 messages sat in that channel, handled by nobody and counted by nothing. It looked like the best result in the table and was the worst. Bounding it made the same run tell the truth, which is why every queue in the shipped design is bounded, with no unbounded option anywhere.

The count is ours

Whatever happens at the bound is counted, and attributed to the subscription it happened to.

A discarded message is gone with no signal to whoever sent it. Classic load shedding at least lets the shedder say no — an HTTP 503 tells a client, which can retry. Here nobody is told. The count is therefore the only evidence the message ever existed, which is why it is not optional and why it is per subscription rather than aggregated.

An aggregate answers "something was dropped" when the useful question is "which consumer is behind".

The policy is the backend's

ShedNewest, ShedOldest, Fail, and Block as a fourth policy the type defines. A backend declares which of them it can honour, and a caller asking for one it cannot gets the backend's native behaviour plus a construction error naming what was dropped.

Why this list is not unified. Core NATS's behaviour is fixed: it refuses the newest and fires an async error, with no hook to evict the oldest and no way to block a publisher that is already fire-and-forget into a write buffer. So ShedNewest is all it can offer. The in-memory backend offers three, ShedNewest, ShedOldest and Fail, because a channel it owns can implement any of them; it does not currently offer Block, which parked a publisher on the queue's own lock and was the reason a bounded shutdown was hard to prove correct (0002).

An earlier version of this design put a queue in front of every backend purely so that an option could mean the same thing everywhere. That bought portability with a second queue on every subscription and a permanent drain obligation, to hide a difference the caller is better off being told about. Declaring and degrading is the honest version, and the one the rest of this estate already uses.

So swapping the backend can change what happens at a full queue, and a caller who cares must ask.

Watching it

Bus.Stats(name) answers two different questions in one call:

  • Pending: the consumer is falling behind, right now.
  • TotalLost(): it fell behind far enough to lose something, cumulatively.

The whole point of the first is to see the second coming. Both exist for every subscription, whatever its Concurrency, because the queue they describe is the backend's single queue: this module adds none of its own. On a NATS backend a subscription's pending count is a local read rather than a round trip, so watching it costs nothing.

Retiring a source while somebody is publishing

Worth naming because it is where a backend implementation is most likely to be wrong, and the conformance suite asserts it.

A source's delivery channel is closed by Retire and written to by every publish that selects it, so retiring needs to exclude the senders. Checking a "retired" flag before sending is not enough: retirement lands between the check and the send, which is a data race and then a send on a closed channel, which is a panic rather than a wrong answer.

The in-memory backend deregisters a retiring source from routing first, so no new publish can select it, then takes a write lock to drain its backlog and close its channel. A publish already past selection holds a read lock for the whole send and completes before that write lock can be acquired, so the two never interleave.

This is in the suite because an earlier version of that backend got it wrong, eighteen local runs under -race did not reproduce it, and one CI run did.

On the word "shed"

Shed is this module's term for a deliberate discard at a chosen bound, as against growing without limit or blocking. It is used as a verb — a queue sheds — and deliberately not as a noun, because "a shed" reads as an object rather than an event.