Skip to content

Bound a subscription and see what it sheds

What happens by default

Every subscription made through Client.Subscribe or Client.QueueSubscribe gets a bounded queue: 4,096 messages or 32 MiB, whichever comes first.

The underlying library's default is 500,000 messages. That is not a bound in any useful sense — it is a promise to exhaust memory at a moment nobody chose, and long after the traffic that caused it has stopped being explicable.

gonats.ClientSettings{
    PendingMsgs:  16384,
    PendingBytes: 128 << 20,
}

What happens when one fills

NATS discards from a full subscription queue. It does not delay the publisher.

That is the right behaviour and it is worth being clear about why: a publisher held up by a slow subscriber stops draining whatever it is reading from, and the problem moves one layer up where it is harder to see. Shedding keeps the failure local to the consumer that caused it.

The cost is that shedding is invisible unless something counts it.

Seeing it

cli, err := client.Connect(ctx, settings, cred,
    client.InProcess(srv),
    client.WithLogger(log),
    client.OnShed(func(subject string, dropped int) {
        metrics.Shed.WithLabelValues(subject).Set(float64(dropped))
    }),
)

OnShed fires with the subject and the running count for that subscription. The count belongs to the subscriber that fell behind, because "something shed" answers a much less useful question than "which consumer is behind".

WithLogger gives you the same event as a warning, if a log line is enough.

A drop nobody counts looks exactly like a quiet system

This is the failure mode the whole feature exists for. A service that is shedding one message in ten and a service with nothing to do produce identical output: none. Wire OnShed to something an operator can see, or accept that you will not find out.

Why the handler form is required

Client.Subscribe takes a callback rather than returning a channel, and that is not a style preference.

SetPendingLimits returns ErrTypeSubscription for a channel subscription. The bound and the Dropped() count are only available on the handler form, so a channel-based API here would be one that cannot keep the promise this module exists to make.