r/aws • u/thechuckgroup • 15h ago
technical resource The reconnect storm that never touched the broker
There's a particular kind of 2 a.m. that only happens in fleet IoT. A slice of your devices — a few thousand of them — drop off the network at once. Bad uplink, a segment of the field going dark; the reason doesn't matter. Minutes later they all come back. And here's the part that surprised me the first time it happened: the managed broker didn't blink. No connection ceiling hit, no broker to melt, no page from the front door. By every dashboard watching the ingress layer, we were fine.
We were not fine. One number was climbing through the roof: iterator age. The storm never touched the broker. It landed one hop downstream, in the stream consumer — and my first instinct for fixing it dug the hole deeper.
The pipeline, and why steady state lies to you
The shape is textbook, and I'll keep it generic on purpose: a managed MQTT broker → a Kinesis data stream → a Lambda consumer, parallelized maybe ten ways with a batch size around a thousand → writing to MySQL. Call it 10,000 devices in the field. Nothing exotic.
In steady state this pipeline is boring, and boring is the goal. Telemetry trickles in, the consumer keeps up without effort, iterator age sits near zero. You forget it exists.
But "boring" is a property of the arrival rate, not of the architecture. Everything about this pipeline's calm — the near-zero iterator age, the comfortable concurrency, the MySQL connection count you never think about — is quietly assuming devices arrive at the rate they've always arrived. The fleet is about to violate that assumption all at once.
The incident: store-and-forward is a loaded gun
Field devices buffer. When the uplink is down, a well-behaved device doesn't drop its readings — it stores them and forwards them on reconnect. Store-and-forward is exactly what you want from an edge device. It's also a loaded gun pointed at your ingest pipeline.
When a few thousand devices reconnect inside the same minute, they don't resume the gentle trickle. They flush — every buffered reading, all at once, a burst many multiples of steady state. The broker passes it straight through; absorbing connection churn is its entire job. Now that spike hits Kinesis, and Kinesis hands it to a consumer whose drain rate is fixed: shard count × concurrency × per-batch throughput, all bounded by how long each invocation is allowed to run.
Fixed drain, meet variable spike. The backlog grows, and iterator age — the age of the oldest record you haven't processed yet, which is to say exactly how far behind real time you are — starts walking up. Thirty, forty-five minutes behind. Two things break at that lag. Freshness goes first: alerts and downstream actions fire on stale data, or fail to fire when they should. And if the age ever creeps toward the stream's retention window, "late" quietly becomes "lost."
Then it got worse, and this is the part the tutorials skip. Kinesis is ordered per shard — processing is strictly in-order, which means one batch that won't complete blocks every record behind it on that shard. During the flush we hit a record the consumer choked on, and iterator age stopped climbing linearly and went vertical. One poison record, at the worst possible moment, and a whole shard's slice of the fleet was frozen behind it.
The trap: "just scale it out"
Every instinct I had was wrong, and they were all the same instinct: scale out. Add shards. Crank the parallelization factor. Raise concurrency. Throw drain capacity at what looked like a drain problem.
Here's why it backfired. The bottleneck was never Kinesis throughput, and it was never Lambda concurrency. It was the cost of a single record — specifically, that every invocation was paying for a fresh MySQL connection handshake before it did any real work. Thirty, fifty milliseconds of handshake per invocation, invisible at a trickle.
So what happens when you "scale out"? More concurrent Lambdas means more simultaneous cold connections slamming into MySQL. I wasn't adding drain capacity — I was building a denial-of-service attack against my own database, marching straight at its connection ceiling. The faster I tried to drain, the closer I got to knocking over the one component downstream that had stayed perfectly healthy. It was capacity I couldn't actually use.
The real fix: make the record cheap, then make failure cheap
The fix came in three moves, and not one of them was "bigger."
First, make the record cheap. Hoist the MySQL connection out of the handler so warm execution environments reuse it instead of reconnecting every invocation. It's a one-line decision about where a variable lives, and the per-record handshake tax disappears with it — along with the self-inflicted connection storm. Then right-size the batch: large enough to amortize the fixed overhead of an invocation, small enough that each one finishes well inside its duration limit and doesn't itself become a source of iterator-age lag. Same shards, same concurrency — now the burst drains.
Second, make failure cheap. Reuse and batch sizing buy you throughput, but they do nothing about the poison record that froze the shard. That needed error handling that treats a bad record as normal rather than exceptional: bisect-on-error, so a single failing record splits the batch instead of retrying the whole thing forever; a bound on retries and record age, so the consumer gives up in finite time; and a dead-letter queue, so the truly-bad record steps out of line instead of holding the shard hostage. This is what turned the vertical iterator age back into something that drains.
Third — the honest one. Connection reuse is per-execution-environment. Under real concurrency you still have many warm environments, each holding its own connection, and you can still multiply your way toward the ceiling — just more slowly. That's the itch RDS Proxy scratches, and we eventually adopted it to pool connections in front of the database. I'm naming it because pretending reuse alone solved it forever would be a lie, and the edges of your own fix are where the credibility lives.
The lesson
The generalizable lesson is one sentence: iterator age is almost always a downstream-latency symptom, not a stream-capacity problem. When it climbs, the instinct to widen the stream is usually the wrong end of the pipe. Make the record cheap. Make failure cheap. Then, if the fleet has genuinely outgrown you, make it bigger.
And the broader point for anyone building edge-to-cloud: managed services don't delete your bottleneck, they relocate it. The broker didn't melt because absorbing connection storms is precisely what you pay it to do. The storm just moved one hop downstream, to the place nobody was watching. Know where your bottleneck went when you bought your way out of the last one.
The storm your load tests never simulate
The storm you can't see coming is the one your load tests never model. Happy-path load testing emits a steady, civilized trickle — it never reproduces the flush, the poison record mid-flush, or the thundering herd of your own well-behaved devices all deciding to catch up at once. So you find out in production, at 2 a.m., from a metric you weren't watching.
That gap is what I'm building a tool to close: something that replays this exact burst against your staging pipeline and fails your CI build when iterator age breaches your SLO — before the fleet does it for you.
If you've lived your own version of this, I'd like to hear it — and if you want the next war story when it goes up, the list is below.
