DynamoDB Streams
use cases where DynamoDB actually wins, how Streams turns table writes into an event feed, the stuck-shard failure mode nobody configures for, and when to reach for SQL instead.
- DATE:
- AUG.27.2026
- READ:
- 16 MIN
DynamoDB gets reached for by default on serverless AWS stacks, and about half the time that’s a reflex, not a decision. This is the case for when it’s actually the right call, what Streams gives you almost for free once you’re there, the specific way Streams processing breaks in production, and where a relational engine genuinely wins instead.
The access pattern DynamoDB is actually good at
Forget “NoSQL vs SQL” as a philosophy question — it’s a distraction. The real question is what your access pattern looks like.
DynamoDB is built for: get/put a single item by a known key, high write concurrency across many independent keys, no cross-item joins, predictable single-digit-millisecond latency regardless of table size. If that’s your shape, DynamoDB’s constraints (no ad-hoc queries, access patterns designed up front via keys and GSIs) aren’t a limitation — they’re the thing forcing you into a design that scales.
Where it stops being a good fit: anything that wants “show me all X where Y joins Z,” ad-hoc reporting, or multi-entity transactions beyond a handful of items (TransactWriteItems caps out at 100 items / 4MB per transaction).
Four use cases that actually fit
1. Session / live-state store — a status field, a counter, a TTL-able flag, keyed by a single ID. Classic key-value, and DynamoDB’s per-item latency doesn’t degrade as the table grows into the billions of rows.
2. Event-sourced audit trail — every state transition in a workflow needs to be reconstructable later, but you never query it live during the workflow itself. DynamoDB holds current state; something downstream turns each change into an immutable record. This is the pattern the rest of this post is about.
3. Idempotency / dedupe table — a request ID as the partition key, a conditional write (attribute_not_exists(pk)) as the guard. If two Lambda invocations race to process the same event, only one write succeeds; the other gets a ConditionalCheckFailedException and backs off. Cheap, and it doesn’t need a lock service.
4. Config / feature-flag cache — a short-TTL, read-through cache for values owned by another system. DynamoDB serves the reads; the table itself isn’t the source of truth, just a fast local mirror kept fresh by whatever emits change events.
DynamoDB Streams: what you get once you’re there
Enable Streams on a table (NEW_AND_OLD_IMAGES is almost always the right view type) and every write becomes an ordered change-event you can consume — without standing up a CDC pipeline, a message broker, or an outbox table. This is the single biggest practical advantage DynamoDB has over a relational engine for event-driven work: change data capture is a checkbox, not a project.
from aws_cdk import aws_dynamodb as dynamodb
table = dynamodb.Table(
self, "WorkflowState",
partition_key=dynamodb.Attribute(
name="itemId", type=dynamodb.AttributeType.STRING
),
billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
stream=dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
point_in_time_recovery=True,
)A few properties of Streams worth internalizing before you build on it:
- Ordering is guaranteed per partition key, not across the table. Two writes to the same item arrive in order; writes to different items make no ordering promise relative to each other.
- Retention is 24 hours, fixed. A consumer that’s down for a day loses everything before it, permanently — there’s no re-fetch.
- A stream is organized into shards, and shards split/merge as the table’s throughput changes. A Lambda Event Source Mapping (ESM) polls each shard independently.
Architecture: table → Streams → immutable audit store
Each shard is polled independently — that’s the part worth seeing rather than reading as a bullet point:
Why route through Streams instead of writing to both DynamoDB and S3 directly from the workflow code: it decouples “did the state transition succeed” from “did the audit write succeed.” A transient S3 hiccup shouldn’t block the workflow, but you also can’t let an audit event silently vanish — Streams plus a properly configured ESM (below) is what lets you get both.
The failure mode nobody configures for: the stuck shard
Here’s the part that bites people in production. The ordering guarantee above is a double-edged sword: the ESM retries a failing batch on a shard until it succeeds or the record ages out of the 24-hour retention window — and while it retries, every newer record on that same shard is stuck behind it.
One malformed event, one downstream dependency having a bad five minutes, one unhandled exception in the consumer — and an entire shard’s worth of newer audit events queue up behind the poison record instead of processing. Left at defaults, this is silent: no records are lost, they’re just stalled, and nothing pages you until someone notices the audit trail is hours behind.
The fix is three ESM settings, none of which are on by default — here’s the same poison record under default behavior versus with them configured:
from aws_cdk import aws_lambda_event_sources as event_sources
from aws_cdk import aws_sqs as sqs
dlq = sqs.Queue(self, "AuditConsumerDlq")
audit_consumer.add_event_source(
event_sources.DynamoEventSource(
table,
starting_position=lambda_.StartingPosition.LATEST,
batch_size=100,
max_batching_window=Duration.seconds(5),
retry_attempts=3,
bisect_batch_on_error=True,
on_failure=event_sources.SqsDlq(dlq),
parallelization_factor=2,
)
)retry_attemptscaps retries instead of retrying until the record falls out of the 24-hour window — bounding how long a shard can be stuck.bisect_batch_on_errorsplits a failing batch in half on repeated failure and retries the halves independently, isolating the one poison record instead of blocking the whole batch (and thus the shard) behind it.on_failure(SQS or EventBridge) ships the exhausted batch’s metadata to a dead-letter queue once retries run out, so processing moves on to newer records instead of stalling indefinitely. Alert on this queue’s depth — a non-empty audit DLQ means gaps in your record of what happened.parallelization_factorlets multiple ESM instances read the same shard concurrently (up to 10), useful when per-item processing is the bottleneck rather than the read itself — though note it doesn’t help the stuck-shard problem on its own; a batch that keeps failing still blocks its own lane.
If 24-hour retention or single-consumer fan-out is the actual constraint — not just retry handling — DynamoDB can also export its change feed to a Kinesis Data Stream instead of (or alongside) native Streams. That buys up to 365-day retention and multiple independent consumers reading the same feed, at the cost of Kinesis’s own shard-hour pricing and a bit more operational surface. Reach for this when you need replay window or fan-out; reach for the ESM settings above when the actual problem is retry/poison-record handling, which is the far more common case.
When SQL genuinely wins instead
None of the above is an argument that DynamoDB is always right. If your workload actually needs joins, ad-hoc queries, or multi-entity transactions, forcing that into single-table DynamoDB design is its own kind of pain.
+--------------------+--------------------+--------------------+ | | DynamoDB | RDS / Aurora | | | | (Postgres) | +--------------------+--------------------+--------------------+ | Access pattern | Key-value, | Ad-hoc joins, | | | designed up front | relational queries | +--------------------+--------------------+--------------------+ | Change feed / CDC | Streams — built | Needs DMS / | | | in, one flag | Debezium / outbox | | | | table | +--------------------+--------------------+--------------------+ | Lambda connections | HTTP calls, no | Needs RDS Proxy | | | pool needed | under concurrency | +--------------------+--------------------+--------------------+ | Scaling | Horizontal, | Vertical + read | | | effectively | replicas | | | unbounded | | +--------------------+--------------------+--------------------+ | Idle cost | On-demand: pay per | Aurora Serverless | | | request, scales to | v2 floor: billed | | | zero | 24/7 even idle | +--------------------+--------------------+--------------------+ | Transactions | Up to 100 items / | Full ACID across | | | 4MB | arbitrary tables | +--------------------+--------------------+--------------------+
The connection-pool line matters more than it looks. DynamoDB is called over HTTP — a burst of concurrent Lambda invocations doesn’t exhaust anything on the DynamoDB side. A relational engine has a hard connection ceiling; a spike in concurrent Lambdas each opening their own connection is a classic way to take down Aurora, and the standard fix (RDS Proxy) is an extra piece of infrastructure you now own and pay for.
The cost line matters for anything with genuinely spiky or low-baseline traffic: on-demand DynamoDB scales down to near-zero cost between bursts, while even Aurora Serverless v2 has a minimum ACU floor billed continuously — you’re paying a baseline at 2am with zero traffic, every night, whether you use it or not.
Rule of thumb: if you can name the exact keys you’ll ever look this data up by, and you don’t need to join it against something else, that’s DynamoDB. The moment you catch yourself designing a GSI just to fake a join a SQL engine would do natively in one query — that’s the signal you picked the wrong tool, not that you need a cleverer GSI.
Streams turns a table into an event log for free — but “for free” doesn’t mean “no configuration required.” The ordering guarantee that makes it useful is the same mechanism that lets one bad record silently stall everything behind it. Configure for that on day one, not after the first incident.