Debezium Scaling Lessons from a Production CDC Architecture
Production CDC architecture breaks under load long before most teams expect it. With Debezium, Kafka Connect, and Postgres, the failure patterns are consistent: WAL pressure builds up, connector lag drifts unnoticed, and snapshot phases exhaust memory under bursty traffic.
This is based on running these pipelines across high throughput systems, including workloads above 10k TPS. The difference between a system that works and one that holds under pressure comes down to observability, WAL discipline, and how connector scaling is handled.
Production Debezium CDC Architecture
Operational reality vs. tutorial defaults under real load (10k+ TPS)
Assumes low throughput and stable networks. Fails under pressure.
- Shared slot coupling multiple connectors
- Default WAL retention settings
- Risk: Slot conflicts and WAL buildup during failover
- One connector for every table
- Standard JVM Heap (often -Xmx1G)
- snapshot.mode: initial on huge tables
- Failure: JVM OOM during snapshot phase
- Low partition count
- No internal metric visibility
- Failure: Silent Lag Accumulation
Forced Full Resnapshots & Extended Outages
Prioritizes observability, isolation, and stability over guarantees.
- Strict 1 slot per connector isolation
- Tuning max_wal_senders & slots
- Monitoring pg_replication_slots
- Benefit: Isolated failure blast radius
- Tables split across connectors, one slot each
- Baseline -Xmx4G + G1GC tuning
- Tuning poll.interval & max.batch.size
- Benefit: Stable 10k+ TPS throughput
- JMX metrics scraped by Prometheus
- Alerting on MilliSecondsBehindSource and slot lag
- Schema changes tested before they reach production
- Benefit: Incident detection in under 30 mins
Sustained High Throughput & Rapid Recovery
Scaling Kafka Connect Under Real Load with Debezium
Production CDC architecture is the discipline of reliably capturing database changes at scale and streaming them into systems like Kafka without data loss, lag accumulation, or operational instability. In practice, this means managing WAL pressure, connector backlogs, schema drift, and downstream backpressure under sustained load.
I have built and operated Debezium CDC pipelines from Postgres to Kafka across more than a dozen production systems, including environments exceeding 10k TPS across 100+ tables. Under these conditions, default configurations fail. WAL backlogs grow silently, snapshots exhaust heap memory, and connector lag drifts for hours before detection.
In multiple production systems, introducing proper observability and WAL tuning reduced incident detection time from hours to minutes and eliminated repeated resnapshot cycles during peak load. What follows are the configurations, metrics, and decisions that consistently hold up in production.
Bottom Line Up Front
Monitor key JMX metrics such as MilliSecondsBehindSource and snapshot status from the first deployment. In several production systems, this alone detected more than 80 percent of incidents before they escalated. Without this baseline, teams end up diagnosing offset drift and WAL issues under pressure instead of preventing them.
Kafka Connect CDC Production Failure Modes Under Load
Tutorials make Debezium appear simple, but Kafka Connect CDC production environments expose real constraints. Multi-broker Kafka clusters, replication slot pressure, and downstream backpressure quickly surface weaknesses in default setups.
In a 12k TPS workload across 120 tables, default polling intervals caused lag to grow into hours. Snapshot phases triggered memory exhaustion in Connect workers, leading to repeated restarts and partial data loss scenarios. These failures appear consistently under bursty load.
The Debezium documentation does explain WAL disk usage and heartbeats, but it cannot tell you how your own write volume and multi-table snapshots will behave in your JVM. Without production-grade monitoring, teams rely on logs and manual correlation between Postgres replication state and Kafka lag, which is too slow for real incidents.
Production CDC Architecture Observability
Observability is the foundation of production CDC architecture. Kafka Connect and Debezium expose critical metrics through JMX that must be scraped and monitored continuously.
MilliSecondsBehindSource(streaming metrics) tracks how far the connector is behind the databaseQueueRemainingCapacityshows when the internal queue is full and the connector is saturated- Slot lag from
pg_replication_slotsshows how much WAL Postgres is holding for the connector snapshot statusdetects stalled or long running snapshotsrecords-per-sechighlights throughput degradation- Per-topic message rates on the broker side identify hot tables and uneven load
In one deployment, alerting on lag exceeding one hour surfaced a replication backlog several hours before it caused downstream outages. This reduced mean time to recovery from multiple hours to under 30 minutes.
Postgres must be monitored alongside Kafka. Queries against pg_replication_slots reveal slot pressure early. Once slot usage approaches critical thresholds, intervention is required immediately to avoid forced resnapshots.
Debezium CDC Architecture Trade-Offs in Production
Across multiple production deployments, the primary optimization target is operational stability rather than theoretical correctness. Systems that prioritize perfect guarantees over throughput and recovery tend to fail under load.
| Approach | Advantages | Trade-offs | When to Use |
|---|---|---|---|
| Log-based CDC (pgoutput) | Low database overhead, native schema evolution | Requires WAL slot management and tuning | Default for Postgres production pipelines |
| Snapshot plus streaming | Full recovery capability | High CPU and memory usage during snapshot | Systems needing rebuild after outages |
| At-least-once delivery | High throughput, simpler operations | Duplicate handling required downstream | Analytics, IoT, event streaming above 1k TPS |
| Exactly-once semantics | No duplicates, strong guarantees | 20 to 50 percent latency overhead | Strict regulatory systems only |
In one production system, enabling exactly-once semantics increased latency from 400 milliseconds to over 900 milliseconds and caused SLA violations. In contrast, downstream deduplication was simpler and more reliable.
A critical rule from production experience is to never share WAL slots across connectors. A replication slot can only have one active consumer, so two connectors pointed at the same slot fight over it. Losing or recreating a slot during failover forces a full resnapshot and long recovery.
Running a Production Postgres CDC Connector Spec
For postgres cdc architecture, I deploy Kafka Connect as a StatefulSet with a hardened Postgres connector spec. This is the baseline I use before tuning for table count, write volume, and failover behavior. It reflects what has held stable across high throughput systems rather than default examples:
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: postgres-cdc
labels:
strimzi.io/cluster: my-connect-cluster
spec:
class: io.debezium.connector.postgresql.PostgresConnector
tasksMax: 1 # the Postgres connector always runs a single task
config:
database.hostname: prod-postgres.example.com
database.port: 5432
database.user: replicator
database.password: ${secrets:cdc/postgres-credentials:password}
database.dbname: inventory
topic.prefix: dbserver1
table.include.list: public.inventory.*,public.customers
slot.name: debezium_slot
plugin.name: pgoutput # Postgres 10+
publication.name: dbz_publication
snapshot.mode: initial
heartbeat.interval.ms: 30000
max.batch.size: 2048
max.queue.size: 8192
max.queue.size.in.bytes: 0
poll.interval.ms: 500This uses the Strimzi KafkaConnector resource and the Strimzi Kubernetes secrets config provider for the password. Apply with kubectl apply -f connector.yaml. The Postgres connector always runs one task, so raising tasksMax does nothing. To scale, split tables across several connectors, each with its own slot and publication, then validate under synthetic load.
Connect worker configuration needs to match the same production assumptions:
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
offset.storage.topic=connect-offsets
config.storage.topic=connect-configs
status.storage.topic=connect-status
offset.storage.replication.factor=3
config.storage.replication.factor=3
# set in the worker environment, not in this file
# KAFKA_HEAP_OPTS="-Xms1G -Xmx4G"
# KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseG1GC"Metrics must be exposed from the start. Port 8083 is the Connect REST API and does not serve Prometheus metrics, so run the Prometheus JMX exporter agent on the workers and scrape its port:
- job_name: 'kafka-connect'
static_configs:
- targets: ['connect:9404'] # JMX exporter agent port
metrics_path: /metricsThis setup is not static. Every production system diverges based on table distribution, write amplification, and downstream pressure, but starting from this baseline avoids the common failure modes that appear when relying on default connector configurations.
Postgres CDC Architecture Implementation
A typical production deployment uses Kafka Connect in a distributed setup with tuned connector configuration:
tasksMax: 1
max.batch.size: 2048
max.queue.size: 8192
poll.interval.ms: 500
heartbeat.interval.ms: 30000Scaling comes from splitting tables across connectors, not from more tasks. JVM heap sizing must match table count and snapshot size, with 4GB as a baseline for mid-sized deployments.
Postgres configuration is equally critical. Settings such as max_replication_slots, max_wal_senders, and wal_sender_timeout must be tuned to avoid slot starvation and connection drops.
Advanced Debezium CDC Patterns
Schema evolution requires explicit handling. The Postgres connector does not use a schema history topic. It learns table structure from the relation messages in the pgoutput stream. Structured formats such as Avro with a schema registry, and testing DDL changes before production, prevent failures downstream.
Connector isolation is another key pattern. Grouping tables into separate connectors limits blast radius when one schema or workload behaves unexpectedly.
Snapshot strategy should remain controlled. Running with snapshot.mode: always, which resnapshots on every restart, often leads to memory pressure and instability. Initial snapshot plus streaming with heartbeat monitoring provides a stable baseline.
Across multiple systems, removing repeated snapshots eliminated recurring out-of-memory failures and reduced recovery time.
Operational Next Steps
Run this under load before trusting it. Introduce artificial lag, pause replication, and push throughput beyond expected peaks. Watch how quickly lag grows, how snapshots behave, and where recovery breaks down.
In production CDC architecture, initial setup is not the hard part. Systems fail during spikes, failovers, and schema changes. The difference between a pipeline that works and one that holds under pressure comes down to observability, WAL discipline, and clear operational boundaries.
Need help with distributed systems, backend engineering or data platforms? See my consulting services.
Founder of Scalytics, co-creator and PMC member of Apache Wayang, and author of KafScale. I write about Kafka, streaming, data platforms, IoT and AI agent infrastructure in production.