novatechflow | Alexander Alten

Kafka Streams uses RocksDB for stateful operations, creating hundreds of database instances per node. Each instance allocates native memory outside JVM heap tracking. A 4GB heap application can consume 20GB+ actual RAM. This article covers production evidence of memory issues, 24x write amplification killing SSDs, compaction storms causing latency spikes, and tuning parameters. Alternatives to RocksDB include in-memory stores, custom StateStore implementations, and remote state backends. Every Kafka Streams application with stateful operations uses RocksDB by default. Most teams discover this fact only after their production system starts exhibiting mysterious latency spikes, memory pressure, or disk I/O that saturates their SSDs. I have debugged RocksDB-related Kafka Streams issues across multiple production deployments. The pattern is consistent: teams scale their Kafka infrastructure, tune their consumer configurations, optimize their topology, and still hit a wall. The wall is...