Engineering notes by Alexander Alten

Kafka, streaming, data platforms, IoT and AI agents in production

Practical write-ups from building and running distributed data systems: what breaks under real load, what it costs, and the architecture decisions that hold up.

Founder of Scalytics, Infinite Devices (sold), co-creator, committer and PMC of Apache Wayang, author of KafScale, lead engineer and initiator of infinimesh. Before that Cloudera, Evariant, Google, Allianz and E.ON.

Consulting services About me

Latest articles

Find duplicate Kafka messages and check consumer lag

To find duplicate Kafka messages and check consumer lag from the command line, you can use kaf-inspect, a small open source (Apache-2.0) Python CLI at github.com/novatechflow/kaf-inspect. It finds duplicates by message value, by key or by a nested JSON field such as order.id, checks lag per partition for any consumer group, searches topics like grep, peeks at the first or last messages, and prints a cluster overview. It never joins a consumer group and never commits offsets, so it is safe to point at production groups. It works with TLS and SASL clusters through librdkafka settings. This post shows the commands, explains how the duplicate check and the lag numbers work, and lists the mistakes that make the results misleading. Most Kafka debugging sessions start with the same two questions. Is the consumer behind? And did something get processed twice? The first one you can answer with kafka-consumer-groups.sh if you have a Kafka distribution and Java on the box you are working...

kafSIEM, an open source Kafka SIEM for air-gapped sites

kafSIEM is an open source (Apache-2.0) entity graph and SIEM layer built on Kafka, maintained by Scalytics at github.com/scalytics/kafSIEM. It reads Kafka traffic from AI agents and public OSINT feeds, writes entities, relationships and provenance into one SQLite file, and serves an analyst desk and a typed OpenAPI. Every graph edge can cite the Kafka record it came from. It runs on a single host with Docker Compose or Helm, needs no cluster database, and Operations mode works on an air-gapped network. Domain packs in plain YAML add detectors for SCADA and drone fleets. This post covers what kafSIEM is, how it works, the three operating modes, how to try it, and what to watch in production. People search for kafSIEM and land on this blog, but until now there was nothing here to read about it. Time to fix that. kafSIEM is the most starred open source project I am involved in. It is built for a question that is simple to ask and hard to answer in most operations teams. We have th...

infinimesh 3.4 brings rules, alarms and OTA to the device layer

Summary infinimesh is an Apache-2.0 IoT backend you run on your own servers. Version 3.4 is in release candidate testing now. It adds rules and alarms with CEL conditions, an OTA contract in the device shadow, a file service, fleet bulk updates, quotas, metrics, CoAP ingest, an audit log, and a stable v1 API. This post explains what changed, why it matters for EU device makers who want to keep their data at home, and how to try it on one machine in a few minutes. Bottom Line Every connected product ends up building the same backend: a list of devices, rules about who may see them, and a record of what each device last said. I have built that backend more than once, and so have most teams I talk to. infinimesh is my attempt to stop doing it again. Version 3.4 is the release where infinimesh grows from "registry and shadow" into a device layer you can actually run a product on. Rules and alarms, an OTA contract, files, fleet tools, quotas and metrics are now in c...

AI Agent Consensus Is Not Truth. It Is a Stale Cache Read.

Summary Agent networks are pitched as a replacement for web retrieval: models already hold the knowledge, so you query a crowd of agents instead of a site. This article argues that consensus between agents produces agreement, not freshness. It covers the split between knowledge and state, correlated error in multi-agent debate, why low rank adapters do not override base model dispositions, provenance decay in agent memory graphs, and what an arbitration layer has to do before agent output can be trusted. Bottom Line Polling a network of agents is a quorum read across replicas that may never have seen a recent write. It returns what the replicas hold, which is not the same as what is true now. Somewhere in the loop a node still has to read the origin, and the moment it does you have reinvented the HTTP request with layers of hearsay in front of it. The practical takeaway: design the arbitration layer first. Disagreement between agents is only useful if something resolves ...

Green in Cloudera Manager is not the same as ready

Summary Cloudera clusters can look finished after installation and still fail operationally under real workload shape. This article argues that Cloudera management is a production control surface, not an admin checklist. It covers monitoring gaps, topology assumptions, API-driven metrics, alert design, database dependencies, and runbook discipline. Installation proves the cluster starts, not that it can be run Cloudera Manager fails when teams treat it as the part that happens after installation instead of the control surface for production. The cluster can pass a setup guide and still be hard to operate under load if alerts, metrics, role placement, storage, service ownership, and rollback paths were never proven. The practical takeaway is simple: start with the incident you cannot afford, then design Cloudera Manager, external observability, and runbooks around the signals that would shorten that incident. The business depends on clusters nobody ...

Debezium CDC architecture in production and how it scales

Production CDC architecture breaks under load long before most teams expect it. With Debezium, Kafka Connect, and Postgres, the failure patterns are consistent: WAL pressure builds up, connector lag drifts unnoticed, and snapshot phases exhaust memory under bursty traffic. This is based on running these pipelines across high throughput systems, including workloads above 10k TPS. The difference between a system that works and one that holds under pressure comes down to observability, WAL discipline, and how connector scaling is handled. Production Debezium CDC architecture Operational reality vs. tutorial defaults under real load (10k+ TPS) The Default "Tutorial" Setup Assumes low throughput and stable networks. Fails under pressure. Source: Postgres Single WAL Slot Shared slot coupling multiple connectors Default WAL retention settin...