Engineering notes by Alexander Alten

Kafka, streaming, data platforms, IoT and AI agents in production

Practical write-ups from building and running distributed data systems: what breaks under real load, what it costs, and the architecture decisions that hold up.

Founder of Scalytics, Infinite Devices (sold), co-creator, committer and PMC of Apache Wayang, author of KafScale, lead engineer and initiator of infinimesh. Before that Cloudera, Evariant, Google, Allianz and E.ON.

Consulting services About me

Latest articles

Chatting with your PDFs does not need a cloud API, just Ollama

To turn PDFs into Markdown and ask questions about them on your own machine, you can use docai, a small Python toolkit I wrote at github.com/novatechflow/docai (on PyPI as docai-toolkit). It runs OCR with local Tesseract or your own OCR endpoint, splits the Markdown into chunks, builds a FAISS index with local sentence-transformers or any OpenAI-compatible embedding server, and answers questions with a local model through Ollama, vLLM, llama.cpp or a hosted API. Since version 0.2.0 it has docai-classify, which sorts a messy file tree into documents, other files and a quarantine list, and docai-ingest, which OCRs many PDFs in parallel and builds an index that survives interruptions. This post is the 2026 refresh of my 2024 post on PDF analysis with LLMs. It covers the install, a working pipeline with Ollama, batch ingestion and the things to check before you trust the answers. docai is licensed CC BY-NC-SA 4.0, so it is free for non-commercial use. In May 2024 I wrote Beyond Ctr...

Coordinate AI agents over Kafka with KafClaw

To let several AI agents work together over Apache Kafka, you can use KafClaw, an open source (Apache-2.0) agent runtime written in Go at github.com/KafClaw/KafClaw and backed by Scalytics. Agents in a group talk through named Kafka topics such as group.<name>.requests, group.<name>.responses and group.<name>.traces, using one small JSON envelope with a type, a correlation ID, a sender ID and a timestamp. Any program that can read and write that JSON can join, so a Python script can hand work to a full Go agent. The Go runtime adds the agent loop, tools, memory, a policy engine and support for many LLM providers, and also runs on its own without Kafka. This post shows the install, the topic layout, the envelope and a short Python example, and explains what to watch when you run it. Most multi-agent setups I see start with one hub. A main agent calls the others directly, keeps the state in its own context and writes a log somewhere if you are lucky. That works ...

Kafka to S3 for large messages with kaf-s3

To move large Kafka messages through S3, you can use kaf-s3-connector, a small open source (MIT) Python library at github.com/novatechflow/kaf-s3. Its producer sends small payloads to Kafka as they are and uploads anything above a size limit (900,000 bytes by default) to an S3 bucket, then writes a short JSON reference with the bucket, key and a SHA-256 to the topic. Its consumer spots those references, downloads the object under a size cap, checks the hash and hands you the original bytes. It works with AWS S3 and S3-compatible storage such as MinIO, and ships with a dead letter topic, Prometheus metrics, a Docker image and a Helm chart. This post shows a local quick start, explains how cleanup and integrity checks work, and says plainly when you need a Kafka to S3 sink instead. Sooner or later someone asks to put a PDF, a model output, a big JSON export or a batch of sensor readings on a Kafka topic. Kafka brokers and producers accept about 1 MB per message out of the box. Yo...

Apache Wayang, the Wunderkind of data processing?

Apache Wayang is an open source cross-platform data processing framework and an Apache top-level project since November 2025. You write a pipeline once, and its cost-based optimizer decides which engine runs each step, for example Postgres, Spark, Flink or a plain Java process, and moves only the data the next step needs. It started as the Rheem research project at the Qatar Computing Research Institute, with papers at VLDB, ICDE and SIGMOD. This post answers whether Wayang is really a wunderkind, explains how it differs from Spark and Trino, shows what the official benchmarks say, and lists where it does not help. Wunderkind is a big word. It means a child who is brilliant early. So is Apache Wayang the wunderkind of data processing? Honestly, no. Wayang is not young and it did not get famous overnight. It took about ten years of research, papers, an Apache incubation and a lot of quiet work on the mailing list. What it has is a good idea that most data teams need and almost n...

Find duplicate Kafka messages and check consumer lag

To find duplicate Kafka messages and check consumer lag from the command line, you can use kaf-inspect, a small open source (Apache-2.0) Python CLI at github.com/novatechflow/kaf-inspect. It finds duplicates by message value, by key or by a nested JSON field such as order.id, checks lag per partition for any consumer group, searches topics like grep, peeks at the first or last messages, and prints a cluster overview. It never joins a consumer group and never commits offsets, so it is safe to point at production groups. It works with TLS and SASL clusters through librdkafka settings. This post shows the commands, explains how the duplicate check and the lag numbers work, and lists the mistakes that make the results misleading. Most Kafka debugging sessions start with the same two questions. Is the consumer behind? And did something get processed twice? The first one you can answer with kafka-consumer-groups.sh if you have a Kafka distribution and Java on the box you are working...

kafSIEM, an open source Kafka SIEM for air-gapped sites

kafSIEM is an open source (Apache-2.0) entity graph and SIEM layer built on Kafka, maintained by Scalytics at github.com/scalytics/kafSIEM. It reads Kafka traffic from AI agents and public OSINT feeds, writes entities, relationships and provenance into one SQLite file, and serves an analyst desk and a typed OpenAPI. Every graph edge can cite the Kafka record it came from. It runs on a single host with Docker Compose or Helm, needs no cluster database, and Operations mode works on an air-gapped network. Domain packs in plain YAML add detectors for SCADA and drone fleets. This post covers what kafSIEM is, how it works, the three operating modes, how to try it, and what to watch in production. People search for kafSIEM and land on this blog, but until now there was nothing here to read about it. Time to fix that. kafSIEM is the most starred open source project I am involved in. It is built for a question that is simple to ask and hard to answer in most operations teams. We have th...

infinimesh 3.4 brings rules, alarms and OTA to the device layer

Summary infinimesh is an Apache-2.0 IoT backend you run on your own servers. Version 3.4 is in release candidate testing now. It adds rules and alarms with CEL conditions, an OTA contract in the device shadow, a file service, fleet bulk updates, quotas, metrics, CoAP ingest, an audit log, and a stable v1 API. This post explains what changed, why it matters for EU device makers who want to keep their data at home, and how to try it on one machine in a few minutes. Bottom Line Every connected product ends up building the same backend: a list of devices, rules about who may see them, and a record of what each device last said. I have built that backend more than once, and so have most teams I talk to. infinimesh is my attempt to stop doing it again. Version 3.4 is the release where infinimesh grows from "registry and shadow" into a device layer you can actually run a product on. Rules and alarms, an OTA contract, files, fleet tools, quotas and metrics are now in c...