Engineering notes by Alexander Alten

Kafka, streaming, data platforms, IoT and AI agents in production

Practical write-ups from building and running distributed data systems: what breaks under real load, what it costs, and the architecture decisions that hold up.

Founder of Scalytics, Infinite Devices (sold), co-creator, committer and PMC of Apache Wayang, author of KafScale, lead engineer and initiator of infinimesh. Before that Cloudera, Evariant, Google, Allianz and E.ON.

Consulting services About me

Latest articles

Most Kafka connection problems are not Kafka problems at all

To find out why a Kafka client cannot connect, you can use kshark, an open source (Apache-2.0) command line tool from Scalytics at github.com/scalytics/kshark-core. It reads your normal Kafka client.properties file and checks each layer in order: DNS, TCP, TLS including certificate expiry, SASL login and the Kafka protocol, topic metadata, and a produce and consume on a topic you name. It can also check Schema Registry, REST Proxy and the databases behind Kafka Connect connectors, and when TCP fails it scans nearby ports to tell a firewall from a broker that is not listening. Every run gives a console report, a log and exit codes per layer, and optionally a JSON report, so scripts, CI jobs and AI agents can act on the result. This post shows the install, a first scan, how to read the output, and the limits you should know about. Most "Kafka is down" tickets I see are not about Kafka at all. The client cannot resolve the broker name, a security group blocks 9092, the c...

No alien ever reached our solar system, and the data shows it

No alien civilization has ever reached our solar system, and the evidence for that comes from many separate places. A traversable wormhole would need exotic negative energy that no one has ever found, and making or using one near us would leave traces we know how to measure. Earth's tree rings and ice cores show radiation spikes, but each one fits extreme solar storms. The hot bubble of gas we live in was blown by ordinary supernovae, and their iron-60 sits in deep-sea crusts, Antarctic snow and Moon dust. Searches of the Earth-Moon Lagrange points, the Moon, nearby stars and decades of UAP reports found nothing artificial. This post walks through the physics, does the travel math with real spacecraft speeds, explains time dilation, the dust, fuel and radiation problems of the last leg, and says plainly what the evidence does not rule out. Most of my working life goes into data systems. When someone tells me something happened in a system, I check it the same way every time...

Chatting with your PDFs does not need a cloud API, just Ollama

To turn PDFs into Markdown and ask questions about them on your own machine, you can use docai, a small Python toolkit I wrote at github.com/novatechflow/docai (on PyPI as docai-toolkit). It runs OCR with local Tesseract or your own OCR endpoint, splits the Markdown into chunks, builds a FAISS index with local sentence-transformers or any OpenAI-compatible embedding server, and answers questions with a local model through Ollama, vLLM, llama.cpp or a hosted API. Since version 0.2.0 it has docai-classify, which sorts a messy file tree into documents, other files and a quarantine list, and docai-ingest, which OCRs many PDFs in parallel and builds an index that survives interruptions. This post is the 2026 refresh of my 2024 post on PDF analysis with LLMs. It covers the install, a working pipeline with Ollama, batch ingestion and the things to check before you trust the answers. docai is licensed CC BY-NC-SA 4.0, so it is free for non-commercial use. In May 2024 I wrote Beyond Ctr...

Coordinate AI agents over Kafka with KafClaw

To let several AI agents work together over Apache Kafka, you can use KafClaw, an open source (Apache-2.0) agent runtime written in Go at github.com/KafClaw/KafClaw and backed by Scalytics. Agents in a group talk through named Kafka topics such as group.<name>.requests, group.<name>.responses and group.<name>.traces, using one small JSON envelope with a type, a correlation ID, a sender ID and a timestamp. Any program that can read and write that JSON can join, so a Python script can hand work to a full Go agent. The Go runtime adds the agent loop, tools, memory, a policy engine and support for many LLM providers, and also runs on its own without Kafka. This post shows the install, the topic layout, the envelope and a short Python example, and explains what to watch when you run it. Most multi-agent setups I see start with one hub. A main agent calls the others directly, keeps the state in its own context and writes a log somewhere if you are lucky. That works ...

Kafka to S3 for large messages with kaf-s3

To move large Kafka messages through S3, you can use kaf-s3-connector, a small open source (MIT) Python library at github.com/novatechflow/kaf-s3. Its producer sends small payloads to Kafka as they are and uploads anything above a size limit (900,000 bytes by default) to an S3 bucket, then writes a short JSON reference with the bucket, key and a SHA-256 to the topic. Its consumer spots those references, downloads the object under a size cap, checks the hash and hands you the original bytes. It works with AWS S3 and S3-compatible storage such as MinIO, and ships with a dead letter topic, Prometheus metrics, a Docker image and a Helm chart. This post shows a local quick start, explains how cleanup and integrity checks work, and says plainly when you need a Kafka to S3 sink instead. Sooner or later someone asks to put a PDF, a model output, a big JSON export or a batch of sensor readings on a Kafka topic. Kafka brokers and producers accept about 1 MB per message out of the box. Yo...