Kafka to S3 for large messages with kaf-s3
To move large Kafka messages through S3, you can use kaf-s3-connector, a small open source (MIT) Python library at github.com/novatechflow/kaf-s3. Its producer sends small payloads to Kafka as they are and uploads anything above a size limit (900,000 bytes by default) to an S3 bucket, then writes a short JSON reference with the bucket, key and a SHA-256 to the topic. Its consumer spots those references, downloads the object under a size cap, checks the hash and hands you the original bytes. It works with AWS S3 and S3-compatible storage such as MinIO, and ships with a dead letter topic, Prometheus metrics, a Docker image and a Helm chart. This post shows a local quick start, explains how cleanup and integrity checks work, and says plainly when you need a Kafka to S3 sink instead. Sooner or later someone asks to put a PDF, a model output, a big JSON export or a batch of sensor readings on a Kafka topic. Kafka brokers and producers accept about 1 MB per message out of the box. Yo...