Coordinate AI agents over Kafka with KafClaw

To let several AI agents work together over Apache Kafka, you can use KafClaw, an open source (Apache-2.0) agent runtime written in Go at github.com/KafClaw/KafClaw and backed by Scalytics. Agents in a group talk through named Kafka topics such as group.<name>.requests, group.<name>.responses and group.<name>.traces, using one small JSON envelope with a type, a correlation ID, a sender ID and a timestamp. Any program that can read and write that JSON can join, so a Python script can hand work to a full Go agent. The Go runtime adds the agent loop, tools, memory, a policy engine and support for many LLM providers, and also runs on its own without Kafka. This post shows the install, the topic layout, the envelope and a short Python example, and explains what to watch when you run it.

---

Most multi-agent setups I see start with one hub. A main agent calls the others directly, keeps the state in its own context and writes a log somewhere if you are lucky. That works on a laptop. It gets hard once you want agents on different machines, in different languages, with different models, and you need to know afterwards who asked what and who answered.

Kafka already solves most of that for services: durable topics, consumer groups, replay and ordering per partition. KafClaw applies the same thing to agents. I explained why we went this way in why we built our multi-agent system on Kafka. This post is about how it works and how to try it.

Code: github.com/KafClaw/KafClaw. Open source under Apache-2.0, written in Go, release binaries for Linux and macOS plus a desktop app. Docs at kafclaw.scalytics.io.

What KafClaw is

KafClaw is two things in one repository.

• A wire protocol on Kafka. A fixed set of topics per group and one JSON envelope. That part has no SDK. If your code can produce and consume JSON on Kafka, it can be an agent in the group.
• A full agent runtime in Go. The kafclaw binary has an agent loop, a tool registry for files, shell, web and memory with sandboxing, a policy engine for message classification, token quotas and rate limits, a scheduler, a local SQLite timeline of everything that happened, and a web dashboard. It also connects to WhatsApp, Slack and Microsoft Teams.

It started as a fork of nanobot and was rewritten in Go. The original attribution is kept in the NOTICE file.

Install KafClaw

The install script downloads the latest release binary for your platform and checks its checksum and signature:

curl --fail --show-error --silent --location \
  https://raw.githubusercontent.com/kafclaw/kafclaw/main/scripts/install.sh \
  | bash -s -- --latest

kafclaw version

Or build from source with Go 1.26:

git clone https://github.com/KafClaw/KafClaw.git
cd KafClaw
make check
make build

Quick start on one machine

Start with local mode. No Kafka, one agent, so you can check the model setup before you add the bus:

kafclaw onboard
kafclaw agent -m "hello"
kafclaw status
kafclaw doctor

The onboarding wizard writes ~/.kafclaw/config.json and creates the workspace identity files (AGENTS.md, SOUL.md, USER.md, TOOLS.md, IDENTITY.md). It asks for the LLM provider. Claude, OpenAI, Gemini, xAI, OpenRouter, DeepSeek, Groq and Scalytics Copilot are built in, and anything OpenAI-compatible works too, including Ollama or vLLM on your own hardware:

kafclaw onboard --non-interactive --profile local \
  --llm openai-compatible \
  --llm-api-base http://localhost:11434/v1 \
  --llm-model llama3.1:8b

Model strings look like claude/claude-sonnet-4-5 or groq/llama-3.3-70b. You can set a primary model and fallbacks per agent, so one agent in the group can run a large hosted model and another a small local one.

Join a group over Kafka

Group mode needs a Kafka cluster. The Go runtime reads the group topics straight from Kafka and writes through the KafScale LFS proxy over HTTP, which is also what stores large memory items in S3. The repository has a Docker Compose file with the whole stack (MinIO, etcd, a KafScale broker, the LFS proxy and three headless agents) in docker-compose.group.yml. Against your own cluster, onboarding looks like this:

kafclaw onboard --non-interactive --profile local-kafka \
  --kafka-brokers localhost:9092 \
  --group-name research \
  --agent-id agent-laptop \
  --role worker \
  --llm skip

kafclaw config set group.lfsProxyUrl "http://localhost:8080"

kafclaw group join research
kafclaw group members

SASL and TLS are supported with --kafka-security-protocol, --kafka-sasl-mechanism and the related flags. If the agent cannot reach the brokers, kafclaw kshark --broker localhost:9092 --test-connection runs the bundled kshark connection checks.

The topic layout

Every group gets its own set of topics, all under group.<name>.:

group.research.announce            # join, leave, heartbeat
group.research.requests            # task requests
group.research.responses           # task responses
group.research.tasks.status        # accepted, in progress, completed, failed
group.research.traces              # trace spans across agents
group.research.observe.audit       # audit trail
group.research.control.roster      # members and capabilities
group.research.control.onboarding  # open or gated onboarding
group.research.memory.shared       # shared knowledge
group.research.memory.context      # short-lived context
group.research.orchestrator        # hierarchy and zones
group.research.skill.<skill>.requests   # routed to agents with that skill

Because it is all Kafka, you get the parts that are hard with a hub for free. You can replay a task from the topic. You can add an observer that reads traces and observe.audit and never touches the agents. And a new agent can read the history before it starts answering. If tracing across agents is your main problem, my post on agent observability for multi-agent systems goes deeper.

The envelope

Every message on every group topic has the same outer shape:

{
  "type": "request",
  "correlation_id": "task-42",
  "sender_id": "agent-laptop",
  "timestamp": "2026-10-24T10:00:00Z",
  "payload": {
    "task_id": "task-42",
    "description": "summarise",
    "content": "Summarise the attached incident notes in five bullet points.",
    "requester_id": "agent-laptop"
  }
}

The types are announce, heartbeat, request, response, task_status, trace, memory, skill_request, skill_response, audit, roster and onboard. A response carries task_id, responder_id, content and a status of completed, failed or rejected.

Talk to the group from Python

This is the part I like most. You do not need the Go runtime to take part. This script sends a task to the group and waits for the first answer, using nothing but confluent-kafka:

import json, uuid
from datetime import datetime, timezone
from confluent_kafka import Producer, Consumer

GROUP, ME = "research", "py-client"
task_id = f"task-{uuid.uuid4().hex[:8]}"
now = lambda: datetime.now(timezone.utc).isoformat()

envelope = {
    "type": "request", "correlation_id": task_id, "sender_id": ME, "timestamp": now(),
    "payload": {"task_id": task_id, "description": "summarise",
                "content": "Summarise the attached incident notes in five bullet points.",
                "requester_id": ME},
}
p = Producer({"bootstrap.servers": "localhost:9092"})
p.produce(f"group.{GROUP}.requests", json.dumps(envelope).encode())
p.flush()

c = Consumer({"bootstrap.servers": "localhost:9092", "group.id": ME, "auto.offset.reset": "latest"})
c.subscribe([f"group.{GROUP}.responses"])
while True:
    msg = c.poll(1.0)
    if msg is None or msg.error():
        continue
    env = json.loads(msg.value())
    if env["payload"].get("task_id") == task_id:
        print(env["sender_id"], env["payload"]["status"], env["payload"]["content"])
        break
c.close()

A Go agent in the group picks the request up from the topic, runs it through its agent loop and answers on responses with the same task ID. The other direction works the same way. A Python worker can consume requests and produce responses.

How a real group splits the work. One setup runs about a dozen agents that manage internal documents and their deadlines, so a team stays within the rules for health certifications.

• Skills. Each agent announces what it can do on group.<name>.control.roster, like searching the web, parsing PDFs or drafting RFP sections. No agent needs to know how the others work.
• Mixed models. Cheap open weight models on vLLM do the high volume work like web search and chunking documents. Claude does the drafting and the checks against health regulations.
• Delegation. When an RFP comes in, a coordinator agent posts tasks to requests, and the search and editing agents answer on responses.
• Zones. Agents that search the public web sit in a public zone, and agents that touch internal compliance documents sit in a private one. They only talk when a zone explicitly bridges them.

What the trace and audit topics showed. In a regulated field, an agent you can't explain is not usable.

• Follow any request. Every envelope carries a trace ID and timestamps, so you can follow one RFP task from agent to agent.
• Find the slow link. When a document deadline slips, group.<name>.traces shows which agent in the chain stalled or failed.
• Replay it. Kafka keeps the log, so a fresh consumer group can read back everything that happened for one trace ID when a compliance officer asks.

Things to know before you run it

• Requests go to every agent. Each Go agent uses its own agent ID as its Kafka consumer group, so every member sees every request on requests, and more than one may answer. Use skill topics when you want a task to reach only the agents that registered for that skill, and match responses on the task ID.
• Delegation has a depth limit. An agent can hand a task to another agent, which can hand it on. The default limit is 3 levels, set with maxDelegationDepth. That limit is what stops two agents from bouncing a task back and forth forever.
• Liveness is heartbeat based. Agents send a heartbeat every 30 seconds by default. The roster shows who is there. It does not promise that an agent will answer.
• Gate who can join. Onboarding is open by default. Set it to gated for a challenge and response handshake, and use Kafka ACLs so only known clients can write to the group topics. Anyone who can write to requests can give your agents work.
• Use zones to separate groups of agents. With the orchestrator enabled, zones are public, shared or private, so the code review agents do not see what the support agents do unless you connect them.
• Remote mode needs a token. The remote profile binds to all interfaces and requires a gateway auth token. Keep the default local bind unless you need remote access.

None of this replaces thinking about your data. Agents on Kafka inherit every schema, ownership and partitioning problem the platform already has. I wrote about that in AI agents on Kafka are only as smart as your architecture, and about why agreement between agents is not the same as being right in AI agent consensus is not truth.

Frequently asked questions

Do all agents have to run KafClaw?

No. An agent is anything that produces and consumes the JSON envelope on the group topics. The Go runtime is one option, a Python or Node service is another.

Which LLM providers does KafClaw support?

Claude, OpenAI, Gemini, xAI, OpenRouter, DeepSeek, Groq, Scalytics Copilot and any OpenAI-compatible endpoint such as Ollama or vLLM. Each agent can use a different model, with fallbacks.

Can I run KafClaw without Kafka?

Yes. The local profile runs a single agent with no Kafka. You add Kafka when you want a group.

Does it work with Confluent, Redpanda or MSK?

The group topics are plain Kafka topics and the Go runtime reads them with a standard Kafka client, with SASL and TLS support. Writing from the Go runtime goes through the KafScale LFS proxy, so plan for that component in group mode.

Where do I see what the agents did?

On the traces, tasks.status and observe.audit topics, in the local SQLite timeline of each agent and in the web dashboard on port 18791.

Source code

KafClaw is on GitHub at github.com/KafClaw/KafClaw, with the full docs at kafclaw.scalytics.io. Issues and pull requests are welcome.

If you want AI agents on your Kafka platform and need help with the topic design, security and operations around them, that is the kind of work I do in my Kafka consulting.