kafSIEM, an open source Kafka SIEM for air-gapped sites

kafSIEM is an open source (Apache-2.0) entity graph and SIEM layer built on Kafka, maintained by Scalytics at github.com/scalytics/kafSIEM. It reads Kafka traffic from AI agents and public OSINT feeds, writes entities, relationships and provenance into one SQLite file, and serves an analyst desk and a typed OpenAPI. Every graph edge can cite the Kafka record it came from. It runs on a single host with Docker Compose or Helm, needs no cluster database, and Operations mode works on an air-gapped network. Domain packs in plain YAML add detectors for SCADA and drone fleets. This post covers what kafSIEM is, how it works, the three operating modes, how to try it, and what to watch in production.

---

People search for kafSIEM and land on this blog, but until now there was nothing here to read about it. Time to fix that. kafSIEM is the most starred open source project I am involved in. It is built for a question that is simple to ask and hard to answer in most operations teams. We have the alerts, we have the change logs, we have the agent logs, so how do they connect?

The short answer is that the evidence sits in different tools and nobody keeps the link back to the source. kafSIEM keeps that link. It reads events from Kafka, builds an entity graph, and every relationship in that graph points back to the record that created it.

Code: github.com/scalytics/kafSIEM. Open source under Apache-2.0, written in Go with a React analyst desk. Docker Compose or Helm. The OSINT mode runs live at osint.scalytics.io.

"We built kafSIEM to provide an ontology on top of Kafka topics. The events were already in Kafka, but nobody could say which device, agent or change a record belonged to, so the answers ended up in spreadsheets. With an ontology, every record becomes a typed entity or edge, and every edge points back to the Kafka record it came from."

Alexander Alten, CTO at Scalytics

What kafSIEM is and what it is not

kafSIEM is a link analysis layer. It takes events you already have, turns them into typed entities (a device, a platform, an agent, a task, a change) and edges between them, and lets an analyst walk that graph. Each edge carries a citation back to the Kafka record it came from.

It is not a replacement for your existing SIEM, your detection stack or your fleet command system. It reads from them. Your SIEM keeps triaging alerts. kafSIEM answers the questions that usually end up in a spreadsheet, such as "which changes on these devices had no work order" or "is this fault on one drone or on the whole fleet".

Why build a SIEM on Kafka

Kafka is already where the events are. Plant telemetry, change events, agent requests and responses, audit records. More importantly, Kafka gives every record a topic, a partition and an offset. That is a stable address. If a graph edge stores that address, you can always go back and show the exact input that produced it.

This is the same reason I argue that agent observability should be built on replayable event streams and not only on logs and traces. Logs tell you something happened. A replayable log with offsets lets you prove what happened, in order, and replay it.

The same idea applies to devices. My post on verifiable assembly records for IoT devices makes the case for evidence that survives teams and time. kafSIEM applies that thinking to operations data.

How kafSIEM works

KafClaw agents, OT systems, other producers
        |
        v
KafScale or any Kafka cluster
        |
        v
kafsiem-collector   (sole writer)
  - decode envelopes
  - update flows, tasks, traces
  - append entities and edges with provenance
        |
        v
/data/agentops.db  (SQLite, WAL mode)
        |
        v
kafsiem-api   (read-only analyst API, /api/v1)
        |
        v
Caddy + web desk

Docker Compose runs four services: the collector, the analyst API, Caddy serving the web desk, and a headless browser bridge that some OSINT sources need. The split matters. The collector is the only writer. The API only reads, and the one thing it may write is a replay request. A noisy source or a heavy analyst query stays on its own side of that line.

The entity graph and provenance

The graph lives in four SQLite tables.

• entities holds typed objects with IDs like platform:auv-7.
• edges holds relationships with valid_from, an optional valid_to and evidence_msg, the source record.
• provenance records ingest and policy decisions per subject.
• entity_geometry stores GeoJSON features for the map.

Because everything is one SQLite file, there is no database cluster to run. That is a big deal on a ship, in a plant enclosure or on a single industrial PC.

Three operating modes

Mode	UI_MODE	What it does
OSINT	OSINT	Public feeds such as CVE and advisory data, a globe view, alert feeds and clusters of related incidents. No Kafka needed.
Operations	AGENTOPS	Reads Kafka traffic, tracks agent flows and fills the entity graph and ontology desk.
Fusion	HYBRID	Both. Kafka flows are matched to OSINT alerts on things like CVE, sector, geography and actor.

In OSINT mode, kafSIEM only links alerts when independent sources agree. A repeat from the same source never counts as confirmation, and the clustering is deterministic. No LLM decides what belongs together.

Domain packs for SCADA and drone fleets

The core knows nothing about plants or drones. That knowledge comes from packs. A pack is a folder of YAML: entity and edge types, detectors, analyst queries, profile views, map layers and report templates. There is no plugin code. The team that is accountable for a detector can read every line of it.

Here is a real detector from the SCADA pack:

id: change-without-work-order
severity: medium
window: 15m
match:
  pattern: |
    SELECT change_id, engineer_id, device_id, applied_at
      FROM view_scada_changes_without_work_order
     WHERE applied_at >= :window_start
explanation_template: >
  Change {{change_id}} touched {{device_id}} without a linked work order in the active window.
suggested_actions:
  - confirm the change request is authorized
  - attach the missing work order or open an incident

Two packs ship today.

• SCADA for plants, zones, devices, firmware, tags, alarms, changes, engineer sessions, work orders and vulnerabilities. Detectors include purdue-violation, change-without-work-order, firmware-drift, stale-session, alarm-flood-after-change and tradecraft-match.
• Drones for platforms, subsystems, software, missions, sorties, EW events, faults and signoffs. Detectors include cohort-failure-early-warning, silent-subsystem, roe-drift and pre-mission-readiness-gap.

Be honest with yourself about maturity. The pack format is fixed, but the packs themselves are still early (the SCADA pack is at version 0.1.0) and heading to a v1 release. Run make pack-lint after every edit. Packs are checked at startup and there is no hot reload, so a pack change means a restart.

Watching AI agent traffic on Kafka

Operations mode was built to observe agents that talk over Kafka, for example KafClaw agent groups. kafSIEM reads the group topics (announce, requests, responses, tasks.status, traces, observe.audit and others) and rebuilds what happened.

• Flows are grouped by correlation_id.
• Traces are grouped by trace_id.
• Task chains follow task_id and parent_task_id.

An agent action gets the same treatment as a human one. Who acted, what it touched, and which record backs it up. If you have read why we built our multi-agent system on Kafka, this is the audit side of that design.

Two details I like from an operations point of view.

• Replay never touches the live consumer group. A replay gets its own consumer group from AGENTOPS_REPLAY_PREFIX, starts at earliest, and can be limited to some topics. The live tracker keeps its offsets.
• Bad records do not block the stream. Rejected records are committed after the outcome is decided and can be mirrored to a reject topic. If that mirror fails, it shows up in health, but the tracker keeps moving.

A minimal Operations config against KafScale looks like this:

AGENTOPS_ENABLED=true
AGENTOPS_BROKERS=kafscale:9092
AGENTOPS_GROUP_NAME=core
AGENTOPS_GROUP_ID=kafsiem-agentops
AGENTOPS_POLICY_PATH=/config/agentops_policy.yaml
AGENTOPS_REPLAY_ENABLED=true
AGENTOPS_REPLAY_PREFIX=kafsiem-agentops-replay
AGENTOPS_REJECT_TOPIC=group.core.agentops.rejects
AGENTOPS_OUTPUT_PATH=/data/agentops.db
UI_MODE=AGENTOPS
PROFILE=agentops-default

Running kafSIEM on an air-gapped network

Operations mode only needs your Kafka cluster and the box kafSIEM runs on. The core loop makes no outside calls and nothing phones home. The storage is a file on local disk. That is why it fits a closed plant network, a vessel or a site with no internet at all.

Two things to plan for. You need the container images once, from GitHub Container Registry or your own registry. And OSINT mode pulls public feeds by design, so on a fully closed network you run Operations mode and bring outside intelligence in through your own process.

Try kafSIEM in a few minutes

The fastest way needs no Kafka, no API keys and no Docker:

git clone https://github.com/scalytics/kafSIEM.git && cd kafSIEM
make help          # list all targets
make demo-osint    # OSINT relations demo with bundled fixtures
make demo-ontology # Operations ontology desk
make demo-fusion   # Fusion desk

For the full stack with Docker:

cp .env.example .env
docker compose up --build
# UI on http://localhost:8080

There is also an installer that checks Docker, asks for the mode and site settings, and starts the published images:

wget -qO- https://raw.githubusercontent.com/scalytics/kafSIEM/main/deploy/install.sh | bash

Read the script before you pipe it into a shell on anything that matters. Once it runs, the API is easy to poke at:

curl -s http://localhost:8080/api/v1/ontology/packs | jq .
curl -s http://localhost:8080/api/v1/flows | jq .
curl -s http://localhost:8080/api/v1/entities/platform/auv-7 | jq .

What to watch in production

• Back up three files, not one. SQLite runs in WAL mode, so agentops.db, agentops.db-wal and agentops.db-shm belong together. Stop the stack or use a backup-aware snapshot. Copying only the main file while the collector runs gives you a broken backup.
• Keep one writer. The collector owns all writes. Do not serve analyst traffic from it and do not point a second writer at the same file.
• Mount packs read-only. Use /packs read-only, /config for policy and /data for state, with immutable images.
• Name your consumer groups on purpose. The live group and the replay prefix should be easy to spot in your Kafka tooling, so nobody resets the wrong one.
• Treat packs like code. They are YAML, but they decide what an analyst sees as an incident. Review them, lint them, version them.

Where kafSIEM fits with KafScale, KafClaw and Lascaris

kafSIEM is one piece of a small stack. KafScale is the Kafka-compatible transport that keeps data in object storage. KafClaw is the agent plane that puts agent traffic on Kafka. kafSIEM observes, builds the graph and serves the analyst. You can use kafSIEM with any Kafka-compatible cluster. It does not need the other two.

If you need this as a governed, supported platform rather than an open source project you run yourself, that is what Lascaris is for. The Lascaris whitepaper on that page explains the architecture.

Frequently asked questions

Is kafSIEM open source?

Yes. It is Apache-2.0 and the full source is on GitHub at scalytics/kafSIEM.

Does kafSIEM replace my SIEM?

No. It reads from your existing tools and adds the entity graph and link analysis on top. Your SIEM keeps doing alert triage.

Does kafSIEM need Kafka?

OSINT mode does not. Operations and Fusion modes read from Kafka, which can be Apache Kafka, KafScale or another Kafka-compatible broker.

Can kafSIEM run air-gapped?

Yes, in Operations mode. The core loop makes no outside calls and the state is one SQLite file. You only need the container images once.

What database does kafSIEM use?

SQLite in WAL mode. There is no external database to install or run.

How do I add my own detectors?

Write a pack in YAML with your entity types, detector SQL and views, run make pack-lint, mount it under /packs and restart.

Source code and docs

• Code: github.com/scalytics/kafSIEM
• Architecture: docs/architecture.md
• Kafka operator guide: docs/agentops-operator-guide.md
• Product page: scalytics.io/kafsiem

Issues and pull requests are welcome. If you run kafSIEM on a real plant or fleet, I want to hear which detector you wrote first.