Most Kafka connection problems are not Kafka problems at all
To find out why a Kafka client cannot connect, you can use kshark, an open source (Apache-2.0) command line tool from Scalytics at github.com/scalytics/kshark-core. It reads your normal Kafka client.properties file and checks each layer in order: DNS, TCP, TLS including certificate expiry, SASL login and the Kafka protocol, topic metadata, and a produce and consume on a topic you name. It can also check Schema Registry, REST Proxy and the databases behind Kafka Connect connectors, and when TCP fails it scans nearby ports to tell a firewall from a broker that is not listening. Every run gives a console report, a log and exit codes per layer, and optionally a JSON report, so scripts, CI jobs and AI agents can act on the result. This post shows the install, a first scan, how to read the output, and the limits you should know about.
Most "Kafka is down" tickets I see are not about Kafka at all. The client cannot resolve the broker name, a security group blocks 9092, the certificate expired last night, someone set PLAIN where the cluster wants SCRAM-SHA-512, or the broker advertises an internal address the client cannot reach. The usual way to find out is a round of nslookup, telnet, openssl s_client and a console producer, and then guessing which of them actually matters.
We built kshark at Scalytics to put that round into one command. It takes the same properties file your client uses, walks the connection from the bottom up and stops at the first layer that breaks, with a hint about what to look at. It is also the tool our agents use when they need to check a Kafka connection, which is why it can write its results as JSON.
Code: github.com/scalytics/kshark-core. Open source under Apache-2.0, one Go binary for Linux, macOS and Windows on amd64 and arm64. Current release v1.1.2. Docs at scalytics.github.io/kshark-core.
What kshark checks
- DNS. Every host in
bootstrap.servershas to resolve. - TCP. A plain connection to each broker port.
- TLS. For
SSLandSASL_SSL, a TLS 1.2 or newer handshake with your CA and client certificate if you set them. It shows the certificate name and expiry date and warns when the server certificate expires within 30 days. - Kafka. SASL login with PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512, an ApiVersions request, then topic metadata for each topic you pass.
- Produce and consume. With
-topicit writes one small test message and reads from the topic. - HTTP services. Schema Registry and REST Proxy, if their URLs are in the properties file.
- Connector targets. The databases behind Kafka Connect connectors, for MongoDB, PostgreSQL, DB2, MySQL, SQL Server, Oracle, Redis and Elasticsearch.
- Network diagnostics. Traceroute and MTU checks, a port scan around the broker when TCP fails, and a check of the broker addresses the cluster advertises, which catches the classic advertised listeners mistake.
Install kshark
Download the release for your platform and check it against the published checksums:
VERSION=1.1.2
BASE=https://github.com/scalytics/kshark-core/releases/download/v${VERSION}
curl -LO ${BASE}/kshark_${VERSION}_linux_amd64.tar.gz
curl -LO ${BASE}/checksums.txt
sha256sum --ignore-missing -c checksums.txt
tar -xzf kshark_${VERSION}_linux_amd64.tar.gz
./kshark --version
On a Mac use darwin_arm64 or darwin_amd64 and shasum -a 256 instead of sha256sum. On Windows the file is a .zip, such as kshark_1.1.2_windows_amd64.zip. Or build from source with Go:
git clone https://github.com/scalytics/kshark-core.git
cd kshark-core
go build -o kshark ./cmd/kshark
Then run ./kshark doctor. It lists which helper tools it found on the machine, such as ping, traceroute, dig and openssl. The network diagnostics need them, so install what is missing on the host you debug from.
Your first scan
kshark reads the same Java properties format as the Kafka clients. Values can use ${VAR}, which keeps the password out of the file:
# client.properties
bootstrap.servers=broker-1.example.com:9092,broker-2.example.com:9092
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.username=kshark
sasl.password=${KAFKA_PASSWORD}
ssl.ca.location=/etc/kafka/ca.pem
chmod 600 client.properties
export KAFKA_PASSWORD='...'
./kshark -props client.properties -topic kshark-healthcheck -y
Without -topic it stops after the connection, login and ApiVersions checks. Without -y it prints the plan and asks before it starts. For Confluent Cloud API keys, --preset cc-plain fills in SASL_SSL and PLAIN. --preset self-scram does the same for SCRAM-SHA-512.
Reading the result
Each check is one row with a component, a target, a layer, a status and often a hint. This is a real run from my machine against a port where nothing was listening:
✅ kafka localhost L3-Network Resolved host
❌ kafka localhost:9 L4-TCP TCP connect failed: dial tcp [::1]:9: connect: connection refused
↳ Hint: Firewall, SG/NACL/NSG, LB listeners, PodNetworkPolicy, or routing.
By default kshark works bottom up and skips the higher layers for a broker once a lower one fails, because a TLS error means nothing when TCP is already down. If you want every layer tested anyway, add -probe-direction=full.
The Kafka errors are translated into something you can act on. A missing topic ACL, a missing group ACL, a failed SASL login and an unavailable leader each get their own message, with a hint such as which ACLs produce and consume need.
The exit code tells you the lowest layer that failed, which is handy in scripts:
| Exit code | Lowest failing layer |
|---|---|
| 0 | Nothing failed |
| 1 | DNS |
| 2 | TCP |
| 3 | TLS |
| 4 | Kafka, HTTP or connector target |
| 5 | Only the network diagnostics |
Check the database behind a Kafka Connect connector
A connector in a failed state often has nothing to do with Kafka. The sink database is unreachable or the password changed. kshark can read the connector config from the Connect REST API and test the target the same way, layer by layer:
export KSHARK_CONNECT_AUTH='admin:secret'
./kshark -y --connect-url https://connect.example.com:8083 \
--connector-name orders-postgres-sink
If the Connect API is not reachable from where you are, pass the connector JSON with --connector-config file.json instead. Combine it with -props and -topic to check the broker and the database in one run.
Use it in scripts, CI and with AI agents
- JSON report.
-json scan.jsonwrites the full report plus a checksum file. Passwords and keys in the echoed config are redacted. - Compare runs.
kshark diff before.json after.jsonshows what changed, andkshark trend --last 10summarises the last JSON reports inreports/. - Repeat.
-watch 5mruns the scan every five minutes and numbers the JSON files. - Hand it to support.
-bundle=/tmp/kshark-diag.tar.gzpacks the report, the log, the redacted properties and system context such as DNS config and routes into one archive. Add-tf-state=terraform.tfstateand it includes a redacted copy of your Terraform state. - Ask a model. Every run writes the analysis prompt as a Markdown file into
reports/, so you can give it to whatever model you trust. With--analyzeand anai_config.jsonin the working directory, kshark sends the redacted report to an OpenAI-compatible endpoint itself and writes an HTML report with the answer.
Our KafClaw agent runtime ships kshark as kafclaw kshark, so an agent that cannot reach its brokers can run the checks itself.
Where kshark helped us
Mostly I use kshark to find and trace connection problems. A few cases where it paid off:
- Linked clusters. Two clusters that had to talk to each other, with SSL and network problems somewhere between them. kshark showed which layer failed on which side.
- Setup mistakes. Wrong security protocol, wrong SASL mechanism, wrong advertised address. The kind of thing that looks like Kafka is down when it isn't.
- Compaction misconfig. Later on it also helped track down topics where the compaction settings were wrong.
Things to know before you run it in production
- The produce test writes a real message. It sends one record with the value
probeand akwirecheckheader to every topic you pass. Use a dedicated health check topic, not a business topic, and give the kshark user Write, Read and Describe on it. - The consume check follows the test message. Without
-group, it reads the partition the test message was written to, so a topic with several partitions works fine. - SASL means PLAIN and SCRAM. GSSAPI (Kerberos) is not wired up in the current code, and OAUTHBEARER and AWS MSK IAM are not supported. For MSK, use a SCRAM listener.
- Write
-bundle=pathwith an equals sign. A bare-bundlewriteskshark-diag-<host>-<timestamp>.tar.gzto the current directory, and-bundle=pathsets your own.-bundle pathwith a space does not set the path. - Reports land in
./reports. The JSON report keeps only the file name you give and is written intoreports/in the current directory, next to the log and the prompt file. The HTML report is only written after a successful AI analysis. - The AI endpoint cannot be on localhost. kshark blocks loopback and link-local addresses for its HTTP calls, including the AI endpoint, so a model on the same machine will be refused. Private network addresses are allowed with a warning.
- Know what the port scan labels mean. When every probed port answers "refused", the host is there and nothing listens, and kshark reports
service_not_listening. A mix of refused and timed out ports with no open port ismixed_filtering. - Treat the properties file like a password. kshark warns when the file is readable by others. Use
${VAR}for secrets, and keep in mind that the reports redact passwords, keys and tokens, but still show host names and topic names. - Build the container yourself. There is a Dockerfile and a Helm chart for a CronJob, but no published image yet. In the container kshark writes its reports to
/app/reports, so mount your volume there. On a read-only root filesystem, use-bundle=/app/reports/diagnostics.tar.gzfor the bundle.
If the connection works and the trouble is in the data, look at my post on corrupted Kafka messages and offset recovery, or production CDC with Debezium if the pain sits in Kafka Connect.
Frequently asked questions
How do I test if I can connect to a Kafka broker?
Run kshark -props client.properties -y with your normal client settings. It checks DNS, TCP, TLS, SASL and the Kafka protocol for each bootstrap server and tells you which layer failed.
Why does my Kafka client time out on connect?
Usually a firewall or security group, a wrong port, or an advertised listener the client cannot reach. kshark scans nearby ports when TCP fails and checks the advertised broker addresses, which narrows it down.
Does kshark work with Confluent Cloud?
Yes. Use the API key and secret as sasl.username and sasl.password and add --preset cc-plain.
Does kshark change anything on my cluster?
Only when you pass -topic. Then it writes one test message per topic. Without -group it reads without a consumer group, so it does not leave committed offsets behind.
Is kshark free?
Yes. The code is open source under Apache-2.0.
Source code
kshark is on GitHub at github.com/scalytics/kshark-core, with unit and fuzz tests and a Docker Compose testbed with Kafka, Kafka Connect, MongoDB, PostgreSQL and DB2 in testbed/. Issues and pull requests are welcome.
If your Kafka clients keep failing in ways nobody can explain, across clouds, private links and firewalls, that is the kind of problem I work on in my Kafka consulting.
Need help with distributed systems, backend engineering or data platforms? See my consulting services.