Skip to main content

Stream IoT data to S3 - the simple way

Image
By
Struggling with delivery, architecture alignment, or platform stability?

I help teams fix systemic engineering issues: processes, architecture, and clarity.
→ See how I work with teams.


This article introduces infinimesh as a Kubernetes-native IoT platform built to integrate massive device fleets without cloud lock-in, and highlights its expanding plugin ecosystem for Elastic, Redis TimeSeries, SAP HANA, Snowflake and cloud-native object storage. Using lightweight Go-based Docker containers, plugins can run securely in user environments—even on AWS free-tier instances—while streaming device data directly into existing infrastructures like S3 or MinIO. The step-by-step example shows how quickly CloudConnect can move IoT data into object storage using docker-compose, with the plugin internally batching device payloads through Redis before exporting them as CSV. In benchmarking, the architecture handled millions of devices sending frequent JSON updates, demonstrating how infinimesh plugins simplify large-scale IoT data integration with minimal resource overhead.

First, a short introduction to infinimesh, an Internet of Things (IoT) platform which runs completely in Kubernetes: 

infinimesh enables the seamless integration of the entire IoT ecosystem independently from any cloud technology or provider. infinimesh easily manages millions of devices in a compliant, secure, scalable and cost-efficient way without vendor lock-ins.

We released some plugins over the last weeks - a task we had on our roadmap for a while. Here is what we have so far:

Elastic

Connect infinimesh IoT seamless into Elastic.

Timeseries

Redis-timeseries with Grafana for Time Series Analysis and rapid prototyping, can be used in production when configured as a Redis cluster and ready to be hosted via Redis-Cloud.

SAP Hana

All code to connect infinimesh IoT Platform to any SAP Hana instance

Snowflake

All code to connect infinimesh IoT Platform to any Snowflake instance.

Cloud Connect

Acloud native connector to integrate Infinimesh IoT directly into Cloud Native Storage of AWS, GCP and Azure. 

All code to connect infinimesh IoT Platform to Public Cloud Provider AWS, GCP and Azure. This plugin enables customers to use their own cloud infrastructure and extend infinimesh to other services, like Scalytics, using their own cloud native data pipelines and integration tools.

We have chosen Docker as main technology, because it enables users to run their own plugins in their own space in their controlled environment. And, since our plugins don't consume so much resources, they fit perfectly into the free tiers of AWS EC2 - I use them here in that blog post.

The plugin repository was structured with developer friendliness in mind. All code is written in Go, and the configuration will be done on dockerfiles. Since you need to put credentials into, we highly advise to run the containers in a controlled and secure environment. 

infinimesh UI

Stream IoT data to S3

Here I like to show how easy it is to combine IoT with already installed infrastructures in public clouds. The most used task we figured is the data stream to S3; most of our customers use S3 either directly with AWS, or by implementing their own object storage using the S3 protocol, like MinIO - which is also Kubernetes native.

Of course a private installation of infinimesh or accounts on infinimesh.cloud and AWS are needed, if using the cloud version of both. Here is a screenshot from the SMA device I used to write this post:

Preparation

  1. Spin up an EC2 instance in the free tier with Linux, a t2.micro instance should fit mostly all needs
  2. Log into the VM and install docker as described in the AWS documentation: Docker basics for Amazon ECS - Amazon Elastic Container Service
  3. Install docker-compose and git:
sudo curl -L \
https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)\
-o /usr/local/bin/docker-compose \
&& sudo chmod +x /usr/local/bin/docker-compose \
&& sudo yum install git -y

That’s all we need as preparation, now log-off and login again to enable the permissions we have set earlier. 

Setup and Run

  1. Clone the plugin - repo:
    git clone https://github.com/2pk03/infinimesh-plugins.git
  2. Edit the CloudConnect/docker-compose.yml and replace CHANGEME with your credentials
  3. Compose and start the connector (-d detaches from the console and let the containers run in background):
    docker-compose -f CloudConnect/docker-compose.yml --project-directory . up --build -d
  4. Check the container logs:
    docker logs plugins_csvwriter_1 -f
We used Go as development language, therefore the resource consumption is low:


After one minute the first CSV file should be arriving in S3. That’s all - easy and straightforward.



 Some developer internals

We have built some magic around to make the use of our plugins as easy as possible for customers and at the same time easy to adapt for developers.

First we iterate over /objects, finding all endpoints marked with [device], call the API for each device and store the data as a sliding window into a local redis store, to buffer network latency. After some seconds we send the captured data as CSV to the desired endpoints. In our tests we transported data from up to 2 Million IoT devices over this plugin, each of those devices send every 15 seconds ten key:value pairs as JSON.

If you need help with distributed systems, backend engineering, or data platforms, check my Services.

Labels:

Most read articles

Building a Model-Agnostic Multi-Agent System with OpenClaw

By
Over one week we rebuilt our AI stack around OpenClaw’s multi-agent architecture to avoid provider lock-in and stop wasting premium tokens. By aligning models to tasks, diversifying fallbacks across providers, enforcing minimal tool access, and switching to memory-first workflows with ephemeral sessions, we reduced token usage per task by about 70% and cut our monthly bill by 77% while improving operational resilience. How We Achieved 77% Cost Reduction and Provider Independence Over the past week, we rebuilt our AI infrastructure around OpenClaw’s multi-agent architecture. The result was a 77% cost reduction , provider independence , and a delegation system that routes work to the most cost-effective model for each job. Below is the technical journey of optimizing a 7-agent squad with OpenClaw. The Challenge: Model Provider Lock-In We started with a simple problem: our entire squad defaulted to a single model provider. This created three issues: Cost inefficiency beca...
Read more

BacNet => MQTT in Production: The Real Cost of Bridging BACnet to MQTT at Scale

By
bacnet2mqtt looks simple in a README and expensive in production. Once BACnet polling, reconnection behavior, stale state, and MQTT publishing collide, teams discover they are not deploying a lightweight adapter but operating infrastructure. This article breaks down where bacnet2mqtt works, where it becomes a bottleneck, and which production patterns reduce the operational damage before incidents, backlogs, and silent data loss turn a building integration into a long-running engineering problem. I inherited a building controls integration problem 18 months ago. Three office floors. 217 BACnet sensors covering temperature, occupancy, and HVAC actuators. The data was trapped inside the building automation network while the business wanted analytics, reporting, and compliance visibility in the data platform. The obvious answer looked easy enough: deploy bacnet2mqtt, bridge BACnet into MQTT, and push the stream into the lakehouse stack. The repository made it sound like a w...
Read more

Connect BACnet to the Cloud with bacnet-mqtt-gateway

By
The bacnet-mqtt-gateway project is an open source protocol bridge that translates BACnet building automation traffic into MQTT messages for cloud and IoT systems. It provides discovery, polling, bidirectional writes, APIs, security, and easy deployment via Docker. Many enterprises struggle to unify BACnet with modern data pipelines and cloud platforms because BACnet is local-network only and not cloud ready. This gateway provides a scalable, secure, production-ready adapter for MQTT ecosystems and smart building integrations. The Problem with BACnet Building automation runs on BACnet . HVAC controllers, lighting systems, metering equipment: they all speak ASHRAE 135 . The protocol handles local control loops well. It fails at cloud ingress. BACnet relies on UDP broadcasts. These do not route over the internet or into VPCs. Your chiller controller cannot talk to AWS IoT Core . Your VAV box cannot publish to an MQTT broker. The air gap between operational technology and modern cl...
Read more