Skip to main content

From Scribe to Apache Flume: How Early Log Pipelines Scaled Beyond a Terabyte a Day

By Anonymous
Struggling with delivery, architecture alignment, or platform stability?

I help teams fix systemic engineering issues: processes, architecture, and clarity.
→ See how I work with teams.


In the early 2010s, large-scale web companies needed reliable pipelines capable of moving terabytes of log data per day. Facebook’s Scribe pioneered this space, and Apache Flume became the first open, community-driven pipeline for distributed log collection. This refreshed article modernizes the original example and explains how Flume’s agents, collectors and failover chains enabled durable ingestion into HDFS long before today’s streaming platforms.

Before today’s streaming stacks existed, Facebook’s Scribe was one of the first systems that could reliably manage multi-terabyte daily log volumes. Soon after, Apache Flume emerged as an incubator project and quickly became a standard solution for large-scale log transport into Hadoop environments.

Flume introduced a simple, extensible model: sources pull or receive log events, channels buffer them reliably and sinks deliver them into storage systems such as HDFS. For the Hadoop era, this was a breakthrough—fully open source, fast, resilient and without any proprietary dependencies.

Simplifying Deployment in Early Hadoop Clusters

Users of the Cloudera Distribution could install the full Flume stack via: 

yum install flume-master
yum install flume-node

Flume supported several input mechanisms:

  • Plain text file ingestion
  • Tail-following for rolling logs
  • Syslog over UDP or TCP
  • Synthetic or test event sources

Example Architecture: 100-Node Web Cluster

Imagine a large web farm producing hundreds of megabytes of raw logs per minute. Flume agents collect the incoming logs, buffer them and forward the events to Flume collectors. The collectors compress, sort and write the log batches into HDFS—with a well-defined directory layout based on time.

Sample Flume Configuration (Historical Format)

collector1.local : autoCollectorSource | collectorSink(
  "hdfs://namenode.local:9000/user/flume/weblogs/%Y-%m-%d/%H00/%M/",
  "%{host}-"
);
collector2.local : autoCollectorSource | collectorSink(
  "hdfs://namenode.local:9000/user/flume/weblogs/%Y-%m-%d/%H00/%M/",
  "%{host}-"
);
collector3.local : autoCollectorSource | collectorSink(...);
collector4.local : autoCollectorSource | collectorSink(...);

agent1.local : syslogTcp("19800") | autoE2EChain;
agent2.local : syslogTcp("19800") | autoE2EChain;

The autoE2EChain provided an automatic failover path: if a collector became unresponsive, the chain reordered the routing priority. The Flume master UI visualized these logical mappings.

Inspecting Agent State

Every agent maintained internal directories indicating event state: 

# cd /tmp/flume/agent/agent1.local/
# ls
done  logged  sending  sent  writing

Active log files being written appeared in writing/: 

-rw-r--r-- 1 flume flume 418M log.00000019.20111011-162503316+0200.seq

Collector Processing and HDFS Delivery

Collectors compressed batches and stored them in minute-sized partitions in HDFS. Bzip2 was widely used due to its block-marker structure, which allows parallelization and efficient splitting. 

# tail -f -n 10 /var/log/flume/*.log
... Creating compressed HDFS file ...
... Finishing checksum group ...
... moved from partial to complete ...
... Closing HDFS file ...

This pipeline reliably produced compressed, time-partitioned, host-tagged log files—a pattern still used by modern data platforms.

How This Maps to Modern Systems

Although today we rely on Kafka, Fluent Bit, Vector, Iceberg ingest pipelines and cloud-native collectors, the architectural principles remain identical: high-throughput ingestion, buffering for durability, structured partitioning and reliable delivery into analytical storage.

Flume represents one of the foundational steps in the evolution of distributed log ingestion.

References

[1] Apache Flume Documentation
[2] Cloudera Flume Installation Guide

Related guides:

If you need help with distributed systems, backend engineering, or data platforms, check my Services.

Labels:

Most read articles

Building a Model-Agnostic Multi-Agent System with OpenClaw

By
Over one week we rebuilt our AI stack around OpenClaw’s multi-agent architecture to avoid provider lock-in and stop wasting premium tokens. By aligning models to tasks, diversifying fallbacks across providers, enforcing minimal tool access, and switching to memory-first workflows with ephemeral sessions, we reduced token usage per task by about 70% and cut our monthly bill by 77% while improving operational resilience. How We Achieved 77% Cost Reduction and Provider Independence Over the past week, we rebuilt our AI infrastructure around OpenClaw’s multi-agent architecture. The result was a 77% cost reduction , provider independence , and a delegation system that routes work to the most cost-effective model for each job. Below is the technical journey of optimizing a 7-agent squad with OpenClaw. The Challenge: Model Provider Lock-In We started with a simple problem: our entire squad defaulted to a single model provider. This created three issues: Cost inefficiency beca...
Read more

BacNet => MQTT in Production: The Real Cost of Bridging BACnet to MQTT at Scale

By
bacnet2mqtt looks simple in a README and expensive in production. Once BACnet polling, reconnection behavior, stale state, and MQTT publishing collide, teams discover they are not deploying a lightweight adapter but operating infrastructure. This article breaks down where bacnet2mqtt works, where it becomes a bottleneck, and which production patterns reduce the operational damage before incidents, backlogs, and silent data loss turn a building integration into a long-running engineering problem. I inherited a building controls integration problem 18 months ago. Three office floors. 217 BACnet sensors covering temperature, occupancy, and HVAC actuators. The data was trapped inside the building automation network while the business wanted analytics, reporting, and compliance visibility in the data platform. The obvious answer looked easy enough: deploy bacnet2mqtt, bridge BACnet into MQTT, and push the stream into the lakehouse stack. The repository made it sound like a w...
Read more

Get Apache Flume 1.3.x running on Windows

By Anonymous
Since we found an increasing interest in the flume community to get Apache Flume running on Windows systems again, I spent some time to figure out how we can reach that. Finally, the good news - Apache Flume runs on Windows. You need some tweaks to get them running. Prerequisites Build system: maven 3x, git, jdk1.6.x, WinRAR (or similar program) Apache Flume agent: jdk1.6.x, WinRAR (or similar program), Ultraedit++ or similar texteditor Tweak the Windows build box 1. Download and install JDK 1.6x from Oracle 2. Set the environment variables    => Start - type " env " into the search box, select " E dit system environment variables ", click Environment Variables, Select " New " from the " Systems variables " box, type " JAVA_HOME " into " variable name " and the path to your JDK installation into "Variable value" (Example:  C:\Program Files (x86)\Java\jdk1.6.0_33 ) 3. Download maven from Apache 4. Set...
Read more