Skip to main content

Getting Started with Apache Flume NG: Flows, Agents and Syslog-to-HDFS Examples

By Anonymous
Apache Flume NG replaced the original master/collector architecture with lightweight agents that can be wired together to form flexible data flows. This guide explains what changed with Flume NG, how the agent–channel–sink model works, and walks through simple configurations for syslog ingestion to a console logger and to HDFS. It’s aimed at engineers who still operate Flume in legacy estates or need to understand it for migrations.

From Flume to Flume NG

Apache Flume is a distributed log and event collection service. With Flume NG, the project moved away from the original master/client and node/collector design and adopted a simpler, more robust architecture based on standalone agents.

Key changes introduced by Flume NG:

  • No external coordination service required for basic operation.
  • No master/client or node/collector roles—only agents.
  • Agents can be chained together to build arbitrary flows and fan-in/fan-out patterns.
  • Lightweight runtime; small heap sizes are sufficient for simple pipelines.
  • General-purpose exec source instead of dedicated tail/tailDir sources.

Requirements

To build Flume from source you need:

  • A JDK (e.g. 1.6+ in the original context; use a supported JDK for modern builds).
  • Maven 3.x.
  • Git or Subversion to fetch the source code.

Building Flume NG from Source

You can check out and build Flume using Git and Maven: 

git clone git://git.apache.org/flume.git
cd flume
git checkout trunk
mvn clean
mvn package -DskipTests

After a successful build, the distribution artifacts are located under: 

flume-ng-dist/target

Copy the desired distribution archive to the host where you want to run Flume, unpack it and you are ready to start configuring agents.

What Is a Flow in Flume NG?

A flow describes the full path of events from Source to Channel to Sink. Sinks can also feed into other agents, effectively becoming sources for downstream flows.

Conceptually, flows can look like this:

source                -> source => channel => sink
    \                /
     \-> channel => sink

source                -> channel => source => channel => sink

Flume NG runs one or more agents. Each agent hosts its own configured sources, channels and sinks.

The Configuration Model

Flume NG configuration is text-based and follows a logical pattern. For each agent, you declare:

  • A list of sources.
  • A list of channels.
  • A list of sinks.

The naming scheme is: 

<agentName>.sources
<agentName>.channels
<agentName>.sinks

and for each component: 

<agentName>.sources.<sourceName>.property = value
<agentName>.channels.<channelName>.property = value
<agentName>.sinks.<sinkName>.property = value

You are free to choose meaningful names for sources, channels and sinks; those names become the identifiers you wire together.

Example 1: Syslog to Console Logger

The following configuration (syslog-agent.cnf) defines a simple flow:

  • Source: receives syslog over TCP.
  • Channel: in-memory channel.
  • Sink: logger sink, prints events to stdout for debugging.
syslog-agent.sources  = Syslog
syslog-agent.channels = MemoryChannel-1
syslog-agent.sinks    = Console

# Source definition
syslog-agent.sources.Syslog.type = syslogTcp
syslog-agent.sources.Syslog.port = 5140

# Wiring
syslog-agent.sources.Syslog.channels = MemoryChannel-1
syslog-agent.sinks.Console.channel   = MemoryChannel-1

# Sink definition
syslog-agent.sinks.Console.type = logger

# Channel definition
syslog-agent.channels.MemoryChannel-1.type = memory

In this example, the agent syslog-agent listens on TCP port 5140 for syslog messages and writes every event to the console via the logger sink.

Example 2: Syslog to HDFS

To persist events into HDFS instead of logging to stdout, you can swap the sink to an HDFS sink: 

syslog-agent.sources  = Syslog
syslog-agent.channels = MemoryChannel-1
syslog-agent.sinks    = HDFS-LAB

# Source definition
syslog-agent.sources.Syslog.type = syslogTcp
syslog-agent.sources.Syslog.port = 5140

# Wiring
syslog-agent.sources.Syslog.channels   = MemoryChannel-1
syslog-agent.sinks.HDFS-LAB.channel    = MemoryChannel-1

# HDFS sink definition
syslog-agent.sinks.HDFS-LAB.type            = hdfs
syslog-agent.sinks.HDFS-LAB.hdfs.path       = hdfs://NN.URI:PORT/flumetest/%{host}
syslog-agent.sinks.HDFS-LAB.hdfs.filePrefix = syslogfiles
syslog-agent.sinks.HDFS-LAB.hdfs.rollInterval = 60
syslog-agent.sinks.HDFS-LAB.hdfs.fileType   = SequenceFile

# Channel definition
syslog-agent.channels.MemoryChannel-1.type = memory

This configuration listens for syslog events and writes them into HDFS, rolling files every 60 seconds with the prefix syslogfiles.

Starting an Agent

Flume NG runs one agent per process. To start an agent with a specific configuration file: 

bin/flume-ng agent -n YOUR_AGENT_NAME -f YOUR_CONFIG_FILE

For the syslog example: 

bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf

Once started, the agent will bind to the configured syslog port and begin routing events through the defined channel and sink.

Further Reading

Related guides:

If you need help with distributed systems, backend engineering, or data platforms, check my Services.

Labels:
Location: Stuttgart, Germany

Most read articles

Building a Model-Agnostic Multi-Agent System with OpenClaw

By
Over one week we rebuilt our AI stack around OpenClaw’s multi-agent architecture to avoid provider lock-in and stop wasting premium tokens. By aligning models to tasks, diversifying fallbacks across providers, enforcing minimal tool access, and switching to memory-first workflows with ephemeral sessions, we reduced token usage per task by about 70% and cut our monthly bill by 77% while improving operational resilience. How We Achieved 77% Cost Reduction and Provider Independence Over the past week, we rebuilt our AI infrastructure around OpenClaw’s multi-agent architecture. The result was a 77% cost reduction , provider independence , and a delegation system that routes work to the most cost-effective model for each job. Below is the technical journey of optimizing a 7-agent squad with OpenClaw. The Challenge: Model Provider Lock-In We started with a simple problem: our entire squad defaulted to a single model provider. This created three issues: Cost inefficiency beca...
Read more

Why Is Customer Obsession Disappearing?

By
Many companies trade real customer-obsession for automated, low-empathy support. Through examples from Coinbase, PayPal, GO Telecommunications and AT&T, this article shows how reliance on AI chatbots, outsourced call centers, and KPI-driven workflows erodes trust, NPS and customer retention. It argues that human-centric support—treating support as strategic investment instead of cost—is still a core growth engine in competitive markets. It's wild that even with all the cool tech we've got these days, like AI solving complex equations and doing business across time zones in a flash, so many companies are still struggling with the basics: taking care of their customers. The drama around Coinbase's customer support is a prime example of even tech giants messing up. And it's not just Coinbase — it's a big-picture issue for the whole industry. At some point, the idea of "customer obsession" got replaced with "customer automation," and no...
Read more

What are the performance implications of cross-platform execution within Wayang?

By
Apache Wayang ® enables cross-platform execution across multiple data processing platforms such as Spark, Flink, Java Streams, PostgreSQL or GraphChi. This capability fundamentally changes the performance behavior of distributed data pipelines. Wayang reduces manual data movement by selecting where each operator should run, but crossing platform boundaries still introduces serialization cost, shifts in locality, different memory strategies and new tuning constraints. Understanding these dynamics is essential before adopting Wayang for multi-platform pipelines at scale. Apache Wayang is a cross-platform data processing framework that lets developers run a single logical pipeline across engines such as Apache Spark, Apache Flink or a native Java backend. It provides an abstraction layer and a cost-based optimizer that selects the execution platform for each operator. This flexibility introduces new performance variables that do not exist in single-engine systems. Engine boundaries ...
Read more