Getting Started with Apache Flume NG
Apache Flume NG replaced the original master/collector architecture with lightweight agents that can be wired together to form flexible data flows. This guide starts with where Flume came from: Facebook’s Scribe, the original Flume with agents, collectors and automatic failover chains, and how it moved terabytes of logs a day into HDFS. It then explains what changed with Flume NG, how the agent, channel and sink model works, and walks through simple configurations for syslog ingestion to a console logger and to HDFS. It’s aimed at engineers who still operate Flume in legacy estates or need to understand it for migrations. Before Flume NG, Scribe and the Original Flume Before today’s streaming stacks, Facebook’s Scribe was one of the first systems that could move multi-terabyte daily log volumes reliably. Soon after, Cloudera open sourced Flume (2010). It entered the Apache Incubator in 2011 and became the standard way to ship logs into Hadoop. It was fully open source, fast a...