Skip to main content

Get Apache Flume 1.3.x running on Windows

By Anonymous
Struggling with delivery, architecture alignment, or platform stability?

I help teams fix systemic engineering issues: processes, architecture, and clarity.
→ See how I work with teams.


Since we found an increasing interest in the flume community to get Apache Flume running on Windows systems again, I spent some time to figure out how we can reach that. Finally, the good news - Apache Flume runs on Windows. You need some tweaks to get them running.

Prerequisites
Build system:
maven 3x, git, jdk1.6.x, WinRAR (or similar program)

Apache Flume agent:
jdk1.6.x, WinRAR (or similar program), Ultraedit++ or similar texteditor

Tweak the Windows build box
1. Download and install JDK 1.6x from Oracle
2. Set the environment variables
   => Start - type "env" into the search box, select "Edit system environment variables", click Environment Variables, Select "New" from the "Systems variables" box, type "JAVA_HOME" into "variable name" and the path to your JDK installation into "Variable value" (Example: C:\Program Files (x86)\Java\jdk1.6.0_33)
3. Download maven from Apache
4. Set the environment variables
   => from the field "System variables" select New, Variable name "M2_HOME", value Path to your maven installation (Example: D:\Maven\apache-maven-3.0.4)
   => from the field "User variables for (your username)" select New, Variable name "M2", value "%M2_HOME%\bin"
   => from the field "User variables for (your username)" select New, Variable name "MAVEN_OPTS", value "-XX:MaxPermSize=1024M"
5. Close the CMD if you have running one
6. Download and install msysgit or use a similar program


Build
Start git and clone flume's git repo to the local harddisk. I created a Source directory on my secondary HDD (D:) and cloned the repo into. Change into the previous checkout'd directory and start to build your Apache Flume installation with maven.
D:\Source\flume>mvn clean
D:\Source\flume>mvn package -DskipTests


You'll see a lot of noise, but after some minutes you get the success:
[INFO] Apache Flume ...................................... SUCCESS [39.780s]
[INFO] Flume NG SDK ...................................... SUCCESS [3.510s]
[INFO] Flume NG Configuration ............................ SUCCESS [1.825s]
[INFO] Flume NG Core ..................................... SUCCESS [3.853s]
[INFO] Flume NG Sinks .................................... SUCCESS [0.874s]
[INFO] Flume NG HDFS Sink ................................ SUCCESS [2.652s]
[INFO] Flume NG IRC Sink ................................. SUCCESS [1.310s]
[INFO] Flume NG HBase Sink ............................... SUCCESS [2.527s]
[INFO] Flume NG Channels ................................. SUCCESS [0.796s]
[INFO] Flume NG JDBC channel ............................. SUCCESS [2.168s]
[INFO] Flume NG Node ..................................... SUCCESS [2.481s]
[INFO] Flume NG file-based channel ....................... SUCCESS [2.371s]
[INFO] Flume NG file backed Memory channel ............... SUCCESS [2.075s]
[INFO] Flume legacy Sources .............................. SUCCESS [0.733s]
[INFO] Flume legacy Avro source .......................... SUCCESS [1.778s]
[INFO] Flume legacy Thrift Source ........................ SUCCESS [1.903s]
[INFO] Flume NG Clients .................................. SUCCESS [0.702s]
[INFO] Flume NG Log4j Appender ........................... SUCCESS [1.498s]
[INFO] Flume NG distribution ............................. SUCCESS [14.477s]
[INFO] Flume NG Integration Tests ........................ SUCCESS [1.529s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:29.310s
[INFO] Finished at: Mon Jul 02 09:19:11 CEST 2012
[INFO] Final Memory: 195M/844M
[INFO] ------------------------------------------------------------------------


As usual, the build you'll find in flume-ng-dist\target\:
02.07.2012  09:19        14.161.677 flume-ng-dist-1.3.0-SNAPSHOT-dist.tar.gz
02.07.2012  09:19           467.362 flume-ng-dist-1.3.0-SNAPSHOT-src.tar.gz
Now grab the *dist.tar.gz and extract them on a Windows server of your choice. I used WinRar for.

Running Flume
Create a config file (I used the one I posted on my Linux related build post) and created the file with Ultraedit++. Why? The included Windows editor save files with Windows LF at the end, and that will hurt Flume. 

You've to edit the log4j.properties file too, simple change the logger facility from 
flume.root.logger=INFO,LOGFILE
into
flume.root.logger=DEBUG,CONSOLE

Start Flume:
D:\flume\flume-1.3.0-SNAPSHOT>"c:\Program Files (x86)\Java\jdk1.6.0_33\bin\java.exe" -Xmx20m -Dlog4j.configuration=file:///%CD%\conf\log4j.properties -cp "d:\flume\flume-1.3.0-SNAPSHOT\lib\*" org.apache.flume.node.Application -f d:\flume\flume-1.3.0-SNAPSHOT\conf\test1.conf -n syslog-agent
Explanation:
Path to java.exe + Java related options + log4j config in Windows Uri style + Flume libs + application + config

Tip: tweak the path to match your installation

You'll see:
.. a lot of DEBUG noise ..
2012-07-02 11:30:54,823 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:126)] Starting Sink Console
2012-07-02 11:30:54,825 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:137)] Starting Source Syslog
2012-07-02 11:30:54,828 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:143)] Polling sink runner starting
2012-07-02 11:31:24,829 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)] Checking file:d:\flume\flume-1.3.0-SNAPSHOT\conf\test1.conf for changes
                                                                                                       
Now fire up a syslog message from a compatible system:
echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo
nc -v WINDOWS SERVER 5140 < /tmp/foo

Tip: Replace WINDOWS SERVER with the server IP of your Windows box

On the Windows Box you'll see:
2012-07-02 11:36:38,114 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process (LoggerSink.java:70)] Event: { headers:{timestamp=1340187132000, Severity=5, host=host, Facility=8} body: 66 6F 6F 5B 33 34 35 5D 3A 20 61 20 73 79 73 6C foo[345]: a sysl }

Summary
I've only tested a syslog flow. Special flows like Apache HBase won't run, since Apache HBase isn't running on Windows (yet). I think Avro will run too, as well as exec sinks. That open a real wide field for Apache Flume's integration into BI solutions or mass eventlog debugging.
For your tracking, the link to the corresponding Jira (FLUME-1335), I think I will write a better Windows service integration in a short time period. Add you as an watcher, so we can create good Karma for the improvement.

Related guides:

If you need help with distributed systems, backend engineering, or data platforms, check my Services.

Labels:

Most read articles

Why Is Customer Obsession Disappearing?

By
Many companies trade real customer-obsession for automated, low-empathy support. Through examples from Coinbase, PayPal, GO Telecommunications and AT&T, this article shows how reliance on AI chatbots, outsourced call centers, and KPI-driven workflows erodes trust, NPS and customer retention. It argues that human-centric support—treating support as strategic investment instead of cost—is still a core growth engine in competitive markets. It's wild that even with all the cool tech we've got these days, like AI solving complex equations and doing business across time zones in a flash, so many companies are still struggling with the basics: taking care of their customers. The drama around Coinbase's customer support is a prime example of even tech giants messing up. And it's not just Coinbase — it's a big-picture issue for the whole industry. At some point, the idea of "customer obsession" got replaced with "customer automation," and no...
Read more

How to scale MySQL perfectly

By
When MySQL reaches its limits, scaling cannot rely on hardware alone. This article explains how strategic techniques such as caching, sharding and operational optimisation can drastically reduce load and improve application responsiveness. It outlines how in-memory systems like Redis or Memcached offload repeated reads, how horizontal sharding mechanisms distribute data for massive scale, and how tools such as Vitess, ProxySQL and HAProxy support routing, failover and cluster management. The summary also highlights essential practices including query tuning, indexing, replication and connection management. Together these approaches form a modern DevOps strategy that transforms MySQL from a single bottleneck into a resilient, scalable data layer able to grow with your application. When your MySQL database reaches its performance limits, vertical scaling through hardware upgrades provides a temporary solution. Long-term growth, though, requires a more comprehensive approach. This invo...
Read more

What the Heck is Superposition and Entanglement?

By
This post is about superposition and interference in simple, intuitive terms. It describes how quantum states combine, how probability amplitudes add, and why interference patterns appear in systems such as electrons, photons and waves. The goal is to give a clear, non mathematical understanding of how quantum behavior emerges from the rules of wave functions and measurement. If you’ve ever heard the words superposition or entanglement thrown around in conversations about quantum physics, you may have nodded politely while your brain quietly filed them away in the "too confusing to deal with" folder.  These aren't just theoretical quirks; they're the foundation of mind-bending tech like Google's latest quantum chip, the Willow with its 105 qubits. Superposition challenges our understanding of reality, suggesting that particles don't have definite states until observed. This principle is crucial in quantum technologies, enabling phenomena like quantum comp...
Read more