Skip to main content


Showing posts from February, 2013

Flume 1.3.1 Windows binary release online

Andy Blozhou, a Chinese Flume enthusiast provide precompiled Windows binaries of Flume-1.3.1, including a startup bat and Avro bat. You can grap this build on their website  : ======== snip ======== This is the flume-ng 1.3.1 windows version for download.   simple usage: unzip the run bin/flume.bat for agent.  run bin/flume-avroclient.bat for avro-client.  Need modify for your own env.  detail: (To compile flume-ng on windows, please reference or my chinese version the windows version of flume 1.3.1 file from 2.unzip the to a directory. 3.install jdk 1.6 from oracle,and set JAVA_HOME of the env. download from http:

LZO Compression with Oozie

It happens, when one of the compression codec is switched to LZO, that Oozie can't start any MR job successfully. Usually this is done per core-site.xml : <property> <name>io.compression.codecs</name> <value>,,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,</value> </property> <property> <name>io.compression.codec.lzo.class</name> <value>com.hadoop.compression.lzo.LzoCodec</value> </property> Oozie reports a ClassNotFound error ( java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found ). To get the jobs running copy or link hadoop-lzo.jar into /var/lib/oozie/ and restart Oozie's server.  The second, most common issue most people forget is to set the shared lib directory: [root@hadoop2 ~]#