novatechflow | Alexander Alten

The Apache Flume Team released yesterday the next large release with number 1.2.0. Here a overview about the fixes and additions (thanks Mike, I copy your overview): Apache Flume 1.2.0 is the third release under the auspices of Apache of the so-called "NG" codeline, and our first release as a top-level Apache project! Flume 1.2.0 has been put through many stress and regression tests, is stable, production-ready software, and is backwards-compatible with Flume 1.1.0. Four months of very active development went into this release: a whopping 192 patches were committed since 1.1.0, representing many features, enhancements, and bug fixes. While the full change log can be found in the link below, here are a few new feature highlights: * New durable file channel  * New client API  * New HBase sinks (two different implementations)  * New Interceptor interface (a plugin processing API)  * New JMX-based monitoring support With this release - the first after e...

Since we found an increasing interest in the flume community to get Apache Flume running on Windows systems again, I spent some time to figure out how we can reach that. Finally, the good news - Apache Flume runs on Windows. You need some tweaks to get them running. Prerequisites Build system: maven 3x, git, jdk1.6.x, WinRAR (or similar program) Apache Flume agent: jdk1.6.x, WinRAR (or similar program), Ultraedit++ or similar texteditor Tweak the Windows build box 1. Download and install JDK 1.6x from Oracle 2. Set the environment variables    => Start - type " env " into the search box, select " E dit system environment variables ", click Environment Variables, Select " New " from the " Systems variables " box, type " JAVA_HOME " into " variable name " and the path to your JDK installation into "Variable value" (Example:  C:\Program Files (x86)\Java\jdk1.6.0_33 ) 3. Download maven from Apache 4. Set...

The first Apache Flume HBase sink introduced a simple way to stream events directly into HBase tables. This modernized walkthrough explains how the sink works, what its limitations are, how Flume resolves HBase configuration files, and how to set up a minimal but functional Flume-to-HBase pipeline. Although this feature originated in early Flume versions, many legacy Hadoop deployments still rely on it today. Overview The HBase sink was added to the Flume trunk and provided direct write support from Flume channels into HBase tables. It relies on synchronous HBase client operations and requires that HBase table metadata already exists. The sink handles flushes, transactions and rollbacks, allowing Flume to treat HBase as a durable storage target. Building Flume from Trunk In early versions the HBase sink was only available in the trunk source. The following sequence checks out Flume and builds it using Maven: git clone git://git.apache.org/flume.git cd flume git checkout tr...

Filtering multiple columns in HBase requires combining several SingleColumnValueFilter instances inside a FilterList. By enforcing boolean AND logic across column families, only rows where all required columns contain valid values are returned. This pattern helps control scan output precisely even in older HBase deployments. HBase is a column-oriented database, storing data by column family and qualifier. When executing a scan, filters help reduce the returned data set to only rows matching specific criteria. A frequent challenge is filtering on more than one column simultaneously . For example, you may require that two or more specific columns must contain valid values before a row qualifies. The practical solution is to use multiple SingleColumnValueFilter objects combined in a FilterList . This gives you boolean AND logic across all defined filters. List<Filter> list = new ArrayList<Filter>(2); // Filter on family "fam1", qualifier "VALUE1...

This article shows how to temporarily stop new Hadoop MapReduce jobs from being submitted by enabling ACLs and configuring mapred-queue-acls.xml. Existing jobs continue to run, which makes this pattern useful for maintenance windows or decommissioning work on a classic MapReduce cluster. In a classic Hadoop MapReduce (MRv1) cluster, there are situations where you want to stop accepting new MapReduce jobs while allowing already running jobs to finish. This is especially useful during maintenance, node decommissioning or cluster reconfiguration. One simple way to achieve this is to enable ACLs on the MapReduce job queue and then configure the submission ACL so that effectively nobody is allowed to submit new jobs. 1. Enable ACLs for MapReduce queues First, configure queue ACLs in $HADOOP/conf/mapred-queue-acls.xml . A typical configuration might allow a set of users and groups to submit jobs and a smaller set of admins to manage them: <configuration> <property...

Kerberized Hadoop clusters use SPNEGO for browser authentication. To sign into UIs such as NameNode, ResourceManager, Oozie or HiveServer2, your browser must support SPNEGO, your client must have a valid Kerberos ticket and DNS and realm mappings must match. This guide explains how to enable SPNEGO for modern Firefox, Chrome and Edge. Most Hadoop Web UIs rely on SPNEGO (Simple and Protected GSSAPI Negotiation) to authenticate users through Kerberos. When a browser accesses a Kerberos-protected endpoint such as: http://namenode-host:9870 http://rm-host:8088 http://oozie-host:11000/oozie the server expects the browser to negotiate Kerberos credentials automatically. If the browser is not configured correctly, the user will see repeated login prompts or 401: Unauthorized . Prerequisites You must have a valid Kerberos ticket: kinit your_user@YOUR.REALM DNS and reverse DNS for the Hadoop services must be correct The SPN for the UI must match: HTTP/hostname@...

Apache Flume NG replaced the original master/collector architecture with lightweight agents that can be wired together to form flexible data flows. This guide explains what changed with Flume NG, how the agent–channel–sink model works, and walks through simple configurations for syslog ingestion to a console logger and to HDFS. It’s aimed at engineers who still operate Flume in legacy estates or need to understand it for migrations. From Flume to Flume NG Apache Flume is a distributed log and event collection service. With Flume NG, the project moved away from the original master/client and node/collector design and adopted a simpler, more robust architecture based on standalone agents. Key changes introduced by Flume NG: No external coordination service required for basic operation. No master/client or node/collector roles—only agents . Agents can be chained together to build arbitrary flows and fan-in/fan-out patterns. Lightweight runtime; small heap sizes are s...

Snappy is a fast compression codec widely used in Hadoop ecosystems. This updated guide shows how to generate data, upload it to HDFS, process it with Hive using Snappy compression, verify the output files, and load Snappy-compressed data back into Hive using modern Hadoop commands. Snappy is a high-performance compression and decompression library originally developed at Google. It is optimized for speed rather than maximum compression ratio, making it a preferred codec in many Hadoop and Hive pipelines. Snappy is integrated in all modern Hadoop distributions (Hadoop 2.x and 3.x) and works with Hive, MapReduce, and increasingly Tez or Spark-based Hive deployments. 1. Create sample input data Generate a small test file: $ seq 1 1000 | awk '{OFS="\001";print $1, $1 % 10}' > test_input.hive $ head -5 test_input.hive 1^A1 2^A2 3^A3 4^A4 5^A5 2. Upload the data into HDFS Updated command: Hadoop now uses hdfs dfs instead of hadoop dfs . $ hdfs dfs ...