novatechflow | Alexander Alten

Filtering multiple columns in HBase requires combining several SingleColumnValueFilter instances inside a FilterList. By enforcing boolean AND logic across column families, only rows where all required columns contain valid values are returned. This pattern helps control scan output precisely even in older HBase deployments. HBase is a column-oriented database, storing data by column family and qualifier. When executing a scan, filters help reduce the returned data set to only rows matching specific criteria. A frequent challenge is filtering on more than one column simultaneously . For example, you may require that two or more specific columns must contain valid values before a row qualifies. The practical solution is to use multiple SingleColumnValueFilter objects combined in a FilterList . This gives you boolean AND logic across all defined filters. List<Filter> list = new ArrayList<Filter>(2); // Filter on family "fam1", qualifier "VALUE1...

This article shows how to temporarily stop new Hadoop MapReduce jobs from being submitted by enabling ACLs and configuring mapred-queue-acls.xml. Existing jobs continue to run, which makes this pattern useful for maintenance windows or decommissioning work on a classic MapReduce cluster. In a classic Hadoop MapReduce (MRv1) cluster, there are situations where you want to stop accepting new MapReduce jobs while allowing already running jobs to finish. This is especially useful during maintenance, node decommissioning or cluster reconfiguration. One simple way to achieve this is to enable ACLs on the MapReduce job queue and then configure the submission ACL so that effectively nobody is allowed to submit new jobs. 1. Enable ACLs for MapReduce queues First, configure queue ACLs in $HADOOP/conf/mapred-queue-acls.xml . A typical configuration might allow a set of users and groups to submit jobs and a smaller set of admins to manage them: <configuration> <property...

Kerberized Hadoop clusters use SPNEGO for browser authentication. To sign into UIs such as NameNode, ResourceManager, Oozie or HiveServer2, your browser must support SPNEGO, your client must have a valid Kerberos ticket and DNS and realm mappings must match. This guide explains how to enable SPNEGO for modern Firefox, Chrome and Edge. Most Hadoop Web UIs rely on SPNEGO (Simple and Protected GSSAPI Negotiation) to authenticate users through Kerberos. When a browser accesses a Kerberos-protected endpoint such as: http://namenode-host:9870 http://rm-host:8088 http://oozie-host:11000/oozie the server expects the browser to negotiate Kerberos credentials automatically. If the browser is not configured correctly, the user will see repeated login prompts or 401: Unauthorized . Prerequisites You must have a valid Kerberos ticket: kinit your_user@YOUR.REALM DNS and reverse DNS for the Hadoop services must be correct The SPN for the UI must match: HTTP/hostname@...

Apache Flume NG replaced the original master/collector architecture with lightweight agents that can be wired together to form flexible data flows. This guide explains what changed with Flume NG, how the agent–channel–sink model works, and walks through simple configurations for syslog ingestion to a console logger and to HDFS. It’s aimed at engineers who still operate Flume in legacy estates or need to understand it for migrations. From Flume to Flume NG Apache Flume is a distributed log and event collection service. With Flume NG, the project moved away from the original master/client and node/collector design and adopted a simpler, more robust architecture based on standalone agents. Key changes introduced by Flume NG: No external coordination service required for basic operation. No master/client or node/collector roles—only agents . Agents can be chained together to build arbitrary flows and fan-in/fan-out patterns. Lightweight runtime; small heap sizes are s...

Snappy is a fast compression codec widely used in Hadoop ecosystems. This updated guide shows how to generate data, upload it to HDFS, process it with Hive using Snappy compression, verify the output files, and load Snappy-compressed data back into Hive using modern Hadoop commands. Snappy is a high-performance compression and decompression library originally developed at Google. It is optimized for speed rather than maximum compression ratio, making it a preferred codec in many Hadoop and Hive pipelines. Snappy is integrated in all modern Hadoop distributions (Hadoop 2.x and 3.x) and works with Hive, MapReduce, and increasingly Tez or Spark-based Hive deployments. 1. Create sample input data Generate a small test file: $ seq 1 1000 | awk '{OFS="\001";print $1, $1 % 10}' > test_input.hive $ head -5 test_input.hive 1^A1 2^A2 3^A3 4^A4 5^A5 2. Upload the data into HDFS Updated command: Hadoop now uses hdfs dfs instead of hadoop dfs . $ hdfs dfs ...

This article walks through exporting HDFS via FUSE and serving it through Samba so that clients can mount it over CIFS. It shows the configuration on RHEL, how to mount HDFS, how to test read and write performance with rsync and dd, and compares the results to direct SCP transfers. The setup is historical but still useful to understand the trade-offs and overhead of exposing HDFS as a network filesystem. Some years ago I experimented with exposing HDFS to non-Hadoop clients. After playing with libhdfs and NFS and not getting the results I wanted, the next idea was: why not try Samba? Samba 3.x was stable, and most operating systems could mount a CIFS share without additional software. The main goal was to understand the setup and performance characteristics of this scenario, because Samba has many tuning options. The environment used in this experiment was RHEL 5.7 with the stock Samba packages. 1. Environment and packages # rpm -qa | grep samba samba-3.0.33-3.29.el5_7.4....

This guide shows how to mount an HDFS filesystem using FUSE and then export part of it over NFS so that remote systems can access HDFS like a local filesystem. The approach is based on classic Hadoop and Linux tools, and includes notes on security, user mapping, and kernel limitations. In some environments, it can be useful to make an HDFS filesystem available across networks as an exported share. This walkthrough describes a working scenario using Linux and Hadoop with tools that are typically included in older Hadoop distributions. The setup uses hadoop-fuse-dfs and libhdfs to mount HDFS locally, and then exports that mount over NFS. Replace namenode.local and <PORT> with values appropriate for your cluster. 1. Install FUSE and libhdfs yum install hadoop-0.20-fuse.x86_64 hadoop-0.20-libhdfs.x86_64 2. Create a mountpoint mkdir /hdfs-mount 3. Test mounting HDFS via FUSE hadoop-fuse-dfs dfs://namenode.local:<PORT> /hdfs-mount -d If the mount succee...

This article explains how to debug runtime issues in classical Hadoop MapReduce clusters using JobTracker stack traces, jps, vmstat, and thread-level CPU analysis. Updated notes show how these same debugging principles apply in modern YARN-based clusters, including ResourceManager, NodeManager and NameNode troubleshooting, improved commands, JMX endpoints, and best practices for memory, networking, and virtualization. In early Hadoop deployments (MRv1), one of the most effective ways to diagnose cluster issues was to inspect JobTracker stack traces and JVM thread states. While modern Hadoop clusters use YARN, the same root causes—network latency, RPC timeouts, NameNode memory pressure, GC stalls, and overloaded system services—still apply today. 1. Legacy Method: Inspect JobTracker Stack Traces In MRv1 clusters, you could view all active JobTracker JVM threads via: http://jobtracker:50030/stacks Example thread dump: Process Thread Dump: 43 active threads Thread 3203101 ...