Archive

2012

Historical Impala Setup on RHEL 6 with Kerberos (Impala 0.3 Era)

This is a historical walkthrough from the early Impala 0.3 beta days, showing how Impala was installed manually on RHEL 6, wired to Hive/HDFS configs, integrated with Kerberos using service principals and keytabs, and started via simple scripts. Treat it as a reference for understanding Impala’s components and legacy Kerberos patterns, not as a modern installation guide. Note (2025): The commands, package names and versions in this article describe an Impala 0.3 beta setup on RHEL/CentOS 6 with Oracle JDK 6 and Cloudera’s early repos. Modern Impala deployments use different packaging, Java versions and security defaults. Use this only for maintaining or understanding legacy CDH-era clusters. A small demo script from that time, which loads census data into Hive and HBase and queries it with impala-shell (with -k for Kerberos), is in my impala-scripts repo. What Impala is (in this historical context) Impala provides fast, interactive SQL directly on data stored in Apache ...

Using Hive’s HBaseStorageHandler with Existing HBase Tables

Hive’s HBaseStorageHandler lets you expose HBase tables as Hive tables so you can run SELECT and INSERT statements over HBase data. This article shows how to configure Hive to talk to HBase, create a new HBase table through Hive and attach Hive EXTERNAL TABLE definitions to existing HBase tables using the correct column family mappings. Because Impala shares the Hive Metastore, it also walks through a full example with US census ZIP code income data: create the HBase table, map it in Hive, bulk load the CSV with Pig and query it from Impala. It reflects a Hive 0.9 / HBase 0.92 and early Impala era setup and is mainly useful for legacy clusters and migrations. Note (2025): This guide describes the original HBase integration introduced around HIVE-705 for Hive 0.9 and HBase 0.92. Modern Hadoop stacks often favour storing analytics data in Parquet or Iceberg and querying it via engines like Hive, Impala, Trino or Spark. Use this pattern primarily when you need to understand or mai...

Hive Metastore on PostgreSQL and MySQL and Fixing a Hanging DROP TABLE

On some older Hive deployments with PostgreSQL as the metastore database, DROP TABLE can hang while PostgreSQL shows UPDATE locks on metastore tables. This often happens when certain privilege tables and indexes were not created correctly during an upgrade or manual schema setup. This note shows a legacy DDL patch you can apply to add the missing tables and indexes so DROP TABLE completes successfully again. It also covers why Hive needs an external metastore instead of embedded Derby, and walks through a MySQL metastore setup: my.cnf, database and user, hive-site.xml, driver distribution, schema check and schema upgrades. Always back up your metastore before running any DDL. Important legacy note: The SQL below matches a specific generation of the Hive metastore schema from around 2013. You should only apply it if you have confirmed that these tables and indexes are missing in your metastore and that the definitions match your Hive version. Always test on a non-production copy ...

Fixing Hive “Too Many Counters” MapReduce Failures

When Hive queries use many operators, MapReduce can hit its default counter limit and fail with a “Too many counters” exception. This short note explains why it happens (Hive creates multiple counters per operator), how to raise mapreduce.job.counters.max safely, and how to estimate how many operators your query uses with EXPLAIN so you can tune the setting without guessing. Symptom A Hive job fails with an error like: Ended Job = job_xxxxxx with exception 'org.apache.hadoop.mapreduce.counters.LimitExceededException (Too many counters: 201 max=200)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask Intercepting System.exit(1) The query might be complex, but otherwise looks syntactically fine. The problem is not your SQL, it’s the number of counters that MapReduce is willing to track. Why this happens, operators and counters Hive uses counters to track statistics for built-in operators (see the Hive LanguageManual UDF / op...

Automating HBase Major Compactions with Cron and Kerberos

Major compactions in HBase can be scheduled during low-traffic hours to reduce load on RegionServers. This guide shows how to trigger a compaction from the HBase shell using a simple Ruby script and how to wrap it in a Kerberos-aware cron job. It also shows the targeted option: find the region that holds a given row key with locate_region and compact only that region instead of the whole table. It reflects common operational practice in legacy Hadoop clusters where maintenance windows still matter. Why Schedule Major Compactions? Major compactions rewrite all store files of an HBase table, improving read performance but putting additional pressure on the cluster. Many administrators run them during off-peak windows. By default HBase runs time-based major compactions on its own ( hbase.hregion.majorcompaction , every 7 days with some jitter), which can hit at peak hours. To control the timing, set hbase.hregion.majorcompaction to 0 to disable the automatic runs and trigger them y...

Enabling JMX Monitoring for Apache Flume Agents

Apache Flume exposes operational metrics through Ganglia and JMX, but JMX remains the most flexible option for modern monitoring stacks. This updated guide shows how to enable JMX inside Flume’s environment configuration, choose a safe port, and connect with jconsole or any JMX-capable monitoring system. It also explains how to size Flume memory: batch memory for sources and sinks, heap and direct memory for the file channel, the extra heap fast replay needs, and an example JVM configuration for a large agent. Together they give you a reliable baseline and full visibility into Flume performance and resource usage. Why Use JMX for Flume Monitoring? While legacy setups often used Ganglia, most modern environments integrate JVM metrics into Prometheus, Datadog or similar platforms. JMX is the simplest and most universal way to expose Flume internals such as channel size, event throughput, memory usage and sink performance. Enable JMX in flume-env.sh Edit $FLUME_HOME/conf/flume-e...

BigData - eine Übersicht

(Dieser Artikel ist auch als Slideshow verfügbar:  http://www.slideshare.net/mapredit/big-data-mit-apache-hadoop ) Mehr und mehr drängt sich BigData als nebulöser Begriff in die Fachpresse. Klar ist, wer mithalten will im Business und innovativ zukünftige Projekte erfolgreich zum Abschluss führen will, kommt um das Thema nicht herum. Doch warum kommt man nicht darum herum? Was ist der Beweggrund für das Sammeln riesiger Datenmengen? Der Weg dahin ist recht einfach und wird von vielen Unternehmen bereits seit Jahren betrieben, nur mit ungleich höherem Aufwand an Manpower und finanziellen Investments. Ein Beispiel: Es werden Logfiles durch riesige Datenfarmen zusammengeführt; wochenlange Jobs laufen über Terrabyte an den gewonnen und aufbereiteten Daten. Tritt in der Kette ein Fehler auf, beginnt der Lauf im Idealfall an der unterbrochenen Stelle - oder von vorn. Doch bis dahin muss eine lange Prozesskette eingehalten werden, um brauchbare Daten für eben diesen einen Job ...

How to Configure the Apache Flume HBase Sink

The first Apache Flume HBase sink introduced a simple way to stream events directly into HBase tables. This modernized walkthrough explains how the sink works, what its limitations are, how Flume resolves HBase configuration files, and how to set up a minimal but functional Flume-to-HBase pipeline. Although this feature originated in early Flume versions, many legacy Hadoop deployments still rely on it today. Overview The HBase sink was added to the Flume trunk and provided direct write support from Flume channels into HBase tables. It relies on synchronous HBase client operations and requires that HBase table metadata already exists. The sink handles flushes, transactions and rollbacks, allowing Flume to treat HBase as a durable storage target. Building Flume from Trunk In early versions the HBase sink was only available in the trunk source. The following sequence checks out Flume and builds it using Maven: git clone https://github.com/apache/flume.git cd flume git checko...

Getting Started with Apache Flume NG

Apache Flume NG replaced the original master/collector architecture with lightweight agents that can be wired together to form flexible data flows. This guide starts with where Flume came from: Facebook’s Scribe, the original Flume with agents, collectors and automatic failover chains, and how it moved terabytes of logs a day into HDFS. It then explains what changed with Flume NG, how the agent, channel and sink model works, and walks through simple configurations for syslog ingestion to a console logger and to HDFS. It’s aimed at engineers who still operate Flume in legacy estates or need to understand it for migrations. Before Flume NG, Scribe and the Original Flume Before today’s streaming stacks, Facebook’s Scribe was one of the first systems that could move multi-terabyte daily log volumes reliably. Soon after, Cloudera open sourced Flume (2010). It entered the Apache Incubator in 2011 and became the standard way to ship logs into Hadoop. It was fully open source, fast a...

Hadoop Snappy compression example with Hive

Snappy is a fast compression codec widely used in Hadoop ecosystems. This updated guide shows how to generate data, upload it to HDFS, process it with Hive using Snappy compression, verify the output files, and load Snappy-compressed data back into Hive using modern Hadoop commands. To use Snappy compression in Hadoop with Hive, turn on compressed output and set the codec to SnappyCodec before you write the table: SET hive.exec.compress.output=true; SET mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec; SET mapreduce.output.fileoutputformat.compress.type=BLOCK; The rest of this post is a full working example. It creates test data, puts it into HDFS, writes a Snappy compressed Hive table, checks the .snappy file and loads it back into Hive. Snappy is a compression library from Google. It is built for speed rather than the smallest files, which is why it is a common choice in Hadoop and Hive pipelines. It works with Hadoop 2.x and 3.x...