novatechflow | Alexander Alten

HBase cross-cluster replication provides asynchronous disaster recovery by streaming WAL edits from a source cluster to one or more destination clusters. It is not a high-availability solution—applications must handle failover logic. This updated guide explains replication modes, requirements, configuration steps and operational considerations for modern HBase deployments. HBase offers built-in multi-site replication for disaster recovery (DR). Replication streams write-ahead log (WAL) edits from one cluster to another cluster or a set of clusters. Because replication is asynchronous, it does not provide automatic failover or zero-data-loss guarantees; applications must handle HA logic at the architectural level. Replication Topologies Modern HBase supports several replication patterns: Master → Slave : a primary cluster replicates edits to one or more secondary clusters. Simple and widely used for DR. Master ↔ Master : two clusters replicate to each other. HBase pre...

This quick tip shows how to list all Hive tables in a database together with their HDFS locations and human-readable sizes using a single bash one-liner. It still works on classic Hive CLI setups and can be adapted easily for Beeline or modern Hive deployments. When you run benchmarks, clean up old data or just want to understand how much space each Hive table consumes, it is useful to see HDFS locations and sizes side by side. Instead of clicking through UIs, you can ask Hive for every table location and then call hdfs dfs -du -h on each path. The Hive + HDFS one-liner The following bash one-liner queries Hive for table locations, extracts the HDFS paths and then prints a human-readable size for each table directory: for file in $(hive -S -e "SHOW TABLE EXTENDED LIKE '\*'" \ | grep "location:" \ | awk 'BEGIN { FS=":" } { printf("hdfs:%s:%s\n",$3,$4) }'); do hdfs dfs -du -h "$file" done Typical outp...

This is a legacy but still useful walkthrough that shows how to expose HBase-resident data to Impala by going through Hive’s Metastore and the HBaseStorageHandler. Using US census ZIP code income data, we create an HBase table, map it with an external Hive table, bulk load the CSV data with Pig and finally query it from Impala. The pattern is mainly relevant today if you are keeping old CDH clusters alive or planning a migration away from Impala-on-HBase towards Parquet or Iceberg tables. Note (2025): This article describes an older CDH/Impala/HBase pattern based on Hive’s HBaseStorageHandler . It is useful if you still maintain legacy Impala-on-HBase workloads or need to understand how such systems were wired. For new designs you will usually land data in Parquet or Iceberg tables and query them with Impala, Trino or Spark instead of reading directly from HBase. Context: Impala, Hive Metastore and HBase Impala uses the Hive Metastore Service to discover tables and their un...

This short guide explains how legacy HBase clusters were tuned with JVM options for CMS and ParallelGC and how MemStore-Local Allocation Buffers (MSLAB) help reduce GC pauses caused by old-generation fragmentation. Use this as a reference if you still run JDK7/8-based HBase 0.92-era installations or are migrating them, not as guidance for modern Java and HBase versions. Note (2025): The GC flags in this article apply to older Java 7/8 HBase deployments where CMS and ParallelGC were common choices. In newer JVMs, the Concurrent Mark-Sweep (CMS) collector is deprecated and removed in favour of collectors like G1GC. Treat the configuration below as legacy tuning for old HBase stacks, not as recommendations for new clusters. Why HBase GC tuning mattered so much HBase region servers keep a lot of data in memory (memstores and block cache). Without careful JVM tuning, garbage collection can easily cause long pauses, region server timeouts and client-visible outages. Adjusting HBA...

Andy Blozhou, a Chinese Flume enthusiast provide precompiled Windows binaries of Flume-1.3.1, including a startup bat and Avro bat. You can grap this build on their website http://abloz.com/flume/windows_download.html  : ======== snip ======== This is the flume-ng 1.3.1 windows version for download. apache-flume-1.3.1-bin.zip   simple usage: unzip the apache-flume-1.3.1-bin.zip run bin/flume.bat for agent.  run bin/flume-avroclient.bat for avro-client.  Need modify for your own env.  detail: (To compile flume-ng on windows, please reference http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html or my chinese version http://abloz.com/2013/02/18/compile-under-windows-flume-1-3-1.html) 1.download the windows version of flume 1.3.1 file apache-flume-1.3.1-bin.zip from http://abloz.com/flume/windows_download.html 2.unzip the apache-flume-1.3.1-bin.zip to a directory. 3.install jdk 1.6 from oracle,and set JAVA_HOME...

When LZO compression is enabled in Hadoop, Oozie may fail to launch MapReduce jobs with a ClassNotFoundException for the LzoCodec. This article explains why it happens, how to fix the missing hadoop-lzo classes on the Oozie server, how to correctly deploy the Oozie sharelib and how to enable the legacy uber-jar feature for MapReduce actions in CDH-era clusters. Note (2025): This article documents behaviour from the CDH 4.x/5.x Oozie + MapReduce stack. Oozie is legacy and many teams migrate to Apache Airflow, Dagster, Argo or cloud-native schedulers, but troubleshooting old clusters still requires understanding these patterns. The solutions here apply to Hadoop MR/LZO deployments still in maintenance or migration mode. Symptom: Oozie fails after enabling LZO compression When you add LZO codecs to core-site.xml , Oozie may suddenly fail to start any MapReduce job. Typical configuration looks like: <property> <name>io.compression.codecs</name> <value...

This is a historical walkthrough from the early Impala 0.3 beta days, showing how Impala was installed manually on RHEL 6, wired to Hive/HDFS configs, integrated with Kerberos using service principals and keytabs, and started via simple scripts. Treat it as a reference for understanding Impala’s components and legacy Kerberos patterns, not as a modern installation guide. Note (2025): The commands, package names and versions in this article describe an Impala 0.3 beta setup on RHEL/CentOS 6 with Oracle JDK 6 and Cloudera’s early repos. Modern Impala deployments use different packaging, Java versions and security defaults. Use this only for maintaining or understanding legacy CDH-era clusters. What Impala is (in this historical context) Impala provides fast, interactive SQL directly on data stored in Apache Hadoop, primarily HDFS and HBase. It reuses: The Hive Metastore and table metadata Hive-compatible SQL syntax ODBC/JDBC drivers and UI components (e.g. Hue Bee...

Hive’s HBaseStorageHandler lets you expose HBase tables as Hive tables so you can run SELECT and INSERT statements over HBase data. This article shows how to configure Hive to talk to HBase, create a new HBase table through Hive and attach Hive EXTERNAL TABLE definitions to existing HBase tables using the correct column family mappings. It reflects a Hive 0.9 / HBase 0.92 era setup and is mainly useful for legacy clusters and migrations. Note (2025): This guide describes the original HBase integration introduced around HIVE-705 for Hive 0.9 and HBase 0.92. Modern Hadoop stacks often favour storing analytics data in Parquet or Iceberg and querying it via engines like Hive, Impala, Trino or Spark. Use this pattern primarily when you need to understand or maintain existing Hive-on-HBase tables, not for new designs. Hive–HBase integration in a nutshell Hive can read from and write to HBase tables using a dedicated storage handler. Once configured, you can: Run SELECT que...