Archive

2013

How HDFS Protects Your Data

HDFS is still one of the most battle-tested storage layers for large-scale data platforms. It combines replication (and erasure coding in newer Hadoop versions), rack-aware placement, continuous checksum verification, and high-availability metadata services to detect failures early and repair them automatically. Simple guardrails matter just as much: trash, snapshots, and keeping hadoop.tmp.dir and hbase.tmp.dir out of /tmp so a reboot cannot wipe edit logs and checkpoints. This makes HDFS a solid foundation for modern data platform engineering and distributed systems work, not just a legacy Hadoop component. Teams still ask how HDFS protects data and what mechanisms exist to prevent corruption or silent data loss. The durability model of HDFS has been described in detail in books like Hadoop Operations by Eric Sammer, and most of the ideas are still relevant for modern Hadoop 3.x clusters. Beyond the built-in mechanisms described below, many organizations also operate a secon...

Connecting SQuirreL SQL to HiveServer2 with Kerberos (Updated)

Kerberos-secured HiveServer2 environments require proper JAAS configuration, a valid Kerberos ticket and the correct Hive JDBC driver. This updated guide explains how to configure SQuirreL SQL with current Hive JDBC drivers, how Kerberos authentication works today and how to avoid the outdated manual classpath and script edits used in early Hadoop distributions. It also explains why HiveServer2 replaced the old Hive CLI, which bypassed security and could not handle many users, and shows how to use Beeline over the same JDBC URL, including running scripts in the background. SQuirreL SQL remains a lightweight and reliable JDBC client for connecting to HiveServer2, especially in on-prem Kerberized clusters. While older Hadoop versions required assembling dozens of JAR dependencies manually, modern Hive distributions ship a shaded JDBC driver that simplifies configuration significantly. Prerequisites for Kerberos Authentication Before launching SQuirreL, ensure that you obtain a ...

Understanding HBase Cross-Cluster Replication for Disaster Recovery

HBase cross-cluster replication provides asynchronous disaster recovery by streaming WAL edits from a source cluster to one or more destination clusters. It is not a high-availability solution; applications must handle failover logic. This updated guide explains replication modes, requirements, configuration steps and operational considerations for modern HBase deployments. HBase offers built-in multi-site replication for disaster recovery (DR). Replication streams write-ahead log (WAL) edits from one cluster to another cluster or a set of clusters. Because replication is asynchronous, it does not provide automatic failover or zero-data-loss guarantees; applications must handle HA logic at the architectural level. Replication Topologies Modern HBase supports several replication patterns: Master → Slave : a primary cluster replicates edits to one or more secondary clusters. Simple and widely used for DR. Master ↔ Master : two clusters replicate to each other. HBase pr...

List Hive Table Sizes in HDFS with a Single Shell Command

This quick tip shows how to list all Hive tables in a database together with their HDFS locations and human-readable sizes using a single bash one-liner. It still works on classic Hive CLI setups and can be adapted easily for Beeline or modern Hive deployments. When you run benchmarks, clean up old data or just want to understand how much space each Hive table consumes, it is useful to see HDFS locations and sizes side by side. Instead of clicking through UIs, you can ask Hive for every table location and then call hdfs dfs -du -h on each path. The Hive + HDFS one-liner The following bash one-liner queries Hive for table locations, extracts the HDFS paths and then prints a human-readable size for each table directory: for file in $(hive -S -e "SHOW TABLE EXTENDED LIKE '\*'" \ | grep "location:" \ | awk 'BEGIN { FS=":" } { printf("hdfs:%s:%s\n",$3,$4) }'); do hdfs dfs -du -h "$file" done Typical outp...

Legacy HBase GC Tuning with CMS, ParallelGC and MSLAB

This short guide explains how legacy HBase clusters were tuned with JVM options for CMS and ParallelGC and how MemStore-Local Allocation Buffers (MSLAB) help reduce GC pauses caused by old-generation fragmentation. Use this as a reference if you still run JDK7/8-based HBase 0.92-era installations or are migrating them, not as guidance for modern Java and HBase versions. Note (2025): The GC flags in this article apply to older Java 7/8 HBase deployments where CMS and ParallelGC were common choices. In newer JVMs, the Concurrent Mark-Sweep (CMS) collector is deprecated and removed in favour of collectors like G1GC. Treat the configuration below as legacy tuning for old HBase stacks, not as recommendations for new clusters. Why HBase GC tuning mattered so much HBase region servers keep a lot of data in memory (memstores and block cache). Without careful JVM tuning, garbage collection can easily cause long pauses, region server timeouts and client-visible outages. Adjusting HBA...