Archive

December 2013

How HDFS Protects Your Data

HDFS is still one of the most battle-tested storage layers for large-scale data platforms. It combines replication (and erasure coding in newer Hadoop versions), rack-aware placement, continuous checksum verification, and high-availability metadata services to detect failures early and repair them automatically. Simple guardrails matter just as much: trash, snapshots, and keeping hadoop.tmp.dir and hbase.tmp.dir out of /tmp so a reboot cannot wipe edit logs and checkpoints. This makes HDFS a solid foundation for modern data platform engineering and distributed systems work, not just a legacy Hadoop component. Teams still ask how HDFS protects data and what mechanisms exist to prevent corruption or silent data loss. The durability model of HDFS has been described in detail in books like Hadoop Operations by Eric Sammer, and most of the ideas are still relevant for modern Hadoop 3.x clusters. Beyond the built-in mechanisms described below, many organizations also operate a secon...