Archive

2014

Linux, Kernel and Cluster Tuning for Hadoop and Large Distributed Systems

This guide explains the essential Linux, kernel, memory, and network tuning techniques required to operate high-performance Hadoop and distributed systems. It covers modern configuration practices for swappiness, transparent huge pages, overcommit behavior, socket and port tuning, file descriptor limits, disk behavior, and DNS resolution. It also covers the cluster layer on top of the OS: hardware sizing for a small on-prem cluster, one HDFS directory per disk, an HA NameNode pair, and YARN and Spark resource settings. A short look back at a 2010 Hadoop 0.20 setup shows how much the operational model has changed. Running Hadoop or any large distributed system at scale requires more than good cluster design. Performance and stability depend heavily on the underlying Linux configuration. This guide revisits the classic Hadoop tuning principles from a modern 2025 perspective, explains what still matters, and documents what has changed in recent kernel versions. These tuning p...

Remove HDP and Ambari completely

Note: Hortonworks HDP is end of life and no longer supported; Cloudera replaced it with CDP. Use this script only to clean up old hosts. It's a bit hard to remove HDP and Ambari completely, so I share my removal script here. Works perfectly for me, just adjust the HDFS directory. In my case it was /hadoop #!/bin/bash echo "==> Stop Ambari and Hue" ambari-server stop && ambari-agent stop /etc/init.d/hue stop sleep 10 echo "==> Erase HDP and Ambari completely" yum -y erase ambari-agent ambari-server ambari-log4j hadoop libconfuse nagios ganglia sqoop hcatalog\* hive\* hbase\* zookeeper\* oozie\* pig\* snappy\* hadoop-lzo\* knox\* hadoop\* storm\* hue\* # remove configs rm -rf /var/lib/ambari-*/keys /etc/hadoop/ /etc/hive /etc/hbase/ /etc/oozie/ /etc/zookeeper/ /etc/falcon/ /etc/ambari-* /etc/hue/ # remove ambaris default hdfs dir rm -rf /hadoop # remove the repos echo "==> Remove HDP and Ambari Repo" rm -rf /etc/yum.repos.d/HDP.repo /...