Archive

October 2011

Syncing Hadoop Clusters Safely Using Modern DistCp and Retention Workflows

This updated guide shows how to synchronize data between Hadoop clusters using modern DistCp practices, HA NameNode paths, correct retention handling, and safe permissions. It modernizes a legacy 2011 workflow into a reliable daily rebase pattern for development and staging clusters without overloading production systems. When developing new pipelines, it is often necessary to rebase a development or staging Hadoop environment with data from production. Historically this was done with simple DistCp scripts. Today, DistCp v2, YARN scheduling, and improved HDFS tooling allow for safer, more performant cluster synchronization while avoiding operational pitfalls. Below is an updated version of a classic rebase workflow: copying the previous day's log data from a production cluster to a development cluster and applying retention by removing older datasets. 1. Variables and runtime setup COPYDATE=$(date -d '-1 day' +"%Y-%m-%d") DELDATE=$(date -d '-3 day...

Hardening Hadoop Clusters with Active Directory, Kerberos, and SELinux

This guide combines legacy experience from early Hadoop deployments with modern best practices to harden clusters using Active Directory, Kerberos, and SELinux. It explains how to integrate Linux hosts with AD, run MapReduce tasks under real user identities, and enforce OS-level controls with SELinux. For smaller or lab clusters it shows a lightweight MIT Kerberos KDC with service principals, keytabs and the HDFS and MapReduce settings. It also explains how to reach Kerberized web UIs such as NameNode, ResourceManager and Oozie through SPNEGO in Firefox, Chrome and Edge, while highlighting where today’s YARN-based stacks differ from the classic JobTracker era. Securing a Hadoop cluster has always been more than just flipping a switch. It touches authentication, operating system hardening, networking, and the way jobs are executed. This article merges and modernizes two older posts from 2011, walking through: Integrating Hadoop nodes with a Windows Active Directory forest usin...

Apache Sqoop with Microsoft SQL Server, Updated Guide

This guide updates an older 2011 post on using Microsoft’s SQL Server Hadoop Connector with Sqoop. It explains how the legacy connector worked, how to configure Sqoop with the current Microsoft JDBC driver for SQL Server, and how to tune Sqoop exports with mappers, JDBC batching, rows per statement and statements per transaction. It also shows how to generate and reuse a precompiled export job JAR, and what to consider now that Apache Sqoop is retired and modern data stacks often use other ingestion tools. Back in 2011, Microsoft shipped a dedicated SQL Server-Hadoop Connector that plugged into Apache Sqoop as an additional connector package. It automated import and export between SQL Server, HDFS, and Hive, and required a separate download plus the Microsoft JDBC driver. Today, the situation is different: Apache Sqoop is retired and has been moved to the Apache Attic (June 2021). It still works but is no longer actively developed or recommended for new projects. The ...