novatechflow | Alexander Alten

Apache Flink ®, Apache Iceberg ® and governed catalogs such as Databricks Unity Catalog or Snowflake are often pitched as a simple path from Apache Kafka ® JSON to managed tables. In reality Flink is a stream processor, Iceberg is an open table format and the catalog handles governance. None of them infers schemas or models messy payloads for you. You still design schemas, mappings and operations under real Java, DevOps and cost constraints. Many architectural diagrams show a clean pipeline: Kafka into Flink, Flink into Iceberg, Iceberg governed by Unity Catalog or queried from Snowflake. In practice this stack has real friction. Flink is not a neutral glue layer. It is a JVM-centric stream processor with non-trivial operational cost. Iceberg is not a storage engine but a table format that imposes structure. Unity Catalog and Snowflake add their own expectations around governance and schema. Apache Flink is a distributed stream processor for stateful event pipelines. Apache Iceberg i...

SQL-on-Hadoop once promised interactive analytics on distributed storage and transformed early big data architectures. Many engines emerged—Hive, Impala, Drill, Phoenix, Presto, Spark SQL, Kylin, and others—each attempting to bridge the gap between Hadoop’s batch-processing roots and the need for low-latency SQL. This article revisits that era, explains why most of these systems faded, and outlines the modern successors that dominate today’s lakehouse and distributed SQL landscape. The SQL-on-Hadoop Era: What We Learned and What Replaced It In the early 2010s, Apache Hadoop became the backbone of large-scale data processing. As businesses demanded interactive analytics on top of HDFS, a wave of SQL engines emerged. The goal: bring familiar relational querying to a distributed storage layer originally designed for MapReduce batch jobs. By 2015, SQL-on-Hadoop was the hottest category in big data. Today, in 2025, most of those systems have disappeared, evolved, or been replac...

This guide explains how to inspect a row key, find the region responsible for that row and perform a targeted major compaction using modern HBase shell commands. Region-level compaction is useful for maintenance, skewed regions and cleanup of deleted data, but should be used carefully due to its I/O impact. Inspecting a Row Key To view a sample of rows from a table: scan 'your_table', { LIMIT => 5 } To inspect a specific row key: get 'your_table', "\x00\x01" Locate the Region for a Specific Row Modern HBase shells allow you to query region boundaries directly. locate_region 'your_table', "\x00\x01" This returns the region name, start key, end key and hosting RegionServer. You can also list all regions for the table: get_regions 'your_table' Triggering a Major Compaction on a Region Once you know the region name (e.g. your_table,,1712087434000.abc123 ), you can run: major_compact 'your_table,,1...

This guide explains the essential Linux, kernel, memory, and network tuning techniques required to operate high-performance Hadoop and distributed systems. It covers modern configuration practices for swappiness, transparent huge pages, overcommit behavior, socket and port tuning, file descriptor limits, disk behavior, and DNS resolution. Legacy options are included where still relevant, with updated recommendations for modern kernels and systemd-based Linux distributions. Running Hadoop or any large distributed system at scale requires more than good cluster design. Performance and stability depend heavily on the underlying Linux configuration. This guide revisits the classic Hadoop tuning principles from a modern 2025 perspective, explains what still matters, and documents what has changed in recent kernel versions. These tuning practices apply not just to Hadoop, but also to Kafka, HBase, Zookeeper, Flink, object storage gateways, and high-ingest distributed systems wh...

HiveServer2 replaced the old Hive CLI because the CLI bypassed all security and governance layers, could not support multi-user concurrency, and created operational risks that modern data platforms cannot tolerate. This updated version explains the historical context, what changed in today’s Hadoop and Hive environments, and why Beeline and JDBC remain the only correct way to access Hive securely and predictably. When Hive 0.11 introduced HiveServer2 (HS2), it marked a necessary break with the legacy Hive CLI model. While the original post explained this transition for early Hadoop distributions, the underlying reasons remain valid even in modern Hive deployments. Today Hive CLI is effectively obsolete, and all secure or governed environments require HS2 as the mandatory entry point. Why the Hive CLI Had to Die 1. The CLI Bypassed All Security The original Hive CLI talked directly to the Hive Metastore and launched MapReduce or Tez jobs without going through a controlled ser...

HBase cross-cluster replication provides asynchronous disaster recovery by streaming WAL edits from a source cluster to one or more destination clusters. It is not a high-availability solution—applications must handle failover logic. This updated guide explains replication modes, requirements, configuration steps and operational considerations for modern HBase deployments. HBase offers built-in multi-site replication for disaster recovery (DR). Replication streams write-ahead log (WAL) edits from one cluster to another cluster or a set of clusters. Because replication is asynchronous, it does not provide automatic failover or zero-data-loss guarantees; applications must handle HA logic at the architectural level. Replication Topologies Modern HBase supports several replication patterns: Master → Slave : a primary cluster replicates edits to one or more secondary clusters. Simple and widely used for DR. Master ↔ Master : two clusters replicate to each other. HBase pre...

This quick tip shows how to list all Hive tables in a database together with their HDFS locations and human-readable sizes using a single bash one-liner. It still works on classic Hive CLI setups and can be adapted easily for Beeline or modern Hive deployments. When you run benchmarks, clean up old data or just want to understand how much space each Hive table consumes, it is useful to see HDFS locations and sizes side by side. Instead of clicking through UIs, you can ask Hive for every table location and then call hdfs dfs -du -h on each path. The Hive + HDFS one-liner The following bash one-liner queries Hive for table locations, extracts the HDFS paths and then prints a human-readable size for each table directory: for file in $(hive -S -e "SHOW TABLE EXTENDED LIKE '\*'" \ | grep "location:" \ | awk 'BEGIN { FS=":" } { printf("hdfs:%s:%s\n",$3,$4) }'); do hdfs dfs -du -h "$file" done Typical outp...

This is a legacy but still useful walkthrough that shows how to expose HBase-resident data to Impala by going through Hive’s Metastore and the HBaseStorageHandler. Using US census ZIP code income data, we create an HBase table, map it with an external Hive table, bulk load the CSV data with Pig and finally query it from Impala. The pattern is mainly relevant today if you are keeping old CDH clusters alive or planning a migration away from Impala-on-HBase towards Parquet or Iceberg tables. Note (2025): This article describes an older CDH/Impala/HBase pattern based on Hive’s HBaseStorageHandler . It is useful if you still maintain legacy Impala-on-HBase workloads or need to understand how such systems were wired. For new designs you will usually land data in Parquet or Iceberg tables and query them with Impala, Trino or Spark instead of reading directly from HBase. Context: Impala, Hive Metastore and HBase Impala uses the Hive Metastore Service to discover tables and their un...

Major compactions in HBase can be scheduled during low-traffic hours to reduce load on RegionServers. This guide shows how to trigger a compaction from the HBase shell using a simple Ruby script and how to wrap it in a Kerberos-aware cron job. It reflects common operational practice in legacy Hadoop clusters where maintenance windows still matter. Why Schedule Major Compactions? Major compactions rewrite all store files of an HBase table, improving read performance but putting additional pressure on the cluster. Many administrators run them during off-peak windows. HBase itself does not provide built-in scheduling, so automation is typically handled with cron or at . Ruby Script for HBase Shell HBase shell executes commands through JRuby, so a simple script triggers the compaction: # m_compact.rb major_compact 't1' exit Cron-Compatible Shell Wrapper Below is an example daily_compact script that refreshes a Kerberos ticket and runs the compaction via the HBase...

Snappy is a fast compression codec widely used in Hadoop ecosystems. This updated guide shows how to generate data, upload it to HDFS, process it with Hive using Snappy compression, verify the output files, and load Snappy-compressed data back into Hive using modern Hadoop commands. Snappy is a high-performance compression and decompression library originally developed at Google. It is optimized for speed rather than maximum compression ratio, making it a preferred codec in many Hadoop and Hive pipelines. Snappy is integrated in all modern Hadoop distributions (Hadoop 2.x and 3.x) and works with Hive, MapReduce, and increasingly Tez or Spark-based Hive deployments. 1. Create sample input data Generate a small test file: $ seq 1 1000 | awk '{OFS="\001";print $1, $1 % 10}' > test_input.hive $ head -5 test_input.hive 1^A1 2^A2 3^A3 4^A4 5^A5 2. Upload the data into HDFS Updated command: Hadoop now uses hdfs dfs instead of hadoop dfs . $ hdfs dfs ...

This guide combines legacy experience from early Hadoop deployments with modern best practices to harden clusters using Active Directory, Kerberos, and SELinux. It explains how to integrate Linux hosts with AD, run MapReduce tasks under real user identities, and enforce OS-level controls with SELinux, while highlighting where today’s YARN-based stacks and tools differ from the classic JobTracker and LinuxTaskController era. Securing a Hadoop cluster has always been more than just flipping a switch. It touches authentication, operating system hardening, networking, and the way jobs are executed. This article merges and modernizes two older posts from 2011, walking through: Integrating Hadoop nodes with a Windows Active Directory forest using Kerberos and LDAP Running MapReduce tasks under real user identities (legacy LinuxTaskController) Hardening nodes with SELinux in targeted or strict mode and building custom policies How these patterns map to modern Hadoop (YARN, ...

This guide updates an older 2011 post on using Microsoft’s SQL Server–Hadoop Connector with Sqoop. It explains how the legacy connector worked, how to configure Sqoop with the current Microsoft JDBC driver for SQL Server, and what to consider now that Apache Sqoop is retired and modern data stacks often use other ingestion tools. Back in 2011, Microsoft shipped a dedicated SQL Server–Hadoop Connector that plugged into Apache Sqoop as an additional connector package. It automated import and export between SQL Server, HDFS, and Hive, and required a separate download plus the Microsoft JDBC driver. Today, the situation is different: Apache Sqoop is retired and has been moved to the Apache Attic (June 2021). It still works but is no longer actively developed or recommended for new projects.:contentReference[oaicite:0]{index=0} The original SQL Server–Hadoop Connector tarball is effectively legacy; most distributions simply use Sqoop’s generic JDBC support plus the Micros...

This guide explains how to collect IIS W3C logs from Windows servers, ingest them into HDFS, define Hive tables for structured analysis, and run partitioned queries that calculate traffic per IP. The workflow reflects an early big-data pattern: mount, copy, structure, partition and analyze. The refreshed version clarifies each step and highlights how automated partition creation made near-real-time traffic analytics possible. Apache-driven webfarms have always been easy to integrate into Hadoop and Hive, but many teams struggled with the same question: How do we collect and analyze IIS logs at scale? This walkthrough shows how IIS W3C logs can be exported, ingested into HDFS, structured in Hive, and processed into partitioned traffic tables for efficient reporting. Configuring IIS for W3C Logging Inside IIS Manager, under a website’s configuration, choose: Logging → Format: W3C From there you can configure: Log output path Fields to capture (IP, URI, user agent...

Sqoop was the standard tool for moving data between relational databases and Hadoop. One of its most useful capabilities was generating a custom job JAR for optimizing export performance. This guide explains how to create the JAR, inspect the generated classes and rerun Sqoop with your precompiled job code to achieve faster, more stable export pipelines. Apache Sqoop (SQL-to-Hadoop) bridged traditional databases and Hadoop ecosystems. A lesser-known feature allowed developers to generate a standalone job JAR directly from an export command, enabling performance tuning and customizations. Generating a Sqoop Export Job JAR Example export command that produces a JAR file: sqoop export \ --connect jdbc:RDBMS:thin:@HOSTNAME:PORT:DBNAME \ --table TABLENAME \ --username USERNAME \ --password PASSWORD \ --export-dir HDFS_DIR \ --direct \ --fields-terminated-by ',' \ --package-name JOBNAME.IDENTIFIER \ --outdir OUTPUT_DIR \ --bindir BIN_DIR After ru...

Hive’s embedded Derby database is fine for local testing, but it breaks down as soon as multiple users and services need to share metadata. This guide shows how to move Hive from the default single-user Derby setup to a shared MySQL metastore: configuring MySQL, creating the Hive schema, wiring Hive to the external database, and distributing drivers and configuration across a Hadoop cluster. Apache Hive provides a SQL-like query language (HiveQL) on top of HDFS, making large-scale data analysis accessible to anyone with SQL experience. By default, however, Hive uses an embedded Derby database for its metastore, which is not suited for multi-user or multi-service environments. To run Hive in a real cluster, you need an external metastore database . The database stores metadata about: Databases and tables Partitions and storage descriptors SerDes, column definitions and privileges This article walks through configuring a MySQL-based Hive metastore , suitable for...