Skip to main content

Setting Up MIT Kerberos ↔ Active Directory Cross-Realm Trust for Secure Hadoop Clusters

By Anonymous
This post explains how to configure a secure cross-realm Kerberos trust between a MIT KDC and Active Directory for Hadoop environments. It covers modern Kerberos settings, realm definitions, encryption choices, KDC configuration, AD trust creation, and Hadoop’s auth_to_local mapping rules. A final section preserves legacy compatibility for older Windows Server versions, ensuring the article can be used across mixed enterprise environments.

Integrating Hadoop with enterprise identity systems often requires establishing a cross-realm Kerberos trust between a local MIT KDC and an Active Directory (AD) domain. This setup allows Hadoop services to authenticate users from AD while maintaining a separate Hadoop-managed realm.

We walk through a full MIT Kerberos ↔ AD trust configuration using a modern setup, while preserving legacy notes for older Windows environments still found in long-lived clusters.

Example Realms

Replace these with your actual realms and hosts:

ALO.LOCAL      → Local MIT Kerberos realm (Hadoop KDC)
HADOOP1.INTERNAL → Host running the MIT KDC
AD.REMOTE     → Active Directory realm (external domain)

The KDC should be located within the Hadoop network. AD may be remote as long as the two domains can network-route to each other on Kerberos ports (88/UDP+TCP and 749/TCP).

1. Install Required Packages

On the MIT KDC (RHEL, CentOS, AlmaLinux, Rocky, etc.)

yum install krb5-server krb5-libs krb5-workstation -y

On Hadoop nodes (clients)

yum install krb5-libs krb5-workstation -y

Install Java JCE unlimited strength policy if required by your JDK distribution.

2. Configure the MIT KDC

/etc/krb5.conf

[libdefaults]
 default_realm = ALO.LOCAL
 dns_lookup_realm = false
 dns_lookup_kdc = false
 forwardable = true
 proxiable = true
 default_tgs_enctypes = aes256-cts aes128-cts rc4-hmac
 default_tkt_enctypes = aes256-cts aes128-cts rc4-hmac

[realms]
 ALO.LOCAL = {
   kdc = hadoop1.internal:88
   admin_server = hadoop1.internal:749
 }
 AD.REMOTE = {
   kdc = ad.remote.internal:88
   admin_server = ad.remote.internal:749
 }

[domain_realm]
 alo.local = ALO.LOCAL
 .alo.local = ALO.LOCAL
 ad.internal = AD.REMOTE
 .ad.internal = AD.REMOTE

[logging]
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmin.log
 default = FILE:/var/log/krb5lib.log

/var/kerberos/krb5kdc/kdc.conf

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 ALO.LOCAL = {
   acl_file = /var/kerberos/krb5kdc/kadm5.acl
   admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
   supported_enctypes = aes256-cts:normal aes128-cts:normal rc4-hmac:normal
 }

/var/kerberos/krb5kdc/kadm5.acl

*/admin@ALO.LOCAL *

Create the realm and start services

kdb5_util create -s -r ALO.LOCAL
service kadmin restart
service krb5kdc restart
chkconfig kadmin on
chkconfig krb5kdc on

Create admin principal

kadmin.local -q "addprinc root/admin"

3. Create the Trust on Active Directory (Modern Workflow)

Run the following from an elevated Windows PowerShell terminal:

# Register the MIT KDC
ksetup /addkdc ALO.LOCAL HADOOP1.INTERNAL

# Create the cross-realm trust
netdom trust ALO.LOCAL /domain:AD.REMOTE /add /realm /passwordt:passw0rd

# Set allowed encryption types
ksetup /SetEncTypeAttr ALO.LOCAL AES256-CTS-HMAC-SHA1-96 AES128-CTS-HMAC-SHA1-96 RC4-HMAC-MD5

After this, AD recognizes the MIT KDC as a trusted external Kerberos realm.

4. Create the AD Trust Principal in MIT Kerberos

kadmin.local: addprinc krbtgt/ALO.LOCAL@AD.REMOTE
password: passw0rd

5. Configure Hadoop’s auth_to_local Rules

core-site.xml

<property>
  <name>hadoop.security.auth_to_local</name>
  <value>
    RULE:[1:$1@$0](.*@\QAD.REMOTE\E$)s/@\QAD.AD.REMOTE\E$//
    RULE:[2:$1@$0](.*@\QAD.REMOTE\E$)s/@\QAD.REMOTE\E$//
    DEFAULT
  </value>
</property>

This strips the AD domain suffix, allowing Hadoop and HDFS to map users from AD to local Linux accounts or group mappings.

6. Test the Trust

kinit username@AD.REMOTE
klist

You should see a ticket-granting ticket (TGT) issued by AD.REMOTE.

Legacy Compatibility Notes

Some environments still operate old Windows Server versions. Use the following only if required:

Windows Server 2003

ktpass /princ ALO.LOCAL /domain AD.REMOTE /TrustEncryp aes128-cts arcfour-hmac

Windows Server 2008 additional step

ksetup /SetEncTypeAttr ALO.LOCAL aes128-cts arcfour-hmac

These modes support legacy encryption requirements and can still be found in long-lived enterprise clusters where upgrading the AD domain is not yet completed.

Conclusion

A cross-realm trust between MIT Kerberos and Active Directory remains a powerful and widely-used authentication setup for Hadoop clusters. The configuration provides secure integration with enterprise identity systems while allowing the Hadoop realm to remain isolated and controlled.

With proper encryption settings, realm mappings, and Hadoop auth_to_local rules, users from AD can authenticate seamlessly and work securely across the Hadoop environment.

Related guides:

If you need help with distributed systems, backend engineering, or data platforms, check my Services.

Labels:

Most read articles

Why Is Customer Obsession Disappearing?

By
Many companies trade real customer-obsession for automated, low-empathy support. Through examples from Coinbase, PayPal, GO Telecommunications and AT&T, this article shows how reliance on AI chatbots, outsourced call centers, and KPI-driven workflows erodes trust, NPS and customer retention. It argues that human-centric support—treating support as strategic investment instead of cost—is still a core growth engine in competitive markets. It's wild that even with all the cool tech we've got these days, like AI solving complex equations and doing business across time zones in a flash, so many companies are still struggling with the basics: taking care of their customers. The drama around Coinbase's customer support is a prime example of even tech giants messing up. And it's not just Coinbase — it's a big-picture issue for the whole industry. At some point, the idea of "customer obsession" got replaced with "customer automation," and no...
Read more

Building a Model-Agnostic Multi-Agent System with OpenClaw

By
Over one week we rebuilt our AI stack around OpenClaw’s multi-agent architecture to avoid provider lock-in and stop wasting premium tokens. By aligning models to tasks, diversifying fallbacks across providers, enforcing minimal tool access, and switching to memory-first workflows with ephemeral sessions, we reduced token usage per task by about 70% and cut our monthly bill by 77% while improving operational resilience. How We Achieved 77% Cost Reduction and Provider Independence Over the past week, we rebuilt our AI infrastructure around OpenClaw’s multi-agent architecture. The result was a 77% cost reduction , provider independence , and a delegation system that routes work to the most cost-effective model for each job. Below is the technical journey of optimizing a 7-agent squad with OpenClaw. The Challenge: Model Provider Lock-In We started with a simple problem: our entire squad defaulted to a single model provider. This created three issues: Cost inefficiency beca...
Read more

What are the performance implications of cross-platform execution within Wayang?

By
Apache Wayang ® enables cross-platform execution across multiple data processing platforms such as Spark, Flink, Java Streams, PostgreSQL or GraphChi. This capability fundamentally changes the performance behavior of distributed data pipelines. Wayang reduces manual data movement by selecting where each operator should run, but crossing platform boundaries still introduces serialization cost, shifts in locality, different memory strategies and new tuning constraints. Understanding these dynamics is essential before adopting Wayang for multi-platform pipelines at scale. Apache Wayang is a cross-platform data processing framework that lets developers run a single logical pipeline across engines such as Apache Spark, Apache Flink or a native Java backend. It provides an abstraction layer and a cost-based optimizer that selects the execution platform for each operator. This flexibility introduces new performance variables that do not exist in single-engine systems. Engine boundaries ...
Read more