Hardening Hadoop Clusters with Active Directory, Kerberos, and SELinux
Securing a Hadoop cluster has always been more than just flipping a switch. It touches authentication, operating system hardening, networking, and the way jobs are executed. This article merges and modernizes two older posts from 2011, walking through:
- Integrating Hadoop nodes with a Windows Active Directory forest using Kerberos and LDAP
- Running MapReduce tasks under real user identities (legacy LinuxTaskController)
- Hardening nodes with SELinux in targeted or strict mode and building custom policies
- A lightweight MIT Kerberos setup for small clusters without AD
- Logging in to Kerberized web UIs with SPNEGO
- How these patterns map to modern Hadoop (YARN, SSSD, realmd, secure clusters)
The examples come from RHEL 5 and classic MRv1 clusters, but the principles still inform today’s YARN-based and Kerberized deployments.
1. Integrating Hadoop Hosts with Active Directory
In many enterprises, the primary identity system is a Windows Active Directory forest. Hadoop nodes should authenticate users against the same source of truth to avoid local user sprawl and inconsistent permissions.
Historically, this was done by enabling “Identity Management for UNIX” in AD, extending the schema with UNIX attributes, and configuring Linux servers to use LDAP + Kerberos. On modern RHEL and derivatives, the same target is more commonly achieved with realmd and sssd, but the core ideas remain:
- AD provides users, groups, and Kerberos tickets.
- Linux resolves users/groups via LDAP (or SSSD-backed NSS).
- Hadoop daemons rely on OS-level identities and Kerberos for secure RPC.
1.1 Legacy RHEL 5 style configuration (LDAP + Kerberos)
On old RHEL systems, authconfig-tui was often used:
# authconfig-tui
Typical settings:
- Authentication → Use LDAP + Use shadow password + Use Kerberos
- Server → Base DN (e.g.
dc=hadoop,dc=company,dc=local) - Kerberos → REALM in uppercase, KDC/admin server set to AD FQDNs
Because AD does not allow anonymous binds, you configure a bind account in /etc/ldap.conf (in modern systems replaced by sssd.conf).
base dc=hadoop,dc=company,dc=local
uri ldap://hadoop.company.local/
binddn [email protected]
bindpw <PASSWORD>
scope sub
ssl no # modern setups should use ldaps or start_tls instead
nss_base_passwd dc=hadoop,dc=company,dc=local?sub
nss_base_shadow dc=hadoop,dc=company,dc=local?sub
nss_base_group dc=hadoop,dc=company,dc=local?sub? \
&(objectCategory=group)(gidnumber=*)
nss_map_objectclass posixAccount user
nss_map_objectclass shadowAccount user
nss_map_objectclass posixGroup group
nss_map_attribute gecos cn
nss_map_attribute homeDirectory unixHomeDirectory
nss_map_attribute uniqueMember member
nss_map_objectclass posixGroup Group
tls_cacertdir /etc/openldap/cacerts
pam_password md5
pam_login_attribute sAMAccountName
pam_filter objectclass=User
Name service configuration (/etc/nsswitch.conf) then includes LDAP:
passwd: files ldap
shadow: files ldap
group: files ldap
Tests:
# getent passwd
# kinit <USERNAME>@HADOOP.COMPANY.LOCAL
If both succeed, the Linux host sees AD users and can obtain Kerberos tickets.
1.2 Home directories via PAM
To provision home directories automatically on login, add to /etc/pam.d/system-auth:
session required pam_mkhomedir.so skel=/etc/skel umask=0022
This is still a common pattern today, though often driven via sssd instead of legacy ldap.conf.
1.3 Modern note on realmd + SSSD
On current RHEL/CentOS/Rocky/Alma Linux releases, the preferred way to join AD is:
# realm join hadoop.company.local -U adminuser
This configures Kerberos, SSSD, and NSS in a single step and avoids hand-editing ldap.conf. The security posture is better (TLS, GSSAPI, sane defaults).
2. Running MapReduce as Real Users (Legacy LinuxTaskController)
In classic MRv1, jobs were often run under a generic service user, which made auditing and isolation difficult. The LinuxTaskController allowed the TaskTracker to launch tasks under the submitting user’s Unix identity.
Configuration in mapred-site.xml looked like this:
<property>
<name>mapred.task.tracker.task-controller</name>
<value>org.apache.hadoop.mapred.LinuxTaskController</value>
</property>
<property>
<name>mapreduce.tasktracker.group</name>
<value>hdfs</value>
</property>
With this in place, jobs submitted by users authenticated via PAM/AD would execute as those users on the TaskTracker, making permissions and auditing align with corporate identity.
The modern equivalent, YARN container-executor
In Hadoop 2.x+ with YARN, the same idea is implemented by the LinuxContainerExecutor and YARN’s secure container execution:
- Use Kerberos for cluster-wide authentication.
- Configure
yarn.nodemanager.container-executor.classappropriately. - Ensure NodeManager runs with a service user and launches containers under the submitting user identity.
If you are running a modern secure cluster, follow the official distro guides for “secure YARN” and Kerberos setup rather than legacy LinuxTaskController documentation.
3. Hardening Hadoop Nodes with SELinux
Beyond authentication and user mapping, the OS itself should enforce mandatory access controls. SELinux adds an additional security layer around Hadoop daemons and supporting services.
3.1 Targeted mode (recommended baseline)
Hadoop and its ecosystem components generally run out of the box with SELinux in targeted mode, which protects selected services. On older RHEL systems:
# setenforce 1
# sestatus
SELinux status: enabled
SELinuxfs mount: /selinux
Current mode: enforcing
Policy from config file: targeted
Persist across reboots by editing /etc/selinux/config:
SELINUX=enforcing
SELINUXTYPE=targeted
SETLOCALDEFS=0
Expect a small performance overhead (5-10%), which is usually acceptable given the security gains.
3.2 Strict mode (high security environments)
For sectors with elevated requirements (healthcare, finance, government), SELinux strict mode can be used. This protects the entire system rather than only selected services, but is significantly more complex to operate.
Legacy configuration example:
# yum install selinux-policy*
# vi /etc/selinux/config
SELINUX=permissive
SELINUXTYPE=strict
SETLOCALDEFS=0
# touch /.autorelabel
# reboot
After a full relabel, the system boots in permissive strict mode and logs violations instead of blocking them. Check:
# dmesg | grep AVC
# cat /var/log/audit/audit.log
Once the required policies are refined, you can move to enforcing mode:
# setenforce 1
# sestatus
Current mode: enforcing
Policy from config file: strict
3.3 Building custom SELinux modules for Hadoop/Java
If SELinux blocks Java/Hadoop operations, you can generate minimal policy modules from audit logs:
# grep java_exec_t /var/log/audit/audit.log | audit2allow -m hdfs > hdfs.te
# cat hdfs.te
module hdfs 1.0;
require {
type sysadm_su_t;
type java_exec_t;
class file execute;
}
#============= sysadm_su_t ==============
allow sysadm_su_t java_exec_t:file execute;
Then build and load the policy:
# grep java_exec_t /var/log/audit/audit.log | audit2allow -M hdfs
# semodule -i hdfs.pp
# semodule -l | grep hdfs
hdfs 1.0
Finally, ensure Hadoop’s network ports are correctly labeled (example for port 9000; adjust for your environment):
# semanage port -a -t java_port_t -p tcp 9000
This approach lets you keep SELinux enforcing while allowing only the minimum set of permissions Hadoop needs.
3.4 Modern SELinux perspective
On modern distributions:
- Targeted mode is typically sufficient and much easier to operate.
- Use vendor-provided SELinux policies for Hadoop if available.
- Reserve strict mode for environments with strong SELinux expertise.
- Use
audit2allowvery carefully; review generated rules before loading them.
4. A Lightweight MIT Kerberos Setup for Small Clusters
Not every cluster needs AD. For a small, lab or isolated cluster, a small MIT Kerberos KDC with only the Hadoop service principals is enough. For multi-realm trust and production setups, see Hadoop and Trusted MITv5 Kerberos.
4.1 Install and configure the KDC
Install the Kerberos server packages and set your realm in krb5.conf. Replace ALO.ALT with your own realm:
[libdefaults]
default_realm = ALO.ALT
dns_lookup_realm = false
dns_lookup_kdc = false
[realms]
ALO.ALT = {
kdc = HADOOP1.ALO.ALT:88
admin_server = HADOOP1.ALO.ALT:749
default_domain = ALO.ALT
}
[domain_realm]
.alo.alt = ALO.ALT
alo.alt = ALO.ALT
[logging]
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmin.log
default = FILE:/var/log/krb5lib.log
Keep DNS or /etc/hosts consistent on all nodes:
192.168.56.101 hadoop1.alo.alt hadoop1
172.22.2.130 hadoop2.alo.alt hadoop2
Allow admin principals, then create the realm:
# cat /var/kerberos/krb5kdc/kadm5.acl
*/[email protected] *
kdb5_util create -s
systemctl start krb5kdc kadmin
4.2 Add service principals and export keytabs
On the KDC host, use kadmin.local to create principals for HDFS, MapReduce, YARN, HBase, HTTP (SPNEGO) and your user:
addprinc -randkey hdfs/[email protected]
addprinc -randkey mapred/[email protected]
addprinc -randkey yarn/[email protected]
addprinc -randkey hbase/[email protected]
addprinc -randkey HTTP/[email protected]
addprinc <USERNAME>@ALO.ALT
Your user principal needs a password. Test it:
su - <USERNAME>
kinit
Export the keytabs and lock down their permissions:
xst -norandkey -k hdfs.keytab hdfs/[email protected] HTTP/[email protected]
xst -norandkey -k mapred.keytab mapred/[email protected] HTTP/[email protected]
xst -norandkey -k yarn.keytab yarn/[email protected] HTTP/[email protected]
chown hdfs:hadoop hdfs.keytab && chmod 400 hdfs.keytab
chown mapred:hadoop mapred.keytab && chmod 400 mapred.keytab
chown yarn:hadoop yarn.keytab && chmod 400 yarn.keytab
4.3 Turn on Kerberos in HDFS
In hdfs-site.xml:
dfs.block.access.token.enable = true
dfs.namenode.keytab.file = <PATH>/hdfs.keytab
dfs.namenode.kerberos.principal = hdfs/[email protected]
dfs.namenode.kerberos.internal.spnego.principal = HTTP/[email protected]
dfs.secondary.namenode.keytab.file = <PATH>/hdfs.keytab
dfs.secondary.namenode.kerberos.principal = hdfs/[email protected]
dfs.datanode.keytab.file = <PATH>/hdfs.keytab
dfs.datanode.kerberos.principal = hdfs/[email protected]
dfs.web.authentication.kerberos.principal = HTTP/[email protected]
dfs.web.authentication.kerberos.keytab = <PATH>/hdfs.keytab
Start the NameNode, check access, and fix the permissions on /tmp:
hadoop fs -ls /
sudo -u hdfs kinit -k -t hdfs.keytab hdfs/[email protected]
sudo -u hdfs hadoop fs -chmod 1777 /tmp
4.4 Turn on Kerberos in MapReduce
In mapred-site.xml:
mapreduce.jobtracker.kerberos.principal = mapred/[email protected]
mapreduce.jobtracker.keytab.file = <PATH>/mapred.keytab
mapreduce.tasktracker.kerberos.principal = mapred/[email protected]
mapreduce.tasktracker.keytab.file = <PATH>/mapred.keytab
And make sure the TaskTracker uses the right group and directories:
# /etc/hadoop/conf/taskcontroller.cfg
hadoop.log.dir=/var/log/hadoop-mapreduce/
mapred.local.dir=/opt/hadoop/hdfs/mapred/local
mapreduce.tasktracker.group=mapred
banned.users=mapred,hdfs,bin
min.user.id=500
Restart the daemons, run kinit, and submit a simple job such as the PI example to check that secure submission works end to end. Use klist to check your ticket.
5. Accessing Kerberized Web UIs with SPNEGO
Once the cluster is Kerberized, the web UIs (NameNode on port 9870, ResourceManager on 8088, Oozie on 11000 and so on) expect the browser to log in with Kerberos through SPNEGO. If the browser is not set up, you get repeated login prompts or 401: Unauthorized.
5.1 Prerequisites
- A valid Kerberos ticket:
kinit [email protected] - Correct forward and reverse DNS for the Hadoop services
- A matching HTTP principal for the UI:
HTTP/hostname@REALM - A browser that allows SPNEGO for the target domain
5.2 Firefox
Open about:config and set:
- network.negotiate-auth.trusted-uris: domains where SPNEGO is allowed, for example
namenode.company.com, .company.com - network.negotiate-auth.delegation-uris: domains allowed for credential delegation, for example
.company.com - network.negotiate-auth.allow-non-fqdn:
falseunless you really need it
Firefox on Linux and macOS uses the system Kerberos libraries and reads /etc/krb5.conf.
5.3 Chrome and Edge
Modern Chrome and Edge use enterprise policies instead of command-line flags. On Linux, create:
/etc/opt/chrome/policies/managed/kerberos.json
with:
{
"AuthServerAllowlist": "*.company.com",
"AuthNegotiateDelegateAllowlist": "*.company.com"
}
On Windows, open the Group Policy Editor:
Computer Configuration →
Administrative Templates →
Google →
Google Chrome →
Authentication
and set AuthServerAllowlist and AuthNegotiateDelegateAllowlist to *.company.com. (Chrome 86 renamed these from the older AuthServerWhitelist and AuthNegotiateDelegateWhitelist. On macOS, deploy the same keys through a configuration profile.)
In old Internet Explorer setups, turn on Windows Integrated Authentication and add the Hadoop UI domain to the Local Intranet sites. Kerberos only negotiates automatically for the intranet zone.
5.4 When SPNEGO fails
- No valid ticket (run
kinit) - The browser does not trust the domain
- DNS does not match the Kerberos principal
- Clock skew between client and KDC
- The
HTTP/hostname@REALMprincipal is missing
Many modern clusters sit behind Apache Knox, an identity-aware proxy or SSO with SAML, OIDC or JWT. In that case you don’t need Kerberos in the browser at all.
6. Practical Considerations and Gotchas
- Time and DNS: Kerberos and AD integration are sensitive to clock skew and DNS misconfiguration. Always deploy NTP and verify forward/reverse DNS.
- Network filtering: Firewalls and some switches may misinterpret Hadoop’s RPC patterns; verify that all needed ports are open and not subject to “DOS protection” rules.
- Service separation: For both performance and security, avoid running NameNode, ResourceManager, and other heavy services all on the same small host.
- Audit trails: With AD/LDAP integration and SELinux, you get stronger auditability; ensure logs are centralized and retained.
Conclusion
Hardening a Hadoop cluster is a layered exercise: identity and authentication via AD and Kerberos, process isolation via Linux task execution or YARN containers, and OS-level enforcement via SELinux. While some of the configuration examples here come from legacy RHEL 5 and MRv1, the same core design principles apply to modern YARN-based, Kerberized clusters.
By combining centralized identities, secure job execution, and mandatory access control, you can move your data platform from “it works” to “it withstands scrutiny” in environments where security is not optional.
For further reading and historical context, see:
- Hadoop security and Kerberos integration guides from your distribution vendor
- SELinux documentation for RHEL / CentOS / derivatives
- Legacy discussions on FUSE-HDFS and SELinux rules: Apache Hadoop general mailing list (archive)
Need help with distributed systems, backend engineering or data platforms? See my consulting services.