Secure your hadoop cluster, Part II

To get absolutely safe you need a bit more time, coffee and Aspirin. You will get headaches, for sure. First the good news, hadoop and the ecosystem run out of the box with an enabled SELinux system in targeting mode. You have to consider a performance loss of 5 - 10%.

To enable SELiux on a box use setenforce 1, to check the system use sestatus.
# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          enforcing
Policy version:                 21
Policy from config file:        targeted

Fine. Thats all. Now we enable SELinux at boot time:
# cat /etc/selinux/config
SELINUX=enforcing
SELINUXTYPE=targeted
SETLOCALDEFS=0

If you use fuse-hdfs check [1] for a valid rule.

The best way to get a system running is always to use SELINUXTYPE=targeted. But in some environments it is neccessary to protect the systems much more (Healthcare, Bank, Military etc.), here we use strict mode. The difference between targeted and strict is small, but really important - strict protects the whole system and uses the given ruleset, targeted protects only the services in the given policy.

To get the strict mode running you have to install the policies first, I did yum install selinux-policy*. Now you have to set your system into the permissive mode

# cat /etc/selinux/config
SELINUX=permissive
SELINUXTYPE=strict
SETLOCALDEFS=0

Then touch /.autorelabel and reboot. If you get into a kernel panic a boot time cancel boot (hitting Space), and append at the grub boot-line selinux=0 and boot.

Cross your fingers, get a large cup of coffee and watch the boot screen. The permissive mode lets boot the system and only reports all violations, but allows us to access the system. After a successfull reboot check dmesg for errors. Now we get a lot of AVC messages, including the hadoop subsystem. Use the system as you useally use. With audit2allow you have the availibity to create own policies, the simplest way is audit2allow -a (see above).

Activate the strict-mode and set SELinux into enforcing:

# setenforce 1
# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          permissive
Policy version:                 21
Policy from config file:        strict

Now you have no access to the system, except your home and /tmp. To set the system into the permissive mode you have to get the role for:
# newrole -r sysadm_r
Password:

now you are able to administer the system including setting back into the permissive mode with setenforce 0.

Now its time to get hdfs running. It is quite more easier then you think, because hadoop is developed in java. So we have to enable java first:

# grep java_exec_t /var/log/audit/audit.log | audit2allow -m hdfs > hdfs.te
# cat hdfs.te
module hdfs 1.0;


require {
type sysadm_su_t;
type java_exec_t;
class file execute;
}


#============= sysadm_su_t ==============
allow sysadm_su_t java_exec_t:file execute;

Now we build a custom module:
# grep java_exec_t /var/log/audit/audit.log | audit2allow -M hdfs
******************** IMPORTANT ***********************
To make this policy package active, execute:
semodule -i hdfs.pp


# semodule -i hdfs.pp

Check if the module was loaded correctly:
# semodule -l | grep hdfs
hdfs 1.0

enable the ports:
# semanage port -a -t java_port_t -p tcp 9000 (9000 as an example, check [2] for default ports)

You see, to harden a hadoop cluster take a bit more time, but it is possible. You get a robust and security enhanced system, but have to pay for with minimal performance loss.


[1] http://mail-archives.apache.org/mod_mbox/hadoop-general/201001.mbox/%3C4B436D53.2030406@hep.caltech.edu%3E
[2] http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/