Skip to main content

Secure your hadoop cluster, Part II

Listen:
To get absolutely safe you need a bit more time, coffee and Aspirin. You will get headaches, for sure. First the good news, hadoop and the ecosystem run out of the box with an enabled SELinux system in targeting mode. You have to consider a performance loss of 5 - 10%.

To enable SELiux on a box use setenforce 1, to check the system use sestatus.
# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          enforcing
Policy version:                 21
Policy from config file:        targeted

Fine. Thats all. Now we enable SELinux at boot time:
# cat /etc/selinux/config
SELINUX=enforcing
SELINUXTYPE=targeted
SETLOCALDEFS=0

If you use fuse-hdfs check [1] for a valid rule.

The best way to get a system running is always to use SELINUXTYPE=targeted. But in some environments it is neccessary to protect the systems much more (Healthcare, Bank, Military etc.), here we use strict mode. The difference between targeted and strict is small, but really important - strict protects the whole system and uses the given ruleset, targeted protects only the services in the given policy.

To get the strict mode running you have to install the policies first, I did yum install selinux-policy*. Now you have to set your system into the permissive mode

# cat /etc/selinux/config
SELINUX=permissive
SELINUXTYPE=strict
SETLOCALDEFS=0

Then touch /.autorelabel and reboot. If you get into a kernel panic a boot time cancel boot (hitting Space), and append at the grub boot-line selinux=0 and boot.

Cross your fingers, get a large cup of coffee and watch the boot screen. The permissive mode lets boot the system and only reports all violations, but allows us to access the system. After a successfull reboot check dmesg for errors. Now we get a lot of AVC messages, including the hadoop subsystem. Use the system as you useally use. With audit2allow you have the availibity to create own policies, the simplest way is audit2allow -a (see above).

Activate the strict-mode and set SELinux into enforcing:

# setenforce 1
# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          permissive
Policy version:                 21
Policy from config file:        strict

Now you have no access to the system, except your home and /tmp. To set the system into the permissive mode you have to get the role for:
# newrole -r sysadm_r
Password:

now you are able to administer the system including setting back into the permissive mode with setenforce 0.

Now its time to get hdfs running. It is quite more easier then you think, because hadoop is developed in java. So we have to enable java first:

# grep java_exec_t /var/log/audit/audit.log | audit2allow -m hdfs > hdfs.te
# cat hdfs.te
module hdfs 1.0;


require {
type sysadm_su_t;
type java_exec_t;
class file execute;
}


#============= sysadm_su_t ==============
allow sysadm_su_t java_exec_t:file execute;

Now we build a custom module:
# grep java_exec_t /var/log/audit/audit.log | audit2allow -M hdfs
******************** IMPORTANT ***********************
To make this policy package active, execute:
semodule -i hdfs.pp


# semodule -i hdfs.pp

Check if the module was loaded correctly:
# semodule -l | grep hdfs
hdfs 1.0

enable the ports:
# semanage port -a -t java_port_t -p tcp 9000 (9000 as an example, check [2] for default ports)

You see, to harden a hadoop cluster take a bit more time, but it is possible. You get a robust and security enhanced system, but have to pay for with minimal performance loss.


[1] http://mail-archives.apache.org/mod_mbox/hadoop-general/201001.mbox/%3C4B436D53.2030406@hep.caltech.edu%3E
[2] http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/

Comments

Popular posts from this blog

Deal with corrupted messages in Apache Kafka

Under some strange circumstances, it can happen that a message in a Kafka topic is corrupted. This often happens when using 3rd party frameworks with Kafka. In addition, Kafka < 0.9 does not have a lock on Log.read() at the consumer read level, but does have a lock on Log.write(). This can lead to a rare race condition as described in KAKFA-2477 [1]. A likely log entry looks like this: ERROR Error processing message, stopping consumer: (kafka.tools.ConsoleConsumer$) kafka.message.InvalidMessageException: Message is corrupt (stored crc = xxxxxxxxxx, computed crc = yyyyyyyyyy Kafka-Tools Kafka stores the offset of each consumer in Zookeeper. To read the offsets, Kafka provides handy tools [2]. But you can also use zkCli.sh, at least to display the consumer and the stored offsets. First we need to find the consumer for a topic (> Kafka 0.9): bin/kafka-consumer-groups.sh --zookeeper management01:2181 --describe --group test Prior to Kafka 0.9, the only way to get this inform

Hive query shows ERROR "too many counters"

A hive job face the odd " Too many counters:"  like Ended Job = job_xxxxxx with exception 'org.apache.hadoop.mapreduce.counters.LimitExceededException(Too many counters: 201 max=200)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask Intercepting System.exit(1) These happens when operators are used in queries ( Hive Operators ). Hive creates 4 counters per operator, max upto 1000, plus a few additional counters like file read/write, partitions and tables. Hence the number of counter required is going to be dependent upon the query.  To avoid such exception, configure " mapreduce.job.counters.max " in mapreduce-site.xml to a value above 1000. Hive will fail when he is hitting the 1k counts, but other MR jobs not. A number around 1120 should be a good choice. Using " EXPLAIN EXTENDED " and " grep -ri operators | wc -l " print out the used numbers of operators. Use this value to tweak the MR s

AI's False Reality: Understanding Hallucination

Artificial Intelligence (AI) has leapfrogged to the poster child of technological innovation, on track to transform industries in a scale similar to the Industrial Revolution of the 1800s. But in this case, as cutting-edge technology, AI presents its own unique challenge, exploiting our human behavior of "love to trust", we as humans face a challenge: AI hallucinations. This phenomenon, where AI models generate outputs that are factually incorrect, misleading, or entirely fabricated, raises complex questions about the reliability and trust of AI models and larger systems. The tendency for AI to hallucinate comes from several interrelated factors. Overfitting – a condition where models become overly specialized to their training data – can lead to confident but wildly inaccurate responses when presented with novel scenarios (Guo et al., 2017). Moreover, biases embedded within datasets shape the models' understanding of the world; if these datasets are flawed or unreprese