Skip to main content

Secure your hadoop cluster, Part II

To get absolutely safe you need a bit more time, coffee and Aspirin. You will get headaches, for sure. First the good news, hadoop and the ecosystem run out of the box with an enabled SELinux system in targeting mode. You have to consider a performance loss of 5 - 10%.

To enable SELiux on a box use setenforce 1, to check the system use sestatus.
# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          enforcing
Policy version:                 21
Policy from config file:        targeted

Fine. Thats all. Now we enable SELinux at boot time:
# cat /etc/selinux/config

If you use fuse-hdfs check [1] for a valid rule.

The best way to get a system running is always to use SELINUXTYPE=targeted. But in some environments it is neccessary to protect the systems much more (Healthcare, Bank, Military etc.), here we use strict mode. The difference between targeted and strict is small, but really important - strict protects the whole system and uses the given ruleset, targeted protects only the services in the given policy.

To get the strict mode running you have to install the policies first, I did yum install selinux-policy*. Now you have to set your system into the permissive mode

# cat /etc/selinux/config

Then touch /.autorelabel and reboot. If you get into a kernel panic a boot time cancel boot (hitting Space), and append at the grub boot-line selinux=0 and boot.

Cross your fingers, get a large cup of coffee and watch the boot screen. The permissive mode lets boot the system and only reports all violations, but allows us to access the system. After a successfull reboot check dmesg for errors. Now we get a lot of AVC messages, including the hadoop subsystem. Use the system as you useally use. With audit2allow you have the availibity to create own policies, the simplest way is audit2allow -a (see above).

Activate the strict-mode and set SELinux into enforcing:

# setenforce 1
# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          permissive
Policy version:                 21
Policy from config file:        strict

Now you have no access to the system, except your home and /tmp. To set the system into the permissive mode you have to get the role for:
# newrole -r sysadm_r

now you are able to administer the system including setting back into the permissive mode with setenforce 0.

Now its time to get hdfs running. It is quite more easier then you think, because hadoop is developed in java. So we have to enable java first:

# grep java_exec_t /var/log/audit/audit.log | audit2allow -m hdfs > hdfs.te
# cat hdfs.te
module hdfs 1.0;

require {
type sysadm_su_t;
type java_exec_t;
class file execute;

#============= sysadm_su_t ==============
allow sysadm_su_t java_exec_t:file execute;

Now we build a custom module:
# grep java_exec_t /var/log/audit/audit.log | audit2allow -M hdfs
******************** IMPORTANT ***********************
To make this policy package active, execute:
semodule -i hdfs.pp

# semodule -i hdfs.pp

Check if the module was loaded correctly:
# semodule -l | grep hdfs
hdfs 1.0

enable the ports:
# semanage port -a -t java_port_t -p tcp 9000 (9000 as an example, check [2] for default ports)

You see, to harden a hadoop cluster take a bit more time, but it is possible. You get a robust and security enhanced system, but have to pay for with minimal performance loss.



Popular posts from this blog

Deal with corrupted messages in Apache Kafka

Under some strange circumstances it can happen that a message in a Kafka topic is corrupted. This happens often by using 3rd party frameworks together with Kafka. Additionally, Kafka < 0.9 has no lock at at the consumer read level, but has a lock on Log.write(). This can cause a rare race condition, as described in KAKFA-2477 [1]. Probably a log entry looks like: ERROR Error processing message, stopping consumer: ($) kafka.message.InvalidMessageException: Message is corrupt (stored crc = xxxxxxxxxx, computed crc = yyyyyyyyyy Kafka-Tools Kafka stores the offset of every consumer in Zookeeper. To read out the offsets, Kafka provides handy tools [2]. But also can be used, at least to display the consumer and the stored offsets. First we need to find the consumer for a topic (> Kafka 0.9): bin/ --zookeeper management01:2181 --describe --group test Prior to Kafka 0.9 the only possibility to get this inform

Hive query shows ERROR "too many counters"

A hive job face the odd " Too many counters:"  like Ended Job = job_xxxxxx with exception 'org.apache.hadoop.mapreduce.counters.LimitExceededException(Too many counters: 201 max=200)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask Intercepting System.exit(1) These happens when operators are used in queries ( Hive Operators ). Hive creates 4 counters per operator, max upto 1000, plus a few additional counters like file read/write, partitions and tables. Hence the number of counter required is going to be dependent upon the query.  To avoid such exception, configure " mapreduce.job.counters.max " in mapreduce-site.xml to a value above 1000. Hive will fail when he is hitting the 1k counts, but other MR jobs not. A number around 1120 should be a good choice. Using " EXPLAIN EXTENDED " and " grep -ri operators | wc -l " print out the used numbers of operators. Use this value to tweak the MR s

GPT & GenAI for Startup Storytelling

OpenAI and Bard   are the most used GenAI tools today; the first one has a massive Microsoft investment, and the other one is an experiment from Google. But did you know that you can also use them to optimize and hack your startup?  For startups, creating pitch scripts, sales emails, and elevator pitches with generative AI (GenAI) can help you not only save time but also validate your marketing and wording. Curious? Here are a few prompt hacks for startups to create,improve, and validate buyer personas, your startup's mission/vision statements, and unique selling proposition (USP) definitions. First Step: Introduce yourself and your startup Introduce yourself, your startup, your website, your idea, your position, and in a few words what you are doing to the chatbot: Prompt : I'm NAME and our startup NAME, with website URL, is doing WHATEVER. With PRODUCT NAME, we aim to change or disrupt INDUSTRY. Bard is able to pull information from your website. I'm not sure if ChatGPT