EMR 005 Enforcing TLSv1.2 for HDFS, YARN, MAPREDUCE2 - qyjohn/AWS_Tutorials GitHub Wiki

(0) Disable the algorithms that we do not want in /usr/lib/jvm/java-1.8.0/jre/lib/security/java.security

jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, DES40_CBC, RC4_40, 3DES_EDE_CBC, TLSv1, TLSv1.1

(1) Create a folder to store the keys

sudo mkdir /keys
sudo chown -R hadoop:hadoop /keys
cd /keys

(2) Generate certificates. In this procedure, we use "hadoop" when prompted for a password. You can of course use your own password.

keytool -keystore keystore.jks -alias localhost -validity 365 -genkey
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
keytool -keystore keystore.jks -alias CARoot -import -file ca-cert 
keytool -keystore keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:hadoop
keytool -keystore keystore.jks -alias localhost -import -file cert-signed

(3) Add the following to hdfs-site.xml

  <property>
    <name>dfs.http.policy</name>
    <value>HTTPS_ONLY</value>
  </property>
  <property>
    <name>dfs.https.enable</name>
    <value>true</value>
  </property>

(4) Add the following to ssl-server.xml

<property>
<name>ssl.server.keystore.type</name>
<value>jks</value>
</property>

<property>
<name>ssl.server.keystore.password</name>
<value>hadoop</value>
</property>
<property>
<name>ssl.server.keystore.keypassword</name>
<value>hadoop</value>
</property>

<property>
<name>ssl.server.keystore.location</name>
<value>/keys/keystore.jks</value>
</property>

<property>
<name>ssl.server.truststore.type</name>
<value>jks</value>
</property>

<property>
<name>ssl.server.truststore.location</name>
<value>/keys/keystore.jks</value>
</property>

<property>
<name>ssl.server.truststore.password</name>
<value>hadoop</value>
</property>

(5) Add the following to ssl-client.xml

<property>
<name>ssl.client.truststore.password</name>
<value>hadoop</value>
</property>

<property>
<name>ssl.client.truststore.type</name>
<value>jks</value>
</property>

<property>
<name>ssl.client.truststore.location</name>
<value>/keys/keystore.jks</value>
</property>

(6) Stop and start namenode and datanode

sudo stop hadoop-hdfs-datanodeopen
sudo stop hadoop-hdfs-namenode
sudo start hadoop-hdfs-namenode
sudo start hadoop-hdfs-datanode

(7) At this point we should be able to verify that namenode is working using TLSv1.2 using the following command:

openssl s_client -connect <ip-address>:50470 -tls1_1
openssl s_client -connect <ip-address>:50470 -tls1_1

(8) Add the following to mapred-site.xml

  <property>
    <name>mapreduce.jobhistory.http.policy</name>
    <value>HTTPS_ONLY</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.https.address</name>
    <value><master node of the EMR cluster>:19889</value>
  </property>
  <property>
    <name>mapreduce.ssl.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>mapreduce.shuffle.ssl.enabled</name>
    <value>true</value>
  </property>

(9) Add the following to yarn-site.xml

  <property>
    <name>yarn.http.policy</name>
    <value>HTTPS_ONLY</value>
  </property>
  <property>
    <name>yarn.log.server.url</name>
    <value>https://<master node of the EMR cluster>:19889/jobhistory/logs</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.https.address</name>
    <value><master node of the EMR cluster>:8090</value>
  </property>
  <property>
    <name>yarn.nodemanager.webapp.https.address</name>
    <value><master node of the EMR cluster>:8042</value>
  </property>

(10) Restart services

sudo stop hadoop-yarn-resourcemanager
sudo start hadoop-yarn-resourcemanager
sudo stop hadoop-yarn-nodemanager
sudo start hadoop-yarn-rnodemanager
sudo stop hadoop-mapreduce-historyserver
sudo start hadoop-mapreduce-historyserver

(11) Test

node manager
openssl s_client -connect ip-172-31-18-204.ap-southeast-2.compute.internal:8042 -tls1_1
openssl s_client -connect ip-172-31-18-204.ap-southeast-2.compute.internal:8042 -tls1_2

yarn resource manager
openssl s_client -connect ip-172-31-18-204.ap-southeast-2.compute.internal:8090 -tls1_1
openssl s_client -connect ip-172-31-18-204.ap-southeast-2.compute.internal:8090 -tls1_2

job history server
openssl s_client -connect ip-172-31-18-204.ap-southeast-2.compute.internal:19889 -tls1_1
openssl s_client -connect ip-172-31-18-204.ap-southeast-2.compute.internal:19889 -tls1_2
⚠️ **GitHub.com Fallback** ⚠️