Assumption

• OS yang digunakan CentOS 7.x x86_64 atau el7 family yang lain(Redhat, CentOS, OracleLinux, etc). Untuk OS yang lain silahkan disesuaikan.
• Menggunakan 3 node:
  • yava00.solusi247.com (Master)
  • yava01.solusi247.com (Worker)
  • yava02.solusi247.com (Worker)

Prerequisite

• CentOS ISO Everything (~ 7 GB)

  Configure CentOS Local Repository

• YAVA local repo (YAVA, YAVA_UTILS, ambari)

  images/yava-repo.png

Preparation

• Password-less SSH

  [root@yava00 ~]# ssh-keygen -t rsa 
  [root@yava00 ~]# ssh-copy-id -i .ssh/id_rsa.pub root@ip-yava00
  [root@yava00 ~]# ssh-copy-id -i .ssh/id_rsa.pub root@ip-yava01
  [root@yava00 ~]# ssh-copy-id -i .ssh/id_rsa.pub root@ip-yava02
  

  Cek apakah sudah bisa autologin.

• Setting DNS

  • Untuk small cluster bisa langsung edit /etc/hosts

    [root@yava00 ~]# cat /etc/hosts
    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    
    #IP             FQDN            Alias
    10.1.80.160     yava00.solusi247.com    yava00
    10.1.80.161     yava01.solusi247.com    yava01
    10.1.80.162     yava02.solusi247.com    yava02
    

    Copy ke worker node

    [root@yava00 ~]# for i in `cat worker-nodes.txt`; do rsync -avz /etc/hosts root@${i}:/etc/hosts; done
    

    Cek baik forward maupun reverse

    [root@yava00 ~]# getent hosts yava01
    10.1.80.161     yava01.solusi247.com yava01
    [root@yava00 ~]# getent hosts 10.1.80.162
    10.1.80.162     yava02.solusi247.com yava02
    [root@yava00 ~]# getent hosts yava00.solusi247.com
    10.1.80.160     yava00.solusi247.com yava00
    

  • Untuk large cluster harus menggunakan DNS server seperti BIND atau dnsmasq

• Enable NSCD

  Gunakan di large cluster saja.

• Setting ulimit

  Tambah max untuk nproc dan nofiles menjadi 32000

  [root@yava00 ~]# cat /etc/security/limits.d/20-nproc.conf
  # Default limit for number of user's processes to prevent
  # accidental fork bombs.
  # See rhbz #432903 for reasoning.
  
  *          soft    nproc     32000
  *          hard    nproc     32000
  *          soft    nofile     32000
  *          hard    nofile     32000
  root       soft    nproc     unlimited
  

  Copy ke worker nodes

  [root@yava00 ~]# for i in `cat worker-nodes.txt`; do rsync -avz /etc/security/limits.d/20-nproc.conf root@${i}:/etc/security/limits.d/20-nproc.conf; done
  

• Disable SELINUX

  [root@yava00 ~]# for i in `cat all-nodes.txt`; do ssh root@${i} "setenforce 0 && sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config"; done
  

• Disable iptables

  [root@yava00 ~]# for i in `cat all-nodes.txt`; do ssh root@${i} "systemctl stop firewalld && systemctl disable firewalld"; done
  

• Adjust kernel parameter for Hadoop

  • Disable ipv6
  • Set swappiness = 1

  Tambahkan baris berikut di /etc/sysctl.conf lalu jalankan sysctl -p

  ########################
  ### Hadoop kernel tuning
  #######################
  vm.swappiness = 1
  
  # Disable ipv6
  net.ipv6.conf.all.disable_ipv6 = 1
  net.ipv6.conf.default.disable_ipv6 = 1
  net.ipv6.conf.lo.disable_ipv6 = 1
  
  ##############################
  ### end of Hadoop kernel tuning
  #############################
  

  Copy ke worker node:

  [root@yava00 ~]# for i in `cat worker-nodes.txt`; do rsync -avz /etc/sysctl.conf root@${i}:/etc/sysctl.conf; done
  

• Disable Transparent Huge Pages (THP)

  Tambahkan baris berikut di /etc/rc.local:

  # Disable THP
  if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
     echo never > /sys/kernel/mm/transparent_hugepage/defrag
  fi
  

  Copy ke worker node:

  [root@yava00 ~]# for i in `cat worker-nodes.txt`; do rsync -avz /etc/rc.local root@${i}:/etc/rc.local; done
  

• Configure CentOS Local Repository

• Configure YAVA Local Repository

• Install JDK

  Install OpenJDK 1.7 on all nodes.

  [root@yava00 ~]# for i in `cat all-nodes.txt`; do ssh root@${i} "yum install -y -q java-1.7.0-openjdk java-1.7.0-openjdk-devel"
  

• Enable NTP

  Install ntp on all nodes

  [root@yava00 ~]# for i in `cat all-nodes.txt`; do ssh root@${i} "yum install -q -y ntp"; done
  

  Start and enable service ntp on all nodes

  [root@yava00 ~]# for i in `cat all-nodes.txt`; do ssh root@${i} "systemctl start ntpd && systemctl enable ntpd"; done
  

  Pastikan jam-nya sama di semua node

  [root@yava00 ~]# for i in `cat all-nodes.txt`; do ssh root@${i} "date"; done
  

• Format Data Disk

• Reboot (jika diperlukan)

Installation