Cluster Guide - achakra/seck GitHub Wiki

Prereqs:

  1. Install Vagrant: http://downloads.vagrantup.com/tags/v1.0.5
  2. Install Virtual Box: https://www.virtualbox.org/wiki/Downloads
  3. Download Maven: http://maven.apache.org/download.html
  4. Download/pull/clone latest seck: https://github.com/pcwerk/seck/
  5. Install Putty (Windows Users only): http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
  6. Install Putty Gen (Windows Users only): http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

Steps:

  1. In command prompt, cd into seck-master from the latest seck.
  2. do: mvn install
  3. do: mvn package
  4. cd into your seck-cluster\target folder
  5. unzip seck-cluster-0.0.5-dev-dist.zip into your seck-cluster\target folder
  6. cd into seck-cluster-0.0.5-dev-dist\vagrant-hadoop-cluster
  7. Add precise64 by typing: vagrant box add precise64 http://files.vagrantup.com/precise64.box
  8. In vagrant-hadoop-cluster, initialize cluster by then typing: vagrant up
  9. Next login into master by typing vagrant ssh master. NOTE: This only works for Mac and Linux users. If you are a Windows User, you will receive a host number and port number. Save that information. Windows Users should continue with the following steps.

Windows Users:

 1. Open Putty Gen
 2. Click "Conversions" then select "Import Key"
 3. Navigate to the the following folder "USER/.vagrant.d/", USER being your user directory.
 4. Select the "insecure_private_key" file.
 5. Then select the "Save private key" option. Save it as key.ppk on your Desktop
 6. Open Putty
 7. Type in the Host number and Port number that was on the command prompt when typing in vagrant ssh master
 8. In Saved Sessions, type hadoop or any name
 9. Go to Connection --> Data
 10. Auto-login username: vagrant
 11. Go to Connection --> Expand SSH --> Auth --> browse --> upload your key.ppk
 12. Go back to session --> logging and click save.
 13. Highlight your saved session and click open.
 14. Click yes on Putty Security Alert

Running Hadoop Clusters(must do the following section of code every time you run the clusters):

  1. Now type "cd /opt/hadoop-1.0.4/bin"

  2. While in hadoop-1.0.4/bin directory type "sudo ./hadoop namenode -format"

  3. If you receive the following warning "REMOTE HOST IDENTIFICATION HAS CHANGED", Enter the following code:

    ssh-keygen -f "/root/.ssh/known_hosts" -R backup ssh-keygen -f "/root/.ssh/known_hosts" -R hadoop1 ssh-keygen -f "/root/.ssh/known_hosts" -R hadoop2 ssh-keygen -f "/root/.ssh/known_hosts" -R hadoop3

     ssh-keygen -f "/root/.ssh/known_hosts" -R 192.168.1.11
     ssh-keygen -f "/root/.ssh/known_hosts" -R 192.168.1.12
     ssh-keygen -f "/root/.ssh/known_hosts" -R 192.168.1.13
     ssh-keygen -f "/root/.ssh/known_hosts" -R 192.168.1.14
    
     ssh backup
     ssh-keygen -f "/root/.ssh/known_hosts" -R master
     ssh-keygen -f "/root/.ssh/known_hosts" -R 192.168.1.10
     ssh master
    
    
     ssh hadoop1
     ssh-keygen -f "/root/.ssh/known_hosts" -R master
     ssh-keygen -f "/root/.ssh/known_hosts" -R 192.168.1.10
     ssh master
    
     ssh hadoop2
     ssh-keygen -f "/root/.ssh/known_hosts" -R master
     ssh-keygen -f "/root/.ssh/known_hosts" -R 192.168.1.10
     ssh master
    
     ssh hadoop3
     ssh-keygen -f "/root/.ssh/known_hosts" -R master
     ssh-keygen -f "/root/.ssh/known_hosts" -R 192.168.1.10
     ssh master
    
  4. now type "sudo ./start-all.sh". Your cluster is now setup.

  5. To stop your clusters enter "sudo ./stop-all.sh"

  6. To stop vagrant enter "vagrant halt" in the command prompt. Do not vagrant destroy.