Setup Kubernetes Cluster on AWS using Kops - AndrewShvedRepo/Spark-Kubernetes-EC2-scratch GitHub Wiki

Kops Kops Runs only on:

  • Mac
  • Linux
  1. Spin up EC2 with Ubuntu image and make sure you can connect to it via ssh
ssh -i "kubernates.pem" [email protected]

or use Putty from Windows.

  1. Download Kops and Install kubectl 
curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64
chmod +x kops-linux-amd64
sudo mv kops-linux-amd64 /usr/local/bin/kops


sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl
  1. Install AWSCLI and its prerequisites
sudo apt update
sudo apt install python3-pip
sudo pip3 install awscli
aws
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
  1. Admin/PowerUser execute the following commands to create user kops in kops group with the required policies attached to the group. Record Access Key ID and Secret Access Key generated by the last command.
aws configure
AWS Access Key ID [xxxxxxxxCSHA]: xxxxxxxxCSHA
AWS Secret Access Key [xxxxxxxxUI1u]: xxxxxxxxUI1u
Default region name [us-east-2]: 
Default output format [None]:
aws iam create-group --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonRoute53FullAccess --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/IAMFullAccess --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonVPCFullAccess --group-name kops

aws iam create-user --user-name kops
aws iam add-user-to-group --user-name kops --group-name kops
aws iam create-access-key --user-name kops
  1. Configure your AWS CLI to use kops user by default
{
    "AccessKey": {
        "UserName": "kops",
        "AccessKeyId": "xxxxxxxx72LQ",
        "Status": "Active",
        "SecretAccessKey": "xxxxxxxxmE9E",
        "CreateDate": "2019-03-12T00:45:51Z"
    }
}
aws configure
AWS Access Key ID [xxxxxxxxCSHA]: xxxxxxxx72LQ
AWS Secret Access Key [xxxxxxxxUI1u]: xxxxxxxxmE9E
Default region name [us-east-2]: 
Default output format [None]:
  1. Create Private Hosted Zone and S3 bucket for your cluster. Note region and VPCId to which your will be deploying your Kubernetes cluster. Change bucket name to reflect your preferences.
aws route53 create-hosted-zone --name sparktest.local --vpc VPCRegion=us-east-2,VPCId=vpc-f42d0b33b --caller-reference sparktest.local
aws s3 mb s3://sparktest-local-state-store  --region us-east-2
  1. Set env variables and create the cluster
export NAME=test.sparktest.local
export KOPS_STATE_STORE=s3://sparktest-local-state-store

kops create cluster \
    --node-count 3 \
    --zones us-east-2a,us-east-2b,us-east-2c \
    --master-zones us-east-2a,us-east-2b,us-east-2c  \
    --topology private \
    --networking calico \
    --dns private $NAME

kops update cluster ${NAME} --yes
⚠️ **GitHub.com Fallback** ⚠️