EMR 009 Grow EBS volumes on EMR clusters - qyjohn/AWS_Tutorials GitHub Wiki

You have an EMR cluster running in production, and data is gradually filling up your EBS volumes. You need to grow the size of the EBS volumes on your EMR cluster.

(0) You will need to be able to use the AWS CLI, with ec2:ModifyVolume and ec2:DescribeVolume permissions.

(1) Save the following bash script as modify-ebs.sh. This bash script uses ec2:ModifyVolume to modify an EBS volume to the desired new size, and monitors the process using ec2:DescribeVolume until the modification is done.

#!/bin/bash
region=$1
volume=$2
size=$3
aws ec2 modify-volume --region $region --volume-id $volume --size $size
volumeStatus=""
while [ $volumeStatus != "completed" ](/qyjohn/AWS_Tutorials/wiki/-$volumeStatus-!=-"completed"-)
do
  sleep 5
  volumeStatus=$(aws ec2 describe-volumes-modifications --region $region --volume-id $volume | jq .VolumesModifications[].ModificationState)
  volumeStatus=${volumeStatus//\"/}
  echo $volumeStatus
done

(2) In your EMR console, identify the EC2 instances in your core instance group. Then, in the EC2 console, identify the EBS volume that stores your data. This is usually your /dev/sdb (but renamed to /dev/xvdb inside the operating system). Then, use the modify-ebs.sh script to modify the EBS volume to the desired size. For example

chmod +x modify-ebs.sh
./modify-ebs.sh us-west-2 vol-12345678901234567 100

(3) Then you will need to SSH into all core nodes, then use the following commands to grow the data partition, then grow the file system (XFS). You can use "df -h" and "lsblk" to review the related information before and after the modification.

lsblk
df -h
sudo growpart /dev/xvdb 2
sudo xfs_growfs -d /mnt
lsblk
df -h

If the partition is using ext2, ext3, or ext4 as the file system, you will need to replace "xfs_growfs" with "resize2fs".

(4) If local disk is encrypted, use the following steps:

• Resize the EBS volume

./modify-ebs.sh $region $volume_id $size

• Resize the partition

sudo growpart /dev/xvdb 2

• Resize the encrypted layer (LUKS volume)

sudo cryptsetup resize /dev/mapper/xvdb2

• Resize XFS

sudo xfs_growfs -d /mnt

This procedure has been tested on an EMR 5.6.0 with c4.2xlarge instances.