Gluster is a high-performance shared file system that can be deployed in Amazon's EC2 cloud.

As explained in Amazon's user guide, an EC2 instance can have 2 types of storage: the "ephemeral" instance store of the host computer's internal hard drives, and the external EBS volumes, which can be thought of similar to plugging in an external drive to a desktop computer. In both cases, the data you store there is only accessible to a single instance at any one time. The main difference between them is what happens when you terminate an instance. Data stored on the ephemeral drive is lost: imagine destroying a desktop computer and smashing its internal hard drive when you do so. Meanwhile data on the EBS volume is still there, since it was on an external drive that you didn't smash up. That said, by default (and typically recommended), you can configure an instance (when you first launch it) to delete its EBS volume(s) when you terminate it as well. If you chose to keep the EBS volume instead, you can then reattach it to another instance, like plugging the external drive into a new computer.

VRPipe, however, needs to read from and write files to the same volume from many different instances at once: it needs a shared file system. Amazon does have a third storage type, S3, that could theoretically be used for this purpose, because the data stored is accessible from multiple instances. However, it does not present a hard-drive-like block-level storage, so can't be mounted like a normal drive. There are emulation layers that can make S3 seem like a normal mountable drive, but they are limited and slow.

A traditional solution might be to configure an NFS share. You can do this if you know how, and it may work, but the performance will not scale and it may become a large bottleneck if you need to have 1000s of instances reading and writing to your file system at once. Instead we recommend using Gluster.

With Gluster you dedicate some number of cheap instances to serving your filesystem. Each instance mounts 1 or more EBS volumes. Gluster effectively pools all the storage in the EBS volumes together and presents them as a single mountable volume. Your separate instances running VRPipe then mount that special volume. Performance and volume size can be scaled up simply by adding more Gluster instances.

The following guide shows you how to get started with an example 2-instance Gluster volume that uses simple mirroring of the data for redundancy instead of for speed (like doing raid-1).

Create a Configured Gluster AMI

1. Launch a t1.micro instance running our public unconfigured VRPipe AMI (or any AMI and install Gluster on it) with a 2GB EBS volume attached as /dev/sdb set to not delete on termination (the root volume on /dev/sda1 should be deleted on termination). Instead of just 2GB you can choose a larger size as appropriate for your needs (it should be big enough to hold your software and data).
   
   
2. SSH to it and set the Gluster daemon to start at boot time, but since we're going to make an AMI of this, delete the UUID file:
   sudo chkconfig glusterd on
sudo rm /var/lib/glusterd/glusterd.info
3. If the EBS volume is new and unformatted, format it now:
   sudo fdisk /dev/sdb and answer n, p, 1, enter, enter, w to create a single full-size primary partition
   sudo mkfs.xfs -f /dev/sdb1
4. Configure the volume to be mounted at boot time:
   sudo mkdir -p /mnt/ebs
sudo chown ec2-user:ec2-user /mnt/ebs
sudo su
echo '/dev/sdb1 /mnt/ebs xfs rw,user,auto 0 0' >> /etc/fstab
exit
mount /mnt/ebs (just to check that it works)
5. Create an AMI of this instance with a new name like 'my_gluster_install'.
6. Terminate the instance.

Create a Mountable Gluster Volume

Of note in this section is that we launch instances with manually specified IP addresses: this is not necessary but very useful in case one of our Gluster instances crashes or otherwise needs to be terminated; we can bring up a new instance with the same IP address as the old one and recover more easily (ie. without having to change any configurations elsewhere).

1. Independently launch 2 new t1.micros running the new AMI, each with a unique manually chosen ip address in the subnet your configured VRPipe instances will be in. The security group and availability zone should also match your VRPipe instances. For example, the eu-west-1a subnet is 172.31.16.0/20 so you might chose 172.31.16.100 for the first, and 172.31.16.101 for the second. These instances should end up with their own independent copies of the formatted EBS volume, mounted as /mnt/ebs (they have their own new EBS volumes, but they start of being based on a snapshot of the EBS volume we formatted in the previous section).
   
2. At this point you can delete the original EBS volume you formatted in the previous section:
   
   
3. Chose one of the instances and consider it the 'master' (in this example, I chose 172.31.16.100), SSH to it and probe the other one to form the network of Gluster instances:
   sudo gluster peer probe 172.31.16.101
   (sudo gluster peer status on the other shows it worked)
4. Create a Gluster volume that replicates the data from the master to the other instance:
   sudo gluster volume create gv0 replica 2 172.31.16.100:/mnt/ebs 172.31.16.101:/mnt/ebs
sudo gluster volume start gv0

You're now finished interacting with these instances, but they must be left running for as long as you want access to the data stored on their Gluster volume gv0. If you ever end up terminating one of them, to recover and gain access to the data again, launch a new instance with the same IP as the old one, but don't attach an EBS volume within the launch wizard (which will be the default). Instead launch it and then attach the existing volume that has (some of) your data on it.

Mount the Gluster Volume

Now that a Gluster volume is available, we'll bring up a new normal VRPipe instance and mount it.

1. Launch an EC2 instance running our public unconfigured VRPipe AMI (or any AMI with VRPipe and glusterfs available; not the my_gluster_install AMI you made in the first section of this guide).
2. Mount the Gluster volume:
   sudo mkdir /shared
sudo chown ec2-user:ec2-user /shared
sudo su
echo '172.31.16.100:/gv0 /shared glusterfs defaults 0 0' >> /etc/fstab (where 172.31.16.100 should be replaced with the address of the 'master' Gluster instance you launched in the previous section)
   exit
sudo mount /shared
sudo chown -R ec2-user:ec2-user /shared

Now /shared is a shared disk! You can move your files (including software installed to your home directory, which may include VRPipe itself) to the shared disk, and update your environment variables. If you used our unconfigured VRPipe AMI you can do this by:

1. mv $SOFTWAREDIR /shared/
2. echo 'export SOFTWAREDIR=/shared/software' > /shared/software/.profile
3. source ~/.bash_profile
4. Create an AMI of this instance with a new name like 'unconfigured vrpipe on mounted gluster'.
5. Terminate any existing VRPipe instances and launch a new one with the new AMI. Make sure the availability zone/subnet and security group are the same as for the Gluster instances.