◀️ Hazelcast Workspaces on VMs 🔗 Reactivating Hazelcast Workspaces on AWS EC2 ▶️

Since Padogrid v0.9.20

This article provides step-by-step instructions for creating and running a hazelcast-addon workspace on AWS EC2 instances.

From the EC2 Dashboard launch five (5) EC2 instances of type t2.micro on three (3) different availability zones and collect their public IP addresses. For our example, we launched two (2) instances on each of the us-east-2a and us-east-2b availability zones, one (1) instance on us-east-2c for the management center, named the instances, and collected their public IP addresses as follows:

We also created the key-value tag of hzaddon and aws-cluster which we will use when we configure the EC2 discovery service later.

For our example, the security group for the EC2 instances should have the following rules defined for both inbound and outbound traffics.

To create a workspace, run create_workspace, which by default runs in the interactive mode. For our example, let's run it in the non-interactive mode by specifying the -quiet option as shown below. (Change the option values to your settings.)

create_workspace -quiet \
-name ws-aws-hazelcast \
-java /Library/Java/JavaVirtualMachines/jdk1.8.0_333.jdk/Contents/Home \
-product /Users/dpark/Padogrid/products/hazelcast-enterprise-5.1.3-slim \
-vm 3.19.14.241,3.135.241.115,52.14.100.181,3.135.226.150,18.216.40.10 \
-vm-user ec2-user \
-vm-key mykey.pem

The above creates the workspace named ws-aws-hazelcast and places all the installations in the /home/ec2-user/Padogrid directory. When you are done with installation later, each EC2 instance will have the following folder contents:

/home/ec2-user/Padogrid
├── downloads
│   ├── hazelcast-enterprise-5.1.3-slim.tar.gz
│   └── jdk-8u333-linux-x64.tar.gz
├── products
│   ├── hazelcast-enterprise-5.1.3-slim
│   ├── hazelcast-management-center-5.1.4
│   ├── jdk1.8.0_333
│   └── padogrid_0.9.20
└── workspaces
    └── ws-aws-hazelcast
        └── clusters
            └── myhz

The non-vm options, -java and -product must be set to Java and Hazelcast home paths in your local file system.

The following table shows the breakdown of the options used in the example.

The following table shows rest of the options not included in the example. These options take default values if not specified.

We launched t2.micro instances which only have 1 GB of memory. We need to lower the Hazelcast member heap size to below this value. Edit the cluster.properties file as follows:

switch_workspace ws-aws-hazelcast
switch_cluster myhz
vi etc/cluster.properties

Change the heap and host properties in that file as shown below. You may not able to run Hazelcast with the max heap size greater than 512 MB on t2.micro EC2 instances. You can create a swapspace to overcome this limitation, however. Please see the next section for instructions on creating a swap space.

# Lower the heap size from 1g to 512m
heap.min=512m
heap.max=512m

Let's set mc.host to the Management Center IP address. Note that the create_workspace command created the myhz cluster by executing the create_cluster command, which by default sets the first VM host you specified using the -vm option as the Management Center host.

# Set the first VM as the management center (us-east-2c zone)
mc.host=3.19.14.241 

Now, set the vm.hosts property to the remaining hosts. Note also that the create_cluster command automatically sets this property to the value of the -vm option. We just need to remove the first VM host in the list which we will use only for the Management Center host.

# Rest of the 4 VMs for members
vm.hosts=3.135.241.115,52.14.100.181,3.135.226.150,18.216.40.10

switch_workspace ws-aws-hazelcast
switch_cluster myhz
vi etc/hazelcast.xml

Hazelcast provides an EC2 discovery service which discovers EC2 instances along with their metadata such as availability zone information for configuring zone-aware HA. Configure the discovery service as shown below with your access key and secret access IDs. Also, configure the partition group to ZONE_AWARE.

<hazelcast ...>
    ...
	<network>
        ...
		<join>
			<multicast enabled="false"></multicast>
			<aws enabled="true">
				<access-key>your-access-key</access-key>
				<secret-key>your-secret-key</secret-key>
				<region>us-east-2</region>
				<security-group-name>hazelcast-cluster</security-group-name>
				<tag-key>hzaddon</tag-key>
				<tag-value>aws-cluster</tag-value>
				<hz-port>5701</hz-port>
				<connection-retries>3</connection-retries>
			</aws>
		</join>
        ...
    </network>
    <partition-group enabled="true" group-type="ZONE_AWARE"></partition-group>
    ...
</hazelcast>

The t2.micro EC2 instances do not include swap spaces. The following example shows how to add a 1-GB swap space to enable virtual memory on all VMs. This allows you increase the max heap size.

# Create swap space of 1 GB. To change the swap space size, change the count.
# For example, to increase to 2 GB, set count=16 (128 MB * 16).
vm_exec "sudo dd if=/dev/zero of=/swapfile bs=128M count=8 && \
sudo dd if=/dev/zero of=/swapfile bs=128M count=8 && \
sudo chmod 600 /swapfile && \
sudo mkswap /swapfile && \
sudo swapon /swapfile && \
sudo swapon -s"

You can start the swap file at boot time by editing the /etc/fstab file. You can skip this step if you won't be rebooting the VMs.

# ssh into VM
ssh -i mykey.pem [email protected]
sudo vi /etc/fstab

Add the following new line at the end of the file, save the file, and then exit:

/swapfile swap swap defaults 0 0

You must carry out the above steps for each VM.

Hazelcast EncryptionReplacer allows you to replace encrypted variables. We can use it to optionally encrypt the AWS access and secret keys. For our example, let's supply a password file and use its content as a part of the encryption password. The my-password.txt file used in the following example can contain any text that you want to make a part of the encryption password. You can find further details from the Hazelcast manual

# Encrypt "your-access-key"
java -cp hazelcast-enterprise-5.1.3.jar \
-DpasswordFile=my-password.txt \
-DpasswordUserProperties=false \
com.hazelcast.config.replacer.EncryptionReplacer "your-access-key"

Output:

$ENC{mHxl92a2Bwo=:531:QFfLsv3cLkytFssIlyTD8w==}

# Encrypt "your-secret-key"
java -cp hazelcast-enterprise-5.1.3.jar \
-DpasswordFile=my-password.txt \
-DpasswordUserProperties=false \
com.hazelcast.config.replacer.EncryptionReplacer "your-secret-key"

Output:

$ENC{kFhAZFHzMtU=:531:Q4nLcHpGb5i01blExoUXog==}

The encrypted keys are then placed in the configuration file along with the EncryptionReplacer properties as follows.

<hazelcast ...>
    ...
    <config-replacers>
        <replacer class-name="com.hazelcast.config.replacer.EncryptionReplacer">
            <properties>
                <property name="passwordFile">/home/ec2-user/Hazelcast/workspaces/ws-aws-hazelcast/clusters/myhz/etc/my-password.txt</property>
                <property name="passwordUserProperties">false</property>
            </properties>
        </replacer>
    </config-replacers>

	<network>
        ...
        <join>
            <multicast enabled="false"></multicast>
            <aws enabled="true">
                <access-key>$ENC{mHxl92a2Bwo=:531:QFfLsv3cLkytFssIlyTD8w==}</access-key>
                <secret-key>$ENC{kFhAZFHzMtU=:531:Q4nLcHpGb5i01blExoUXog==}</secret-key>
                <region>us-east-2</region>
                <security-group-name>hazelcast-cluster</security-group-name>
                <tag-key>hzaddon</tag-key>
                <tag-value>aws-cluster</tag-value>
                <hz-port>5701</hz-port>
                <connection-retries>3</connection-retries>
            </aws>
        </join>
        ...
    </network>
    <partition-group enabled="true" group-type="ZONE_AWARE"></partition-group>
    ...
</hazelcast>

For our example, we have installed Hazelcast Enterprise which requires a license key. The license keys are normally placed in the RWE .hazelcastenv.sh file ($PADOGRID_WORKSPACES_HOME/.hazelcastenv.sh), which is synchronized when you execute the vm_sync command (see next section). If you need to use license keys other than the ones set in the .hazelcastenv.sh file then you can set them in the clusters's setenv.sh file, which overrides the environment variables set in the .hazelcastenv.sh file.

# Change directory to the current workspace, 'ws-aws-hazelcast', i.e., $PADOGRID_WORKSPACE
cd_cluster
vi bin_sh/setenv.sh

Add the license keys in setenv.sh to override .hazelcastenv.sh (Do this only if you need to override the license keys set in .hazelcastenv.sh):

# Hazelcast IMDG license
IMDG_LICENSE_KEY=***
#Jet license
JET_LICENSE_KEY=***
# Management Center
MC_LICENSE_KEY=***

Run vm_sync to synchronize the workspace.

vm_sync

The above command reports the following:

Scanning VMs... Please wait.
Deploying padogrid_0.9.20 to 3.19.14.241...
Deploying padogrid_0.9.20 to 3.135.241.115...
Deploying padogrid_0.9.20 to 52.14.100.181...
Deploying padogrid_0.9.20 to 3.135.226.150...
Deploying padogrid_0.9.20 to 18.216.40.10...

Workspace sync: ws-aws-hazelcast
   Synchronizing 3.19.14.241...
   Synchronizing 3.135.241.115...
   Synchronizing 52.14.100.181...
   Synchronizing 3.135.226.150...
   Synchronizing 18.216.40.10...

Updating remote (VM) '.bashrc' if needed...
...
Workspace sync complete.

vm_sync will display warning messages indicating the new EC2 instances do not have the required software installed. Download the required software and install them by running the vm_install command as shown in the following example.

vm_install -product ~/Downloads/jdk-8u333-linux-x64.tar.gz
vm_install -product ~/Downloads/hazelcast-enterprise-5.1.3-slim.tar.gz
vm_install -product ~/Downloads/hazelcast-management-center-5.1.4.tar.gz

Start the cluster.

start_cluster

To monitor the cluster:

show_cluster

To view logs:

# member1
show_log

# member2
show_log -num 2

We've configured PadoGrid to run the Management Center on one of the VMs.

Start the Management Center.

start_mc

The start_mc command will display the Management Center URL with the internal host name. To get the URL with the public IP address, run the show_mc command.

show_mc

For our example, it displays the following:

----------------------------------------------------------------
    CLUSTER: myhz
CLUSTER_DIR: /home/ec2-user/Padogrid/workspaces/rwe-hazelcast/ws-aws-hazelcast/clusters/myhz
       Name: myhz-mc-8080
      STATE: Down
        PID: N/A
 Deployment: VM
        URL: http://3.19.14.241:8080/hazelcast-mancenter
----------------------------------------------------------------

You can run the perf_test app to ingest data into the cluster and monitor the map sizes increase from the Management Center.

First, create the perf_test app and edit its configuration file.

create_app
cd_app perf_test
vi etc/hazelcast-client.xml

Add the same EC2 discovery settings in the client-cache.xml file as we did for the cluster. If you will be running the client app from outside the EC2 environment then you must also add the <use-public-ip> element as shown below.

<hazelcast-client ...>
...
   <network>
      <aws enabled="true">
         <access-key>your-access-key</access-key>
         <secret-key>your-secret-key2</secret-key>
         <region>us-east-2</region>
         <security-group-name>hazelcast-cluster</security-group-name>
         <tag-key>hzaddon</tag-key>
         <tag-value>aws-cluster</tag-value>
         <hz-port>5701</hz-port>
         <connection-retries>3</connection-retries>
         <!-- For running apps outside EC2 -->
         <use-public-ip>true</use-public-ip>
      </aws>
   </network>
...
<hazelcast-client>

Ingest data into the cluster.

cd bin_sh
./test_ingestion -run

You should see the cluster filling up with data from the Management Center.

You can include additional EC2 instances to the workspace by entering the new instance IP addresses in the workspace setenv.sh file. Let's launch a new EC2 instance for running client apps. We ran the perf_test app locally from your laptop in the previous section. We'll run the same app from the new EC2 instance you launched. For our example, the new EC2 instance has the following IP address.

Let's add the IP address to the workspace setenv.sh file.

cd_workspace
vi vmenv.sh

Append the new IP address (3.18.102.53) to the VM_HOSTS environment variable as follows:

VM_HOSTS="3.18.113.154 3.21.44.203 18.218.42.247 18.222.225.117 18.188.216.237,3.18.102.53"

After saving the setenv.sh file, run vm_sync to synchronize the VMs so that the new instance will have hazelcast-addon installed.

vm_sync

The vm_sync command output should be similar to what we have seen before but this time it deploys PadoGrid only to the new VM.

Scanning VMs... Please wait.
Deploying padogrid-0.9.20 to 3.18.102.53...

Workspace sync: ws-aws-hazelcast
   Synchronizing 3.19.14.241...
   Synchronizing 3.135.241.115...
   Synchronizing 52.14.100.181...
   Synchronizing 3.135.226.150...
   Synchronizing 18.216.40.10...
   Synchronizing 3.18.102.53...

Updating remote (VM) '.bashrc' if needed...
...
Workspace sync complete.

Let's install Java and Hazelcast as before. We can skip deploying the management center to the new VM since we won't be running it on the new VM.

vm_install -product ~/Downloads/jdk-8u333-linux-x64.tar.gz
vm_install -product ~/Downloads/hazelcast-enterprise-5.1.3-slim.tar.gz

Let's login to the new EC2 instance and run the perf_test app from there.

# First change directory to the workspace where the key file (mykey.pem) is located.
cd_workspace

# ssh into the client VM
ssh -i mykey.pem [email protected]

When we ran vm_sync earlier, it also deployed the perf_test app to all the VMs. Note that we set the <use-public-ip> element was set to true. If you have the security group set differently than our example, you may need to change this value to false so that the app uses the internal IP instead.

Edit etc/hazelcast-client.xml

cd_app perf_test
vi etc/hazelcast-client.xml

Set <use-public-ip> to false.

<hazelcast-client ...>
...
   <network>
      <aws enabled="true">
         ...
         <!-- For running apps inside EC2 -->
         <use-public-ip>false</use-public-ip>
      </aws>
   </network>
...
<hazelcast-client>

Run test_ingestion.

cd bin_sh
./test_ingestion -run

If you terminate the EC2 instances without removing the workspace, then your workspace will be preserved in your local machine. This means you can later reactivate the workspace by simply launching new EC2 instances and configuring the workspace with the new public IP addresses. The following link provides step-by-step instructions describing how to reactivate VM workspaces.

Reactivating Hazelcast Workspaces on AWS EC2

1. Stop the cluster

If you want to remove the cluster from all the VMs, then you must first stop the cluster and execute the remove_cluster command.

# Stop cluster and management center
stop_cluster 
stop_mc

2. Remove the workspace

If you want to preserve the workspace so that you can later reactivate it then skip this step.

# Simulate removing workspace from all VMs. Displays removal steps but does not
# actually remove the workspace.
remove_workspace -workspace ws-aws-hazelcast -simulate

# Remove workspace from all VMs. Runs in interactive mode.
remove_workspace -workspace ws-aws-hazelcast

3. Terminate the EC2 instances

From the EC2 Dashboard terminate the EC2 instances.

◀️ Hazelcast Workspaces on VMs 🔗 Reactivating Hazelcast Workspaces on AWS EC2 ▶️