Hazelcast Workspaces on VMs - padogrid/padogrid GitHub Wiki
◀️ Hazelcast WAN Example :link: Hazelcast-on-AWS-EC2 ▶️
Hazelcast/Jet Workspaces on VMs
PadoGrid provides a simple way to manage Hazelcast/Jet clusters on laptop, PadoGrid pods, and VMs. This article describes how to create and run a Hazelcast/Jet cluster on multiple VMs and/or PMs (Physical Machines). The instructions provided apply to any VMs and PMs including AWS, Azure, GCP, physical machines, VirtualBox, vSphere, PadoGrid pods (VMs), etc.
With PadoGrid, you can have a complete control over Hazelcast/Jet running on VMs from your local machine. PadoGrid preserves the state of VM configurations and deployments on your local machine so that you can conveniently activate or deactivate the VM environment at any time. After you are done running VMs, you can tear them down and reinstate the very same environment later with a single command. This saves you time and reduces VM costs when you are working in a cloud environment, for example.
Installation Steps
There are four (4) main steps to installing PadoGrid on VMs. Once these steps are performed, you can then synchronize your local machine with the remote VMs by simply executing the vm_sync command, which enables the local control of VMs.
- Setup password-less SSH login
- Download required software
- Create VM workspace
- Synchronize VM Workspace
Password-less SSH Login
The first step is to check to make sure you are able to login to the VMs that you want to cluster without the password. For AWS EC2, for example, this is already done for you so you can skip this section.
If you need to manually setup password-less SSH login, then follow the instructions here.
:exclamation: Make sure you can ssh from any of the VMs including your local machine without the password.
Download Software
PadoGrid requires the following software products. If they are not installed on VMs, then download their tarball or zip distributions.
For our demo, we'll assume they are downloaded in the following directory. Note that they must be tarball or zip distributions.
/home/dpark/Downloads
├── jdk1.8.0_212.tar.gz
└── hazelcast-enterprise-4.0.1.tar.gz
Create VM Workspace
Create a VM workspace on your local machine, e.g., your laptop.
create_workspace -name ws-hz -vm
The above command interactively prompts for software installation paths information. The -vm option enables VMs and -name option names the workspace. The following shows my laptop environment. The environment variables that begin with VM_ are specific to the remote VMs. All other environment variables without the prefix VM_ pertain to your laptop environment. For example, VM_PADOGRID_HOME is the PadoGrid installation path in the VM hosts you specified.
Please answer the prompts that appear below. If you are not able to complete
the prompts at this time then use the '-quiet' option to bypass the prompts.
You can complete the requested values later in the generated 'setenv.sh' file
You can abort this command at any time by entering 'Ctrl-C'.
Enter Java home path.
[/home/dpark/Padogrid/products/jdk1.8.0_212]:
Enter the local product home directory path. Choose one
from the defaults listed below or enter another.
/home/dpark/Padogrid/products/hazelcast-enterprise-4.0.1
/home/dpark/Work/linux/hazelcast-jet-enterprise-3.2.2
/home/dpark/Padogrid/products/pivotal-gemfire-9.10.0
[]:
/home/dpark/Padogrid/products/hazelcast-enterprise-4.0.1
Enter workspace name.
[ws-hz]:
Enter default cluster name.
[myhz]:
Enable VM? Enter 'true' or 'false' [true]:
Enter VM JDK home path.
[/home/dpark/Padogrid/products/jdk1.8.0_212]:
/home/dpark/Hazelcast/jdk1.8.0_212
Enter VM product home path.
[/home/dpark/Padogrid/products/hazelcast-enterprise-4.0.1]:
/home/dpark/Hazelcast/hazelcast-enterprise-4.0.1
Enter VM padogrid home path.
[/home/dpark/Padogrid/products/padogrid_0.9.7]:
/home/dpark/Hazelcast/padogrid_0.9.7
Enter VM workspaces path.
[/home/dpark/Padogrid/workspaces/myrwe]:
/home/dpark/Hazelcast/workspaces/myrwe
Enter VM host names or IP addresses separated by comma.
[]:
ubuntu1,ubuntu2,ubuntu3,ubuntu4
Enter VM user name.
[]: dpark
Enter VM SSH private key path. If you don't have the key file (.pem) then
leave it blank for now. You can place the file in the workspace directory or
set the path in the workspace 'setenv.sh' file later.
[]:
You have entered the following.
JAVA_HOME: /home/dpark/Padogrid/products/jdk1.8.0_212
PRODUCT_HOME: /home/dpark/Padogrid/products/hazelcast-enterprise-4.0.1
PADOGRID_WORKSPACE: /home/dpark/Padogrid/workspaces/myrwe/ws-hz
Default Cluster: myhz
VM_ENABLED: true
VM_JAVA_HOME: /home/dpark/Hazelcast/jdk1.8.0_212
VM_PRODUCT_HOME: /home/dpark/Hazelcast/hazelcast-enterprise-4.0.1
VM_PADOGRID_HOME: /home/dpark/Hazelcast/padogrid_0.9.7
VM_PADOGRID_WORKSPACES_HOME: /home/dpark/Hazelcast/workspaces/myrwe
VM_PADOGRID_WORKSPACE: /home/dpark/Hazelcast/workspaces/myrwe/ws-hz
VM_HOSTS: ubuntu1,ubuntu2,ubuntu3,ubuntu4
VM_USER: dpark
VM_PRIVATE_KEY_FILE:
Enter 'c' to continue, 'r' to re-enter, 'q' to quit: c
----------------------------------------------------------------
WORKSPACE: /home/dpark/Padogrid/workspaces/myrwe/ws-hz
Cluster Created: myhz
CLUSTER_TYPE: imdg
Deployment: VM
CLUSTER_DIR: /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz
CONFIG_DIR: /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/etc
CONFIG_FILE: /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/etc/hazelcast.xml
LOG_DIR: /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/log
MEMBER_START_PORT: 5701
DEBUG_START_PORT: 9101
JMX_START_PORT: 12001
PROMETHEUS_START_PORT: 8091
MC_HTTP_PORT: 8080
MC_HTTPS_PORT: 8443
MC_JMX_PORT: 9001
MC_JMX_RMI_PORT: 9051
The default hazelcast.xml file has been placed in the CONFIG_DIR directory.
Modify it or replace it with the appropriate file.
One (1) member has been configured (added) to run in this cluster as follows:
Member: myhz-padogrid-01
Working Dir: /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/run/myhz-padogrid-01
1. Make sure to replace or update hazelcast.xml:
/home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/etc/hazelcast.xml
2. Place your application jar files in one of the 'plugins' directories:
/home/dpark/Padogrid/workspaces/myrwe/ws-hz/plugins/
/home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/plugins/
3. Configure cluster properties in:
/home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/etc/cluster.properties
4. You must configure password-less ssh login to all VMs. On AWS, this
is already configured but you will need to enter the correct private key
file path in the cluster.properties file.
5. To change VM host names, list the VM host names in the cluster.properties file
for management center and members.
mc.host=host1
vm.hosts=host1,host2,...
6. Deploy the cluster configuration changes you made to all the VM hosts listed in the
previous step by running 'vm_copy' or 'vm_sync'.
vm_copy /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz
7. To start the cluster:
start_cluster -cluster myhz
8. To verify the cluster state:
show_cluster -cluster myhz
9. To view log file:
show_log -cluster myhz
----------------------------------------------------------------
Workspace created:
/home/dpark/Padogrid/workspaces/myrwe/ws-hz
The workspace directory also contains the following files.
initenv.sh - Workspace initialization file. Source this file in to
enter the workspace.
setenv.sh - You can set your workspace-wide environment variables in
this file.
.addonenv.sh - A hidden file containing the workspace environment variables.
You can overwrite them in setenv.sh as needed.
IMPORTANT:
----------
To use this workspace, execute the 'switch_workspace' command as follows:
switch_workspace ws-hz
You have created a VM workspace. To add or remove VM hosts, set VM_HOSTS in
the workspace 'setenv.sh' file. Note that VM_HOSTS applies to the entire workspace
whereas the 'vm.hosts' property in the cluster 'etc/cluster.properties' file applies
only to the defining cluster. By default, 'vm.hosts' is initially set to VM_HOSTS.
/home/dpark/Padogrid/workspaces/myrwe/ws-hz/setenv.sh
The above example shows that the installation paths in the VMs are different from the local installation paths. We choose to install all the software components in the /home/dpark/Hazelcast for the VMs.
If you specify the private key file, i.e., ecs.pem in our example, then it is automatically copied to the newly created workspace directory. It is also later deployed to the VMs when you execute the vm_sync command.
You can also run the create_workspace in the non-interactive mode by specifying the -quiet option. The following command produces the same workspace configurations as the above interactive example.
create_workspace -quiet \
-name ws-hz \
-cluster myhz \
-java /home/dpark/Padogrid/products/jdk1.8.0_212 \
-product /home/dpark/Padogrid/products/hazelcast-enterprise-4.0.1 \
-vm ubuntu1,ubuntu2,ubuntu3,ubuntu4 \
-vm-java /home/dpark/Hazelcast/jdk1.8.0_212 \
-vm-product /home/dpark/Hazelcast/hazelcast-enterprise-4.0.1 \
-vm-padogrid /home/dpark/Hazelcast/padogrid_0.9.7 \
-vm-workspaces /home/dpark/Hazelcast/workspaces/myrwe \
-vm-user dpark
SSH Private Key
If a private key file is required and you have not specified it in the previous example, then you can place it in the workspace directory. PadoGrid automatically picks up the first .pem file found in the workspace directory if the VM_PRIVATE_KEY_FILE environment variable is not set in the $PADOGRID_WORKSPACE/setenv.sh file. The following shows the contents of the ws-hz workspace directory we created.
# If a private key file (.pem) is required then place it in in the
# ws-hz workspace directory.
switch_workspace ws-hz
tree -L 1 .
.
├── apps
├── bundles
├── clusters
├── docker
├── initenv.sh
├── k8s
├── lib
├── plugins
├── pods
└── setenv.sh
Test VM Workspace
Before you sync the local workspace with VMs - which may initially take some time to complete - if you need to install software products, we should first test the environment to make sure SSH works properly. This is done by executing vm_test.
vm_test
vm_test outputs the following:
------------------------------------------------------------------------------------------
Workspace: ws-hz
Environment:
VM_ENABLED=true
VM_HOSTS=ubuntu1,ubuntu2,ubuntu3,ubuntu4
VM_USER=dpark
VM_PRIVATE_KEY_FILE=
VM_JAVA_HOME=/home/dpark/Hazelcast/jdk1.8.0_212
VM_HAZELCAST_HOME=/home/dpark/Hazelcast/hazelcast-enterprise-4.0.1
VM_PADOGRID_HOME=/home/dpark/Hazelcast/padogrid_0.9.7
VM_PADOGRID_WORKSPACE=/home/dpark/Hazelcast/workspaces/myrwe/ws-hz
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
Network Test:
ubuntu1 - OK
ubuntu2 - OK
ubuntu3 - OK
ubuntu4 - OK
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
WARNING:
/home/dpark/Hazelcast/jdk1.8.0_212
Java is not installed on the following VMs. The workspace will not be operational
until you install Java on these VMs.
ubuntu1 ubuntu2 ubuntu3 ubuntu4
To install Java on the above VMs, download the correct version of Java and execute 'vm_install'.
Example:
vm_install -java jdk1.8.0_212.tar.gz
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
WARNING:
/home/dpark/Hazelcast/hazelcast-enterprise-4.0.1
Hazelcast is not installed on the following VMs. The workspace will not be operational
until you install Hazelcast on these VMs.
ubuntu1 ubuntu2 ubuntu3 ubuntu4
To install Hazelcast on the above VMs, download the correct version of Hazelcast and
execute 'vm_install'.
Example:
vm_install -product hazelcast-enterprise-4.0.1.tar.gz
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
/home/dpark/Hazelcast/padogrid_0.9.7
padogrid is not installed on the following VMs. It will be automatically installed
when you execute 'vm_sync'.
ubuntu1 ubuntu2 ubuntu3 ubuntu4
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
Summary:
One or more VM issues found. Please correct them before executing 'vm_sync'.
Workspace Issues:
None.
Network Issues:
None.
Software Issues:
Java missing from the following VMs. Install Java with 'vm_install -java'.
ubuntu1 ubuntu2 ubuntu3 ubuntu4
Hazelcast missing from the following VMs. Install Hazelcast with 'vm_install -product'.
ubuntu1 ubuntu2 ubuntu3 ubuntu4
------------------------------------------------------------------------------------------
vm_test scans all the VMs and reports any issues it encounters. For example, the abovevm_test report shows that SSH sessions are working fine but Java and Hazelcast are not installed on all of the VMs. To install Java and Hazelcast on the VMs, we execute the vm_install command as follows.
vm_install -java /home/dpark/Downloads/jdk-8u212-linux-x64.tar.gz \
-product /home/dpark/Downloads/hazelcast-enterprise-4.0.1.tar.gz
vm_install outputs the following:
Java installation
Local Source: /home/dpark/Downloads/jdk-8u212-linux-x64.tar.gz
Remote Target: /home/dpark/Hazelcast/jdk1.8.0_212
Installing ubuntu1...
Installing ubuntu2...
Installing ubuntu3...
Installing ubuntu4...
Java installation complete.
Hazelcast installation
Local Source: /home/dpark/Downloads/hazelcast-enterprise-4.0.1.tar.gz
Remote Target: /home/dpark/Hazelcast/hazelcast-enterprise-4.0.1
Installing ubuntu1...
Installing ubuntu2...
Installing ubuntu3...
Installing ubuntu4...
Hazelcast installation complete.
Run 'vm_test' to check installation and configuration status.
If you run vm_test after installing Java and Hazelcast, you should see no issues in the VM workspace.
vm_test
Output:
------------------------------------------------------------------------------------------
Workspace: ws-hz
Environment:
VM_ENABLED=true
VM_HOSTS=ubuntu1,ubuntu2,ubuntu3,ubuntu4
VM_USER=dpark
VM_PRIVATE_KEY_FILE=
VM_JAVA_HOME=/home/dpark/Hazelcast/jdk1.8.0_212
VM_HAZELCAST_HOME=/home/dpark/Hazelcast/hazelcast-enterprise-4.0.1
VM_PADOGRID_HOME=/home/dpark/Hazelcast/padogrid_0.9.7
VM_PADOGRID_WORKSPACE=/home/dpark/Hazelcast/workspaces/myrwe/ws-hz
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
Network Test:
ubuntu1 - OK
ubuntu2 - OK
ubuntu3 - OK
ubuntu4 - OK
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
/home/dpark/Hazelcast/padogrid_0.9.7
padogrid is not installed on the following VMs. It will be automatically installed
when you execute 'vm_sync'.
ubuntu1 ubuntu2 ubuntu3 ubuntu4
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
Summary:
No issues found. To sync the VM workspace, execute 'vm_sync'.
Workspace Issues:
None.
Network Issues:
None.
Software Issues:
None.
------------------------------------------------------------------------------------------
Hazelcast License Keys
For our example, we have installed Hazelcast Enterprise which requires a license key. The license keys are normally placed in the RWE .hazelcastenv.sh file ($PADOGRID_WORKSPACES_HOME/.hazelcastenv.sh), which is synchronized when you execute the vm_sync command (see next section). If you need to use license keys other than the ones set in this file then you can set them in the workspace's setenv.sh file, which overrides the environment variables set in the .hazelcastenv.sh file.
# Change directory to the current workspace, 'ws-hz', i.e., $PADOGRID_WORKSPACE
cd_workspace
vi setenv.sh
Add the license keys in setenv.sh to override .hazelcastenv.sh:
# Hazelcast IMDG license
IMDG_LICENSE_KEY=***
#Jet license
JET_LICENSE_KEY=***
# Management Center
MC_LICENSE_KEY=***
You can now proceed to sync the VM workspace.
Synchronize VM Workspace
In the previous section, we have locally created and tested a VM workspace. Before we can use it on the remote VMs, we need to synchronize it with the VMs. To do so, we execute the vm_sync command, which synchronizes the workspace you just created and automatically installs PadoGrid on all the VMs if it is not already installed.
vm_sync
vm_sync outputs the following:
Deploying padogrid_0.9.7 to ubuntu1...
Deploying padogrid_0.9.7 to ubuntu2...
Deploying padogrid_0.9.7 to ubuntu3...
Deploying padogrid_0.9.7 to ubuntu4...
Workspace sync: ws-hz
Synchronizing ubuntu1...
Synchronizing ubuntu2...
Synchronizing ubuntu3...
Synchronizing ubuntu4...
Workspace sync complete.
Configure Hazelcast Cluster
In our example, we have created the default cluster named, myhz. Let's switch into the myhz cluster.
switch_cluster myhz
The cluster directory has the following files.
myhz/
├── bin_sh
│ └── setenv.sh
├── etc
│ ├── cache.xml
│ ├── cluster.properties
│ ├── gemfire.properties
│ ├── hibernate.cfg-mysql.xml
│ ├── hibernate.cfg-postgresql.xml
│ ├── log4j2.properties
│ └── prometheus.yml
├── lib
├── log
├── plugins
└── run
etc/cluster.properties
In the myhz directory, you will find etc/cluster.properties which defines cluster-level properties. Let's edit this file to set the Management Center host (mc.host) to ubuntu4 and the cluster VMs (vm.hosts) to ubuntu1,ubuntu2,ubuntu3.
# Browse the cluster properties and change them as needed.
# Pay attention to the following two (2) properties:
# mc.host=
# vm.privateKeyFile=
vi etc/cluster.properties
# Set Management Center host
mc.host=ubuntu4
# Set the private key file path. If this property is set then it overwrites the
# VM_PRIVATE_KEY_FILE environment variable set in the workspace's setenv.sh file.
#vm.privateKeyFile=~/Hazelcast/Workspaces/ecs.pem
# Include additional VM host names. Host names (or IP addresses) must be
# comma separated with no spaces. Spaces are not supported.
vm.hosts=ubuntu1,ubuntu2,ubuntu3
etc/hazelcast.xml
You can also update the hazelcast.xml file at this time as needed.
vi etc/hazelcast.xml
Sync Changes
Any changes you made can be easily deployed to all the VMs specified by the VM_HOSTS environment variable in the $PADOGRID_WORKSPACE/setenv.sh by running the vm_sync command, which copies the entire workspace directory to the VMs. Be aware that this can take some time to complete if you have many VMs and large binary files in the workspace.
After the VMs have been synchronized, if you make changes to only a few files, then instead of executing vm_sync again, you can execute vm_copy which copies only the specified file or directory to the VMs.
switch_cluster myhz
# To copy a single file
vm_copy etc/cluster.properties
# To copy the entire directory
vm_copy .
vm_sync vs vm_copy
The main difference between vm_sync and vm_copy is that vm_sync copies the entire workspace and vm_copy copies only the specified file or directory. Both commands allow you to apply the changes you made to all of the VMs. As you might have guessed, vm_sync will take longer to complete since it copies the entire workspace, but it is more convenient to use than vm_copy since you don't have to remember the files you modified. These commands will quickly become your favorite commands if you often reconfigure Hazelcast clusters or update application library files.
:exclamation: Note that vm_copy by default only copies the files that are in the workspace directory hierarchy. If you try to copy a file that is outside of the workspace directory then it will fail and output an error message. If you need to copy non-workspace files, specify the -mirror option, which copies the absolute file path to the same absolute file path in the remote VMs.
Start Cluster
You are now ready to start the myhz cluster.
# Start cluster
start_cluster
# Start management center
start_mc
Monitor Cluster
To monitor the cluster:
show_cluster
show_mc
View Log
To view logs
# ubuntu1
show_log
# ubuntu2
show_log -num 2
Stop/Kill Cluster
To stop or kill the cluster:
stop_cluster
stop_mc
kill_cluster
Running from VMs
You can also run VM workspaces from any of the VMs. As with the local environment, you must first setup the PadoGrid environment on the VM in which you want to run workspaces by sourcing in the workspaces initenv.sh file.
# SSH into one of the VMs
ssh ubuntu1
# Source in intienv.sh
. ~/Hazelcast/workspaces/myrwe/initenv.sh
# Optionally add the above line in .bashrc (or .bash_profile) so that PadoGrid is
# automatically initialized when you login next time. Make sure to include the '-quiet' option.
echo ". ~/Hazelcast/workspaces/myrwe/initenv.sh -quiet" >> ~/.bash_rc
# Once you have initenv.sh sourced in you can then execute any of the PadoGrid
# commands as before. The following starts the `myhz` cluster.
switch_workspace ws-hz
swtich_cluster myhz
start_cluster
Teardown
If you want to remove the cluster from all the VMs, then you must first stop the cluster and execute the remove_cluster command. The workspace can be removed using the remove_workspace command.
# Stop cluster & management center
stop_cluster
stop_mc
# Remove cluster. Unlike other commands, this command requires the '-cluster' option.
remove_cluster -cluster myhz
# Simulate removing workspace from all VMs. Displays removal steps but does not
# actually remove the workspace.
remove_workspace -workspace ws-hz -simulate
# Remove workspace from all VMs. Runs in interactive mode.
remove_workspace -workspace ws-hz