Quick Start - padogrid/padogrid GitHub Wiki
◀️ PadoGrid in 5 Minutes :link: Introduction ▶️
⏰ In a hurry? Try PadoGrid in 5 Minutes.
Prerequisites
The following is a list of software required by PadoGrid.
Software | Required? | Description |
---|---|---|
bash |
Yes | PadoGrid supports only bash . |
curl |
Yes | Padogrid uses curl to read online contents. If you are using Cygwin, make sure you have curl installed. |
JDK 1.8+ | No | The JDK version range is defined by each product. All the products support up to JDK 11 with some supporting later versions as well. Java is not required if you are running Kubernetes or Docker apps. |
Maven 3.x+ | No | Maven is required only when installing bundles or apps. |
git |
No | Git is required only when installing bundles or apps. |
jq |
No | Some apps, e.g., grafana , depend on jq . |
Install PadoGrid
install_padogrid
Auto Installation: Download and run install_padogrid
. The following example uses the curl
command to download the install_padogrid
script.
# Download install_script
curl -sO https://raw.githubusercontent.com/padogrid/padogrid/develop/padogrid-deployment/src/main/resources/common/bin_sh/install_padogrid
# Make install_script executable
chmod 755 install_padogrid
# Run install_script
./install_padogrid
The install_padogrid
command interactively downloads and installs the products you selected from its menu. If you want to install non-interactively, specify the -quiet
option. This option downloads all the products and hence it may take some time to complete depending on your Internet download speed.
./install_padogrid -quiet
You can also install a single product by specifying the -product
option. For example, the following installs the latest Hazelcast OSS version. Run install_padogrid -?
to see the usage.
./install_padogrid -quiet -product hazelcast-oss
Manual Installation
If you are unable to run the install_padogrid
script perhaps because your target machine is behind a firewall, then follow the steps below.
- Create a directory where you will be creating the PadoGrid environment. For our example, we will use the default PadoGrid directory,
~/Padogrid
.
mkdir -p ~/Padogrid/downloads
- Download the latest PadoGrid distribution and place it in the
~/Padogrid/downloads
directory.
- Download products of your choice from the following link and place them in the
~/Padogrid/downloads
directory.
Supported Data Grid Products and Downloads
- Untar the downloaded product distributions. The following example shows a complete list of downloaded product distributions.
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/padogrid_0.9.32.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/apache-geode-1.13.2.tgz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/hadoop-3.3.1.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/hazelcast-4.2.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/hazelcast-enterprise-4.2.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/hazelcast-jet-4.5-slim.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/hazelcast-jet-enterprise-4.5.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/hazelcast-management-center-4.2021.04.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/mosquitto-2.0.15.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/pivotal-gemfire-9.10.5.tgz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/redis-7.0.2.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/redis-stack-server-6.2.2-v4.bionic.x86_64.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/snappydata-1.2.0-bin.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/spark-3.1.2-bin-hadoop3.2.tgz
unzip -d ~/Padogrid/products/ ~/Padogrid/downloads/fmw_14.1.1.0.0_coherence_quick_Disk1_1of1.zip
-
If you used
install_padogrid
to install Mosquitto, then you will need to build the distribution as follows.macOS
cd /home/dpark/Padogrid/products/mosquitto-2.0.10 export OPENSSL_ROOT_DIR=<openssl root directory path> cmake install -DDOCUMENTATION=NO -DWITH_PLUGINS=NO . make cd -
Linux (Debian)
cd /home/dpark/Padogrid/products/mosquitto-2.0.10 sudo apt install g++ libc-ares-dev libssl-dev libcjson-dev make cd -
If you prefer to install a binary distribution then follow the instructions in the following link. (Make sure the
mosquitto
executable is included inPATH
after the installation.) -
Redis OSS comes in the form of source code only. You will need to build the distribution once installed as follows.
cd ~/Padogrid/products/redis-7.0.2 make
-
Redis Stack distributions for macOS come without the top (home) directory. Make sure to create the home directory before unzipping the distribution. The directory name must be same as the distribution file name without the file extension as shown in the example below.
mkdir ~/Padogrid/products/redis-stack-server-6.2.2-v4.catalina.x86_64 unzip -d ~/Padogrid/products/redis-stack-server-6.2.2-v4.catalina.x86_64/ \ ~/Padogrid/downloads/redis-stack-server-6.2.2-v4.catalina.x86_64.zip
-
For Coherence, download the Quick Installer and follow the instructions in the link below.
Installing Oracle Coherence for Java
The following example shows how to install Coherence in the PadoGrid products directory.
unzip -d /tmp/ ~/Downloads/fmw_14.1.1.0.0_coherence_quick_Disk1_1of1.zip java -jar /tmp/fmw_14.1.1.0.0_coherence_quick.jar ORACLE_HOME=/home/dpark/Padogrid/products/Oracle rm /tmp/fmw_14.1.1.0.0_coherence_quick.jar
Upon installation, run
update_padogrid
and enter/home/dpark/Padogrid/products/Oracle/coherence
for the Coherence home path as shown below.update_padogrid
Output:
... Enter Coherence home path. Leave blank to skip. []: /home/dpark/Padogrid/products/Oracle/coherence ...
- Run
create_rwe
and follow the instructions.
~/Padogrid/products/padogrid_0.9.32/bin_sh/create_rwe
Initialize RWE
Once you have successfully installed PadoGrid, make sure to source in the RWE initenv.sh
file PadoGrid generated. It initializes your RWE-wide environment variables. You should also include it in ~/.bashrc
or ~/.bash_profile
so that it gets automatically initialized when you login next time.
# Initialize PadoGrid RWE
. ~/Padogrid/workspaces/myrwe/initenv.sh -quiet
# Run the following only if you did not use 'install_padogrid' to install
# PadoGrid.
echo ". ~/Padogrid/workspaces/myrwe/initenv.sh -quiet" >> ~/.bashrc
Java
Java is required to run most of the products. If Java is not installed then install any Java version supported by the product(s) that you want to run. The install_padogrid
usage provides the Java download links. Make sure to install JDK, not JRE.
install_padogrid -?
If Java is installed, you can set JAVA_HOME
in PadoGrid by running the interactive command, update_padogrid
, as follows.
update_padogrid -product java
This completes PadoGrid installation.
The subsequent sections provide directory structure details and workspace examples.
Installation Directory
By default, the install_padogrid
installs all the products in your home directory under ~/Padogrid
. All of your work such as workspaces and clusters will be stored in this directory. You can change the default directory by running interactively or move the directory after the installation completes. The following shows a typical view of the PadoGrid directory. Product distribution downloads are stored in the downloads
directory and they are installed in the products
directory. The snapshopts
directory is a place holder for future use and the workspaces
directory is for storing RWEs and their workspaces. You can safely remove the downloads at any time.
~/Padogrid
├── downloads
│ ├── apache-geode-1.13.2.tgz
│ ├── hazelcast-4.2.tar.gz
│ ├── hazelcast-enterprise-4.2.tar.gz
│ ├── hazelcast-jet-4.5-slim.tar.gz
│ ├── hazelcast-jet-enterprise-4.5.tar.gz
│ ├── hazelcast-management-center-4.2021.04.tar.gz
│ ├── hadoop-3.3.1.tar.gz
│ ├── mosquitto-2.0.15.tar.gz
│ ├── padogrid_0.9.32.tar.gz
│ ├── redis-7.0.2.tar.gz
│ ├── redis-stack-server-6.2.2-v4.bionic.x86_64.tar.gz
│ ├── snappydata-1.2.0-bin.tar.gz
│ └── spark-3.1.2-bin-hadoop3.2.tgz
├── products
│ ├── apache-geode-1.13.2
│ ├── hazelcast-4.2
│ ├── hazelcast-enterprise-4.2
│ ├── hazelcast-jet-4.5-slim
│ ├── hazelcast-jet-enterprise-4.5
│ ├── hazelcast-management-center-4.2021.04
│ ├── hadoop-3.3.1
│ ├── mosquitto-2.0.15
│ ├── padogrid_0.9.32
│ ├── redis-7.0.2
│ ├── redis-stack-server-6.2.2-v4.bionic.x86_64
│ ├── snappydata-1.2.0-bin
│ └── spark-3.1.2-bin-hadoop3.2
├── snapshots
└── workspaces
└── myrwe
Additional Products
Not all products can be installed by the install_paodgrid
command due to their online sign-in requirements. The following products must be downloaded manually. Click on them to download.
GemFire
To install GemFire, untar the downloaded file in the ~/Padogrid/products
directory as follows.
tar -C ~/Padogrid/products -xzf ~/Downloads/pivotal-gemfire-9.10.6.tgz
Coherence
Coherence installation is a bit involved. Plese see the Manual Installation section for instructions.
For our example, we will assume you have installed Coherence in the following directory:
~/Padogrid/products/Oracle/coherence
If you will be building Coherence apps using Maven, then you should also install the Coherence library files in the local Maven repository as described in the following link. Coherence is not available in public Maven repos.
Installing Coherence Library to Local Maven Repo
Directory Structure
If you installed all of the data grid products, you should have a directory structure similar to the following.
tree -L 2 ~/Padogrid/
Output:
~/Padogrid
├── downloads
├── products
│ ├── apache-geode-1.13.2
│ ├── hazelcast-4.2
│ ├── hazelcast-enterprise-4.2
│ ├── hazelcast-jet-4.5-slim
│ ├── hazelcast-jet-enterprise-4.5
│ ├── hazelcast-management-center-4.2021.04
│ ├── mosquitto-2.0.15
│ ├── Oracle/coherence
│ ├── padogrid_0.9.32
│ ├── pivotal-gemfire-9.10.6
│ ├── redis-7.0.2
│ └── spark-3.1.2-bin-hadoop3.2
├── snapshots
└── workspaces
└── myrwe
Your First Cluster
If you ran install_padogrid -quiet
then it created your first PadoGrid cluster, myrwe/myws/myhz
, in the workspaces
directory. By default, PadoGrid creates a cluster with the latest OSS version of Hazelcast if install_padogrid
is run in the non-interactive mode. Your directory structure should look similar to the following.
~/Padogrid/workspaces/
└── myrwe
├── initenv.sh
├── myws
│ ├── apps
│ ├── bundles
│ ├── clusters
│ │ └── myhz
│ │ ├── bin_sh
│ │ ├── etc
│ │ ├── lib
│ │ ├── log
│ │ ├── plugins
│ │ └── run
│ ├── docker
│ ├── initenv.sh
│ ├── k8s
│ ├── lib
│ ├── plugins
│ ├── pods
│ └── setenv.sh
└── setenv.sh
If you ran install_padogrid
without the -quiet
option, then you were prompted for which product to use for the first cluster.
View PadoGrid Environment
You can view the entire PadoGrid envrionment that includes all the RWEs and their workspaces you created by running padogrid
.
padogrid
Output:
.______ ___ _______ ______ _______ .______ __ _______ ™
| _ \ / \ | \ / __ \ / _____|| _ \ | | | \
| |_) | / ^ \ | .--. | | | | | | __ | |_) | | | | .--. |
| ___/ / /_\ \ | | | | | | | | | |_ | | / | | | | | |
| | / _____ \ | '--' | '--' | | |__| | | |\ \----.| | | '--' |
| _| /__/ \__\ |_______/ \______/ \______| | _| '._____||__| |_______/
Copyright (c) 2020-2023 Netcrest Technologies, LLC. All rights reserved.
Version: v0.9.32
Manual: https://github.com/padogrid/padogrid/wiki
Bundles: https://github.com/padogrid/catalog-bundles/blob/master/all-catalog.md
Root Workspaces Environments (RWEs)
-----------------------------------
/home/dpark/Padogrid/workspaces
└── myrwe
└── myws [java_1.8.0_265, padogrid_0.9.32]
Current Workspace
-----------------
PRODUCT: hazelcast
Product Version: 5.3.6
PADOGRID_WORKSPACE: /home/dpark/Padogrid/workspaces/myrwe/myws
CLUSTER: myhz
CLUSTER_TYPE: imdg
POD: local
POD_TYPE: local
Start Your First Cluster
PadoGrid provides unified lifecycle management commands for manging all the supported data grid products. For example, to start a cluster, you would use start_cluster
no matter what the data grid product is. The following shows how to start the myhz
cluster created by install_padogrid
.
# 1. The 'install_padogrid' command created the default workspace named 'myws' and
# the cluster you specified, i.e., 'myhz'.
switch_workspace myws
switch_cluster myhz
# Start both Hazelcast cluster and Management Center
start_cluster -all
# 2. Check cluster status and member log. Use the management center URL
# it displays to monitor the cluster.
show_cluster -long
show_log
# 3. Create (or install) the perf_test app. To install other apps, use the '-app' option.
# To install the same app under different name, use the '-name' option.
create_app
# 4. Change directory to perf_test and run its scripts to ingest mock data.
cd_app perf_test; cd bin_sh
./test_ingestion -run
./test_tx -run
# 5. Stop both cluster and management center
stop_cluster -all
Hazelcast Enterprise
If you are running Hazelcast Enterprise or Jet Enterprise, then you need to enter your Hazelcast enterprise license keys in the RWE .hazelcastenv.sh
file as shown below. Other products do not require license keys.
vi ~/Padogrid/workspaces/myrwe/.hazelcastenv.sh
switch_rwe myrwe
vi .hazelcastenv.sh
IMDG_LICENSE_KEY=***
JET_LICENSE_KEY=***
MC_LICENSE_KEY=***
Adding or Changing Products
PadoGrid maintains RWE and workspace configuration information in the setenv.sh
file located in their respective directory. The environement variables you set in the RWE setenv.sh
file apply to all of its workspaces. In each workspace, you can ovrride any of the environment variables in their setenv.sh
file.
setenv.sh
RWE You can change or add product paths in RWE as follows.
cd_rwe
vi setenv.sh
The RWE setenv.sh
has contents similar to the following. You can freely set the paths in this file. Note that the IN_POD
and PRODUCT_HOME
environment variables shown below are reserved. IN_POD
indicates whether the clusters are running inside or outside of a pod (Vagrant). Hence, the home paths set in the if
block below are for clusters running outside of pods, i.e., on your host OS.
:exclamation: ThePRODUCT_HOME
environment variable must be set to one of the product home paths. It is used internally to identify the default product.
if [ "$IN_POD" != "true" ]; then
export PADOGRID_HOME="/home/dpark/Padogrid/products/padogrid_0.9.32"
export PADO_HOME=""
export JAVA_HOME="/etc/alternatives/java_sdk_1.8.0"
export COHERENCE_HOME="/home/dpark/Padogrid/products/Oracle/coherence"
export GEMFIRE_HOME=""
export GEODE_HOME="/home/dpark/Padogrid/products/apache-geode-1.13.2"
export HAZELCAST_HOME="/home/dpark/Padogrid/products/hazelcast-4.2"
export HAZELCAST_MC_HOME="/home/dpark/Padogrid/products/hazelcast-management-center-4.2021.04"
export JET_HOME="/home/dpark/Padogrid/products/hazelcast-jet-4.5-slim"
export JET_MC_HOME=""
export MOSQUITTO_HOME=""
export REDIS_HOME=""
export SNAPPYDATA_HOME=""
export SPARK_HOME="/home/dpark/Padogrid/products/spark-3.1.2"
export HADOOP_HOME="/home/dpark/Padogrid/products/hadoop-3.3.1"
export PRODUCT_HOME="$HAZELCAST_HOME"
fi
setenv.sh
Workspace To change product paths specific to your workspace, you would set them in the workspace setenv.sh
file.
cd_workspace
vi setenv.sh
The workspace setenv.sh
file has contents similar to the following. You can freely set the paths in this file. The following example downgrades Hazelcast from 4.2 (see above) to 4.1.2. All others are left unchanged to use the default values set in the RWE setenv.sh
file.
if [ "$IN_POD" != "true" ]; then
#export PADOGRID_HOME=
#export PADO_HOME=
#export JAVA_HOME=
export HAZELCAST_HOME="/home/dpark/Padogrid/products/hazelcast-4.1.2"
export HAZELCAST_MC_HOME="$HAZELCAST_HOME/management-center"
#export JET_HOME=""
#export JET_MC_HOME=""
#export COHERENCE_HOME=""
#export GEMFIRE_HOME=""
#export GEODE_HOME=""
#export MOSQUITTO_HOME=""
#export REDIS_HOME=""
#export SNAPPYDATA_HOME=""
#export SPARK_HOME=""
#export HADOOP_HOME=""
fi
update_padogrid
Command
The update_padogrid
command allows you to update products without manually editing the setenv.sh
files. This is an interactive command that lists products from which you can select the ones you want to apply to the current RWE, the current workspace, or all the workspaces in the current RWE.
update_padogrid
Example Output:
Products Directoy:
/home/dpark/Padogrid/products
The following product versions are found in the products directory.
The current workspace versions are highlighted.
PadoGrid
[0] (none)
[1] 0.9.32
[2] 0.9.31
[3] 0.9.30
[4] 0.9.29
...
Enter a version to add [0]: 1
Selected: [1] 0.9.32
Pado
[0] (none)
Enter a version to add [0]:
Selected: [0] (none)
GemFire
[0] (none)
[1] 9.10.5
[2] 9.10.4
[3] 9.10.2
Enter a version to add [0]: 1
Selected: [1] 9.10.5
Geode
[0] (none)
[1] 1.13.2
[2] 1.13.1
[3] 1.13.0
[4] 1.12.0
Enter a version to add [0]:
...
You can also update a single product by specifying the -product
option. For example, the following updates the Hazelcast OSS product version.
update_padogrid -product hazelcast-oss
Output:
Products Directoy:
/home/dpark/Padogrid/products
The following product versions are found in the products directory.
The current workspace versions are highlighted.
Hazelcast OSS
[0] (none)
[1] 4.2.1
[2] 4.2
[3] 4.1.2
[4] 4.1
[5] 4.0.3
[6] 4.0.2
[7] 3.8.1
[8] 3.8
Enter a version to add [0]:
Run the following to view the usage.
update_padogrid -?
Creating and Starting Clusters
You can create any number of clusters in a workspace using the create_cluster
command. This section shows how to create and start a cluster for each product. The commands used in this section are listed below.
Command | Decription |
---|---|
create_cluster |
Create new clusters. Use the -product option to specify the product name. |
switch_cluster |
Switch into the specified cluster. |
start_cluster |
Start the current (or switched) cluster. |
show_cluster |
Show the current cluster status. |
show_mc |
Show the current Hazelcast management center status. |
stop_cluster |
Stop the current cluster. |
✏️ Note that instead of running these commands for the current cluster, you can also use the -cluster
option to specify the cluster that you want to target.
If you have not installed the the products described in the subequent sections, then run install_padogrid
or manually install them and include their paths in the RWE setenv.sh
file or the workspace setenv.sh
file as described in the Adding or Changing Products section.
Geode
# Create the default 'mygeode' cluster.
create_cluster -product geode
# Switch into the 'mygeode' cluster
switch_cluster mygeode
# Start `mygeode' cluster
start_cluster
# Show detailed cluster status
show_cluster -long
The show_cluster
command should display the Pulse URL as shown below. Enter the URL in the browser to monitor the cluster.
Pulse URL: http://localhost:7070/pulse
GemFire
# Create the default 'mygemfire' cluster.
create_cluster -product gemfire
# Switch into the 'mygemfire' cluster
switch_cluster mygemfire
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long
The show_cluster
command should display the Pulse URL as shown below. Enter the URL in the browser to monitor the cluster.
Pulse URL: http://localhost:7070/pulse
Hazelcast Platform/IMDG
# Create the default 'myhz' cluster.
create_cluster -product hazelcast
# Switch into the 'myhz' cluster
switch_cluster myhz
# Start the current (switched) cluster and management center
start_cluster -all
# Show detailed cluster status
show_cluster -long
The show_cluster
command should display the Management Center URL shown below. Enter the URL in the browser to monitor the cluster.
Management Center URL: http://localhost:8080/hazelcast-mancenter
Hazelcast Jet
# Create the default 'myjet' cluster.
create_cluster -product jet
# Switch into the 'myjet' cluster
switch_cluster myjet
# Start the current (switched) cluster and management center
start_cluster -all
# Show detailed cluster status
show_cluster -long
# Show management center status
show_mc
The show_mc
command is specific to Hazelcast Platform/IMDG and Jet. It should display the Management Center URL shown below. Enter the URL in the browser to monitor the cluster.
Management Center URL: http://localhost:8080
Redis
# Create the default 'myredis' cluster.
create_cluster -product redis
# Switch into the 'myredis' cluster
switch_cluster myredis
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long
SnappyData
# Create the default 'mysnappy' cluster.
create_cluster -product snappydata
# Switch into the 'mysnappy' cluster
switch_cluster mysnappy
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long
The show_cluster
command should display the Dashboard URL as shown below. Enter the URL in the browser to monitor the cluster.
Dashboard: http://localhost:5050/dashboard
Coherence
# Create the default 'mycoherence' cluster.
create_cluster -product coherence
# Switch into the 'mycoherence' cluster
switch_cluster mycoherence
# Start `mycoherence' cluster
start_cluster
# Show detailed cluster status
show_cluster -long
Coherence does not provide a web console but provides a JVisualVM plug-in for monitoring Coherence cluster members. Please follow the instructions in the following link.
Hadoop
# Create the default 'myhadoop' cluster.
create_cluster -product hadoop
# Switch into the 'myhadoop' cluster
switch_cluster myhadoop
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long
The show_cluster
command should display the NameNode URL as shown below. Enter the URL in the browser to monitor the cluster.
NameNode: http://localhost:9870
Kafka
# Create the default 'mykafka' cluster.
create_cluster -product kafka
# Switch into the 'mykafka' cluster
switch_cluster mykafka
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long
✏️ To monitor Kafka, download Confluent monitoring tools.
Kafka Confluent
# Create the default 'mykafka' cluster.
create_cluster -product confluent
# Switch into the 'myconfluent' cluster
switch_cluster myconfluent
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long
✏️ To monitor Kafka, download Confluent monitoring tools.
Mosquitto
# Create the default 'mymosquitto' cluster.
create_cluster -product mosquitto
# Switch into the 'mymosquitto' cluster
switch_cluster mymosquitto
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long
Spark
# Create the default 'myspark' cluster.
create_cluster -product spark
# Switch into the 'myspark' cluster
switch_cluster myspark
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long
The show_cluster
command should display the Master Web UI URL as shown below. Enter the URL in the browser to monitor the cluster.
Master Web UI: http://localhost:8080
Teardown
To stop the current cluster, run the stop_cluster
command.
stop_cluster -all