Quick Start - padogrid/padogrid GitHub Wiki

◀️ PadoGrid in 5 Minutes :link: Introduction ▶️


⏰ In a hurry? Try PadoGrid in 5 Minutes.


Prerequisites

The following is a list of software required by PadoGrid.

Software Required? Description
bash Yes PadoGrid supports only bash.
curl Yes Padogrid uses curl to read online contents. If you are using Cygwin, make sure you have curl installed.
JDK 1.8+ No The JDK version range is defined by each product. All the products support up to JDK 11 with some supporting later versions as well. Java is not required if you are running Kubernetes or Docker apps.
Maven 3.x+ No Maven is required only when installing bundles or apps.
git No Git is required only when installing bundles or apps.
jq No Some apps, e.g., grafana, depend on jq.

Install PadoGrid

Auto Installation: install_padogrid

Download and run install_padogrid. The following example uses the curl command to download the install_padogrid script.

# Download install_script
curl -sO https://raw.githubusercontent.com/padogrid/padogrid/develop/padogrid-deployment/src/main/resources/common/bin_sh/install_padogrid
# Make install_script executable
chmod 755 install_padogrid
# Run install_script
./install_padogrid

The install_padogrid command interactively downloads and installs the products you selected from its menu. If you want to install non-interactively, specify the -quiet option. This option downloads all the products and hence it may take some time to complete depending on your Internet download speed.

./install_padogrid -quiet

You can also install a single product by specifying the -product option. For example, the following installs the latest Hazelcast OSS version. Run install_padogrid -? to see the usage.

./install_padogrid -quiet -product hazelcast-oss

Manual Installation

If you are unable to run the install_padogrid script perhaps because your target machine is behind a firewall, then follow the steps below.

  1. Create a directory where you will be creating the PadoGrid environment. For our example, we will use the default PadoGrid directory, ~/Padogrid.
mkdir -p ~/Padogrid/downloads
  1. Download the latest PadoGrid distribution and place it in the ~/Padogrid/downloads directory.

PadoGrid Releases

  1. Download products of your choice from the following link and place them in the ~/Padogrid/downloads directory.

Supported Data Grid Products and Downloads

  1. Untar the downloaded product distributions. The following example shows a complete list of downloaded product distributions.
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/padogrid_0.9.32.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/apache-geode-1.13.2.tgz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/hadoop-3.3.1.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/hazelcast-4.2.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/hazelcast-enterprise-4.2.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/hazelcast-jet-4.5-slim.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/hazelcast-jet-enterprise-4.5.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/hazelcast-management-center-4.2021.04.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/mosquitto-2.0.15.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/pivotal-gemfire-9.10.5.tgz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/redis-7.0.2.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/redis-stack-server-6.2.2-v4.bionic.x86_64.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/snappydata-1.2.0-bin.tar.gz
tar -C ~/Padogrid/products/ -xzf ~/Padogrid/downloads/spark-3.1.2-bin-hadoop3.2.tgz
unzip -d ~/Padogrid/products/ ~/Padogrid/downloads/fmw_14.1.1.0.0_coherence_quick_Disk1_1of1.zip
  • If you used install_padogrid to install Mosquitto, then you will need to build the distribution as follows.

    macOS

    cd /home/dpark/Padogrid/products/mosquitto-2.0.10
    export OPENSSL_ROOT_DIR=<openssl root directory path>
    cmake install -DDOCUMENTATION=NO -DWITH_PLUGINS=NO .
    make
    cd -
    

    Linux (Debian)

    cd /home/dpark/Padogrid/products/mosquitto-2.0.10
    sudo apt install g++ libc-ares-dev libssl-dev libcjson-dev
    make
    cd -
    

    If you prefer to install a binary distribution then follow the instructions in the following link. (Make sure the mosquitto executable is included in PATH after the installation.)

    https://mosquitto.org/download/

  • Redis OSS comes in the form of source code only. You will need to build the distribution once installed as follows.

    cd ~/Padogrid/products/redis-7.0.2
    make
    
  • Redis Stack distributions for macOS come without the top (home) directory. Make sure to create the home directory before unzipping the distribution. The directory name must be same as the distribution file name without the file extension as shown in the example below.

    mkdir ~/Padogrid/products/redis-stack-server-6.2.2-v4.catalina.x86_64
    unzip -d ~/Padogrid/products/redis-stack-server-6.2.2-v4.catalina.x86_64/ \
             ~/Padogrid/downloads/redis-stack-server-6.2.2-v4.catalina.x86_64.zip
    
  • For Coherence, download the Quick Installer and follow the instructions in the link below.

    Installing Oracle Coherence for Java

    The following example shows how to install Coherence in the PadoGrid products directory.

    unzip -d /tmp/ ~/Downloads/fmw_14.1.1.0.0_coherence_quick_Disk1_1of1.zip
    java -jar /tmp/fmw_14.1.1.0.0_coherence_quick.jar ORACLE_HOME=/home/dpark/Padogrid/products/Oracle
    rm /tmp/fmw_14.1.1.0.0_coherence_quick.jar
    

    Upon installation, run update_padogrid and enter /home/dpark/Padogrid/products/Oracle/coherence for the Coherence home path as shown below.

    update_padogrid
    

    Output:

    ...
    Enter Coherence home path. Leave blank to skip.
    []:
    /home/dpark/Padogrid/products/Oracle/coherence
    ...
    
    
  1. Run create_rwe and follow the instructions.
~/Padogrid/products/padogrid_0.9.32/bin_sh/create_rwe

Initialize RWE

Once you have successfully installed PadoGrid, make sure to source in the RWE initenv.sh file PadoGrid generated. It initializes your RWE-wide environment variables. You should also include it in ~/.bashrc or ~/.bash_profile so that it gets automatically initialized when you login next time.

# Initialize PadoGrid RWE
. ~/Padogrid/workspaces/myrwe/initenv.sh -quiet

# Run the following only if you did not use 'install_padogrid' to install
# PadoGrid.
echo ". ~/Padogrid/workspaces/myrwe/initenv.sh -quiet" >> ~/.bashrc

Java

Java is required to run most of the products. If Java is not installed then install any Java version supported by the product(s) that you want to run. The install_padogrid usage provides the Java download links. Make sure to install JDK, not JRE.

install_padogrid -?

If Java is installed, you can set JAVA_HOME in PadoGrid by running the interactive command, update_padogrid, as follows.

update_padogrid -product java

This completes PadoGrid installation.

The subsequent sections provide directory structure details and workspace examples.


Installation Directory

By default, the install_padogrid installs all the products in your home directory under ~/Padogrid. All of your work such as workspaces and clusters will be stored in this directory. You can change the default directory by running interactively or move the directory after the installation completes. The following shows a typical view of the PadoGrid directory. Product distribution downloads are stored in the downloads directory and they are installed in the products directory. The snapshopts directory is a place holder for future use and the workspaces directory is for storing RWEs and their workspaces. You can safely remove the downloads at any time.

~/Padogrid
├── downloads
│   ├── apache-geode-1.13.2.tgz
│   ├── hazelcast-4.2.tar.gz
│   ├── hazelcast-enterprise-4.2.tar.gz
│   ├── hazelcast-jet-4.5-slim.tar.gz
│   ├── hazelcast-jet-enterprise-4.5.tar.gz
│   ├── hazelcast-management-center-4.2021.04.tar.gz
│   ├── hadoop-3.3.1.tar.gz
│   ├── mosquitto-2.0.15.tar.gz
│   ├── padogrid_0.9.32.tar.gz
│   ├── redis-7.0.2.tar.gz
│   ├── redis-stack-server-6.2.2-v4.bionic.x86_64.tar.gz
│   ├── snappydata-1.2.0-bin.tar.gz
│   └── spark-3.1.2-bin-hadoop3.2.tgz
├── products
│   ├── apache-geode-1.13.2
│   ├── hazelcast-4.2
│   ├── hazelcast-enterprise-4.2
│   ├── hazelcast-jet-4.5-slim
│   ├── hazelcast-jet-enterprise-4.5
│   ├── hazelcast-management-center-4.2021.04
│   ├── hadoop-3.3.1
│   ├── mosquitto-2.0.15
│   ├── padogrid_0.9.32
│   ├── redis-7.0.2
│   ├── redis-stack-server-6.2.2-v4.bionic.x86_64
│   ├── snappydata-1.2.0-bin
│   └── spark-3.1.2-bin-hadoop3.2
├── snapshots
└── workspaces
    └── myrwe

Additional Products

Not all products can be installed by the install_paodgrid command due to their online sign-in requirements. The following products must be downloaded manually. Click on them to download.

GemFire

To install GemFire, untar the downloaded file in the ~/Padogrid/products directory as follows.

tar -C ~/Padogrid/products -xzf ~/Downloads/pivotal-gemfire-9.10.6.tgz

Coherence

Coherence installation is a bit involved. Plese see the Manual Installation section for instructions.

For our example, we will assume you have installed Coherence in the following directory:

~/Padogrid/products/Oracle/coherence

If you will be building Coherence apps using Maven, then you should also install the Coherence library files in the local Maven repository as described in the following link. Coherence is not available in public Maven repos.

Installing Coherence Library to Local Maven Repo

Directory Structure

If you installed all of the data grid products, you should have a directory structure similar to the following.

tree -L 2 ~/Padogrid/

Output:

~/Padogrid
├── downloads
├── products
│   ├── apache-geode-1.13.2
│   ├── hazelcast-4.2
│   ├── hazelcast-enterprise-4.2
│   ├── hazelcast-jet-4.5-slim
│   ├── hazelcast-jet-enterprise-4.5
│   ├── hazelcast-management-center-4.2021.04
│   ├── mosquitto-2.0.15
│   ├── Oracle/coherence
│   ├── padogrid_0.9.32
│   ├── pivotal-gemfire-9.10.6
│   ├── redis-7.0.2
│   └── spark-3.1.2-bin-hadoop3.2
├── snapshots
└── workspaces
    └── myrwe

Your First Cluster

If you ran install_padogrid -quiet then it created your first PadoGrid cluster, myrwe/myws/myhz, in the workspaces directory. By default, PadoGrid creates a cluster with the latest OSS version of Hazelcast if install_padogrid is run in the non-interactive mode. Your directory structure should look similar to the following.

~/Padogrid/workspaces/
└── myrwe
    ├── initenv.sh
    ├── myws
    │   ├── apps
    │   ├── bundles
    │   ├── clusters
    │   │   └── myhz
    │   │       ├── bin_sh
    │   │       ├── etc
    │   │       ├── lib
    │   │       ├── log
    │   │       ├── plugins
    │   │       └── run
    │   ├── docker
    │   ├── initenv.sh
    │   ├── k8s
    │   ├── lib
    │   ├── plugins
    │   ├── pods
    │   └── setenv.sh
    └── setenv.sh

If you ran install_padogrid without the -quiet option, then you were prompted for which product to use for the first cluster.

View PadoGrid Environment

You can view the entire PadoGrid envrionment that includes all the RWEs and their workspaces you created by running padogrid.

padogrid

Output:

.______      ___       _______   ______     _______ .______       __   _______ ™
|   _  \    /   \     |       \ /  __  \   /  _____||   _  \     |  | |       \
|  |_)  |  /  ^  \    |  .--.  |  |  |  | |  |  __  |  |_)  |    |  | |  .--.  |
|   ___/  /  /_\  \   |  |  |  |  |  |  | |  | |_ | |      /     |  | |  |  |  |
|  |     /  _____  \  |  '--'  |  '--'  | |  |__| | |  |\  \----.|  | |  '--'  |
| _|    /__/     \__\ |_______/ \______/   \______| | _| '._____||__| |_______/
Copyright (c) 2020-2023 Netcrest Technologies, LLC. All rights reserved.
Version: v0.9.32
 Manual: https://github.com/padogrid/padogrid/wiki
Bundles: https://github.com/padogrid/catalog-bundles/blob/master/all-catalog.md

Root Workspaces Environments (RWEs)
-----------------------------------
/home/dpark/Padogrid/workspaces
└── myrwe
    └── myws [java_1.8.0_265, padogrid_0.9.32]

Current Workspace
-----------------
           PRODUCT: hazelcast
   Product Version: 5.3.6
PADOGRID_WORKSPACE: /home/dpark/Padogrid/workspaces/myrwe/myws
           CLUSTER: myhz
      CLUSTER_TYPE: imdg
               POD: local
          POD_TYPE: local

Start Your First Cluster

PadoGrid provides unified lifecycle management commands for manging all the supported data grid products. For example, to start a cluster, you would use start_cluster no matter what the data grid product is. The following shows how to start the myhz cluster created by install_padogrid.

# 1. The 'install_padogrid' command created the default workspace named 'myws' and
#    the cluster you specified, i.e., 'myhz'.
switch_workspace myws
switch_cluster myhz
# Start both Hazelcast cluster and Management Center
start_cluster -all

# 2. Check cluster status and member log. Use the management center URL
#    it displays to monitor the cluster.
show_cluster -long
show_log

# 3. Create (or install) the perf_test app. To install other apps, use the '-app' option.
#    To install the same app under different name, use the '-name' option.
create_app

# 4. Change directory to perf_test and run its scripts to ingest mock data.
cd_app perf_test; cd bin_sh
./test_ingestion -run
./test_tx -run

# 5. Stop both cluster and management center
stop_cluster -all

Hazelcast Enterprise

If you are running Hazelcast Enterprise or Jet Enterprise, then you need to enter your Hazelcast enterprise license keys in the RWE .hazelcastenv.sh file as shown below. Other products do not require license keys.

vi ~/Padogrid/workspaces/myrwe/.hazelcastenv.sh
switch_rwe myrwe
vi .hazelcastenv.sh
IMDG_LICENSE_KEY=***
JET_LICENSE_KEY=***
MC_LICENSE_KEY=***

Adding or Changing Products

PadoGrid maintains RWE and workspace configuration information in the setenv.sh file located in their respective directory. The environement variables you set in the RWE setenv.sh file apply to all of its workspaces. In each workspace, you can ovrride any of the environment variables in their setenv.sh file.

RWE setenv.sh

You can change or add product paths in RWE as follows.

cd_rwe
vi setenv.sh

The RWE setenv.sh has contents similar to the following. You can freely set the paths in this file. Note that the IN_POD and PRODUCT_HOME environment variables shown below are reserved. IN_POD indicates whether the clusters are running inside or outside of a pod (Vagrant). Hence, the home paths set in the if block below are for clusters running outside of pods, i.e., on your host OS.

:exclamation: ThePRODUCT_HOME environment variable must be set to one of the product home paths. It is used internally to identify the default product.

if [ "$IN_POD" != "true" ]; then
   export PADOGRID_HOME="/home/dpark/Padogrid/products/padogrid_0.9.32"
   export PADO_HOME=""
   export JAVA_HOME="/etc/alternatives/java_sdk_1.8.0"
   export COHERENCE_HOME="/home/dpark/Padogrid/products/Oracle/coherence"
   export GEMFIRE_HOME=""
   export GEODE_HOME="/home/dpark/Padogrid/products/apache-geode-1.13.2"
   export HAZELCAST_HOME="/home/dpark/Padogrid/products/hazelcast-4.2"
   export HAZELCAST_MC_HOME="/home/dpark/Padogrid/products/hazelcast-management-center-4.2021.04"
   export JET_HOME="/home/dpark/Padogrid/products/hazelcast-jet-4.5-slim"
   export JET_MC_HOME=""
   export MOSQUITTO_HOME=""
   export REDIS_HOME=""
   export SNAPPYDATA_HOME=""
   export SPARK_HOME="/home/dpark/Padogrid/products/spark-3.1.2"
   export HADOOP_HOME="/home/dpark/Padogrid/products/hadoop-3.3.1"
   export PRODUCT_HOME="$HAZELCAST_HOME"
fi

Workspace setenv.sh

To change product paths specific to your workspace, you would set them in the workspace setenv.sh file.

cd_workspace
vi setenv.sh

The workspace setenv.sh file has contents similar to the following. You can freely set the paths in this file. The following example downgrades Hazelcast from 4.2 (see above) to 4.1.2. All others are left unchanged to use the default values set in the RWE setenv.sh file.

if [ "$IN_POD" != "true" ]; then
  #export PADOGRID_HOME=
  #export PADO_HOME=
  #export JAVA_HOME=
  export HAZELCAST_HOME="/home/dpark/Padogrid/products/hazelcast-4.1.2"
  export HAZELCAST_MC_HOME="$HAZELCAST_HOME/management-center"
  #export JET_HOME=""
  #export JET_MC_HOME=""
  #export COHERENCE_HOME=""
  #export GEMFIRE_HOME=""
  #export GEODE_HOME=""
  #export MOSQUITTO_HOME=""
  #export REDIS_HOME=""
  #export SNAPPYDATA_HOME=""
  #export SPARK_HOME=""
  #export HADOOP_HOME=""
fi

update_padogrid Command

The update_padogrid command allows you to update products without manually editing the setenv.sh files. This is an interactive command that lists products from which you can select the ones you want to apply to the current RWE, the current workspace, or all the workspaces in the current RWE.

update_padogrid

Example Output:

Products Directoy:
   /home/dpark/Padogrid/products

The following product versions are found in the products directory.
The current workspace versions are highlighted.

PadoGrid
   [0] (none)
   [1] 0.9.32
   [2] 0.9.31
   [3] 0.9.30
   [4] 0.9.29
   ...
Enter a version to add [0]: 1
Selected: [1] 0.9.32

Pado
   [0] (none)
Enter a version to add [0]:
Selected: [0] (none)

GemFire
   [0] (none)
   [1] 9.10.5
   [2] 9.10.4
   [3] 9.10.2
Enter a version to add [0]: 1
Selected: [1] 9.10.5

Geode
   [0] (none)
   [1] 1.13.2
   [2] 1.13.1
   [3] 1.13.0
   [4] 1.12.0
Enter a version to add [0]:

...

You can also update a single product by specifying the -product option. For example, the following updates the Hazelcast OSS product version.

update_padogrid -product hazelcast-oss

Output:

Products Directoy:
   /home/dpark/Padogrid/products

The following product versions are found in the products directory.
The current workspace versions are highlighted.

Hazelcast OSS
   [0] (none)
   [1] 4.2.1
   [2] 4.2
   [3] 4.1.2
   [4] 4.1
   [5] 4.0.3
   [6] 4.0.2
   [7] 3.8.1
   [8] 3.8
Enter a version to add [0]:

Run the following to view the usage.

update_padogrid -?

Creating and Starting Clusters

You can create any number of clusters in a workspace using the create_cluster command. This section shows how to create and start a cluster for each product. The commands used in this section are listed below.

Command Decription
create_cluster Create new clusters. Use the -product option to specify the product name.
switch_cluster Switch into the specified cluster.
start_cluster Start the current (or switched) cluster.
show_cluster Show the current cluster status.
show_mc Show the current Hazelcast management center status.
stop_cluster Stop the current cluster.

✏️ Note that instead of running these commands for the current cluster, you can also use the -cluster option to specify the cluster that you want to target.

If you have not installed the the products described in the subequent sections, then run install_padogrid or manually install them and include their paths in the RWE setenv.sh file or the workspace setenv.sh file as described in the Adding or Changing Products section.

Geode

# Create the default 'mygeode' cluster.
create_cluster -product geode
# Switch into the 'mygeode' cluster 
switch_cluster mygeode
# Start `mygeode' cluster
start_cluster
# Show detailed cluster status
show_cluster -long

The show_cluster command should display the Pulse URL as shown below. Enter the URL in the browser to monitor the cluster.

Pulse URL: http://localhost:7070/pulse

GemFire

# Create the default 'mygemfire' cluster.
create_cluster -product gemfire
# Switch into the 'mygemfire' cluster 
switch_cluster mygemfire
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long

The show_cluster command should display the Pulse URL as shown below. Enter the URL in the browser to monitor the cluster.

Pulse URL: http://localhost:7070/pulse

Hazelcast Platform/IMDG

# Create the default 'myhz' cluster.
create_cluster -product hazelcast
# Switch into the 'myhz' cluster 
switch_cluster myhz
# Start the current (switched) cluster and management center
start_cluster -all
# Show detailed cluster status
show_cluster -long

The show_cluster command should display the Management Center URL shown below. Enter the URL in the browser to monitor the cluster.

Management Center URL: http://localhost:8080/hazelcast-mancenter

Hazelcast Jet

# Create the default 'myjet' cluster.
create_cluster -product jet
# Switch into the 'myjet' cluster 
switch_cluster myjet
# Start the current (switched) cluster and management center
start_cluster -all
# Show detailed cluster status
show_cluster -long
# Show management center status
show_mc

The show_mc command is specific to Hazelcast Platform/IMDG and Jet. It should display the Management Center URL shown below. Enter the URL in the browser to monitor the cluster.

Management Center URL: http://localhost:8080

Redis

# Create the default 'myredis' cluster.
create_cluster -product redis
# Switch into the 'myredis' cluster 
switch_cluster myredis
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long

SnappyData

# Create the default 'mysnappy' cluster.
create_cluster -product snappydata
# Switch into the 'mysnappy' cluster 
switch_cluster mysnappy
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long

The show_cluster command should display the Dashboard URL as shown below. Enter the URL in the browser to monitor the cluster.

Dashboard: http://localhost:5050/dashboard

Coherence

# Create the default 'mycoherence' cluster.
create_cluster -product coherence
# Switch into the 'mycoherence' cluster 
switch_cluster mycoherence
# Start `mycoherence' cluster
start_cluster
# Show detailed cluster status
show_cluster -long

Coherence does not provide a web console but provides a JVisualVM plug-in for monitoring Coherence cluster members. Please follow the instructions in the following link.

Monitoring Coherence

Hadoop

# Create the default 'myhadoop' cluster.
create_cluster -product hadoop
# Switch into the 'myhadoop' cluster 
switch_cluster myhadoop
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long

The show_cluster command should display the NameNode URL as shown below. Enter the URL in the browser to monitor the cluster.

NameNode: http://localhost:9870

Kafka

# Create the default 'mykafka' cluster.
create_cluster -product kafka
# Switch into the 'mykafka' cluster 
switch_cluster mykafka
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long

✏️ To monitor Kafka, download Confluent monitoring tools.

Kafka Confluent

# Create the default 'mykafka' cluster.
create_cluster -product confluent
# Switch into the 'myconfluent' cluster 
switch_cluster myconfluent
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long

✏️ To monitor Kafka, download Confluent monitoring tools.

Mosquitto

# Create the default 'mymosquitto' cluster.
create_cluster -product mosquitto
# Switch into the 'mymosquitto' cluster 
switch_cluster mymosquitto
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long

Spark

# Create the default 'myspark' cluster.
create_cluster -product spark
# Switch into the 'myspark' cluster 
switch_cluster myspark
# Start the current (switched) cluster
start_cluster
# Show detailed cluster status
show_cluster -long

The show_cluster command should display the Master Web UI URL as shown below. Enter the URL in the browser to monitor the cluster.

Master Web UI: http://localhost:8080

Teardown

To stop the current cluster, run the stop_cluster command.

stop_cluster -all

◀️ PadoGrid in 5 Minutes :link: Introduction ▶️