Multitenancy - padogrid/padogrid Wiki

◀️ VM-Enabled Pods :link: Multitenancy Best Practices ▶️


Since Padogrid v0.9.20


PadoGrid by design includes support for multi-tenant distributed workspaces, allowing a user or a group of users to own a workspace or a sandbox environment that is completely isolated from other workspaces.

We define the term workspace owner as one or more users that have full permissions (RWX) to a given workspace. Each workspace is independently managed and distributed by the workspace owner. This capability allows each workspace to branch out on their own to allocate system resources based on their needs. Some multi-tenant workspace examples are listed below.

  • A developer owns a workspace that requires only local resources such as a small Hazelcast cluster with masked data ingested.
  • A data scientist owns a workspace that spans several nodes with data analytics tools such as GemFire, Spark, and JupyterLab.
  • An application team owns a workspace that spans several nodes integrating many endpoints such as data sources, middleware, application server, load-balancer, and microservices.
  • A support team owns several workspaces for testing new releases, reproducing problems, capturing timelines of events, archiving system images, and etc.
  • A production team owns several workspaces, each dedicated for a datacenter.

Multi-Tenant Workspace Setup Sequence

1. Create system user account: padogrid

Managing multi-tenant workspaces is made simple by having the padogrid system account that has access to all RWEs and workspaces. The padogrid account serves as an administrator for creating and removing workspaces, adding and removing users and user groups, allocating workspace resources, etc. The following example shows how to create the padogrid account. Note that the home directory of padogrid should be /opt/padogrid to be consistent with PadoGrid Docker containers.

# Create a system account for user 'padogrid'. Set home directory to /home/padogrid
sudo useradd -m -r -U -s /bin/bash -d /home/padogrid padogrid
sudo passwd padogrid

# Add padogrid the sudo group
sudo usermod -aG sudo padogrid

2. Login as padogrid and install PadoGrid

2.1. Create /opt/padogrid directory where PadoGrid will be installed

su padogrid
sudo mkdir /opt/padogrid
sudo chown padogrid /opt/padogrid
sudo chgrp padogrid /opt/padogrid
chmod 750 /opt/padogrid

2.1. Install PadoGrid in /opt/padogrid, not /home/padogrid/Padogrid

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/padogrid/padogrid/develop/padogrid-deployment/src/main/resources/common/bin_sh/install_padogrid)"

Inputs:

:exclamation: Enter /opt/padogrid at the prompt.

Enter the PadoGrid environment base directory path. All of the products will be installed in this directory.  [/opt/padogrid/Padogrid]:
/opt/padogrid
...

2.2. Update .bashrc (assuming you have created the default myrwe RWE from the previous step)

echo ". /opt/padogrid/workspaces/myrwe/initenv.sh -quiet" >> /home/padogrid/.bashrc
. /opt/padogrid/workspaces/myrwe/initenv.sh -quiet

2.3. Set permissions for PadoGrid base directories. The following security restrictions disallow PadoGrid users from downloading and installing products in the /opt/padogrid directory.

cd /opt/padogrid
# Allow only rX for groups
chmod -R u=rwX,g=rX,o-rwx,ug+s downloads products snapshots workspaces

2.4. Set permissions for the default workspace, myws to block all other users. This workspace should be used as a padogrid user's private workspace. You can rename it to a more suitable name as needed.

switch_rwe
# Set permissions for the RWE directory itself
chmod 750 .
# Set permissions for env files
chmod 640 .*env.sh* *env.sh*
# Set permissions for the default workspace. Allow only owner 'padogrid'.
chmod -R u=rwX,g-rwx,o-rwx,u+s myws
setfacl -L -R -d -m u::rwX -m g::000 -m o::000 myws

3. Create an RWE and set permissions to RWE files

:exclamation: You must follow the steps shown in the example below for creating all new RWEs.

3.1. Login as padogrid

su padogrid
cd /opt/padogrid

3.2. Create an RWE in which workspaces to be created

create_rwe -rwe rwe-test

3.3. Set file permissions for RWE files. Users are not allowed to modify these files.

# Switch to the newly created RWE
switch_rwe rwe-test
# Set permissions for the RWE directory itself
chmod 750 .
# Set permissions for env files
chmod 640 .*env.sh* *env.sh*
# Set permissions for the default workspace. Allow only owner 'padogrid'.
chmod -R u=rwX,g-rwx,o-rwx,u+s myws
setfacl -L -R -d -m u::rwX -m g::000 -m o::000 myws

3.4. Enable group permissions in RWE's setenv.sh

vi setenv.sh

At the bottom of setenv.sh set GROUP_PERMISSIONS_ENABLED to "true":

# Enable group permissions for workspace owners. If "true" then RWX permissions are
# given to the user's secondary group.
GROUP_PERMISSIONS_ENABLED="true"

4. Add workspaces

With the security restrictions established for rwe-test in the previous section, PadoGrid effectively isolates each workspace from others by group. To access a workspace, a user must now belong to the group that owns the workspace.

:exclamation: You must follow the steps shown in the example below for creating all new workspaces.

4.1. Login as padogrid

su padogrid

4.2. Create group

# Add 'finance' group
sudo groupadd finance

# Optionally, add 'padogrid' to the seconary group so that it has access to that
# group's workspaces. Note that you must re-login for this to take effect.
# (Without this, commands such as 'show_rwe' may report incurate workspace information
# if padogrid does not have read permissions to certain artifacts.)
sudo usermod -aG finance padogrid

4.3. Create workspace per primary or secondary group

# Create workspace 'ws-finance'.
create_workspace -name ws-finance -quiet
cd_rwe
sudo chgrp -R finance ws-finance
chmod -R u=rwX,g=rwX,o-rwx,ug+s ws-finance
setfacl -L -R -d -m u::rwX -m g::rwX -m o::000 ws-finance

5. Add users to group

To grant access to a workspace, users must be added in the group that owns the workspace.

:exclamation: You must follow the steps shown in the example below for adding users to a group.

5.1. Login as padogrid

su padogrid

5.2 Create user

# Create the primary group to which the users will belong to
sudo groupadd staff

# Add user 'mike' to the primary group
sudo useradd -m -g staff -s /bin/bash mike

5.3. Add user to group

# Add 'mike' in the secondary groups 'finance' and 'padogrid'.
# Always assign a secondary group to a workspace. Otherwise, you will
# not be able to revoke workspace privileges from users.
sudo usermod -aG finance,padogrid mike

5.4. Update .bashrc of the user. This needs to be done as root.

# Login as root
su -

echo ". /opt/padogrid/workspaces/rwe-test/initenv.sh -quiet" >> /home/mike/.bashrc

User Administration

Adding Group

Adding a new group is analogous to adding a group in the OS.

# Add 'staff' group. Used as the primary group in examples
sudo groupadd staff

# Add 'finance' group. Used as a secondary group in examples.
sudo groupadd finance

# Optionally, add 'padogrid' to the seconary group so that it has access to that 
# group's workspaces. Note that you must re-login for this to take effect.
# (Without this, commands such as 'show_rwe' may report incurate workspace information
# if padogrid does not have read permissions to certain artifacts.)
sudo usermod -aG finance padogrid

Adding User Account

Adding a new user account is analogous to adding a user in the OS.

# Add the user 'mitch' to the primary group 'finance'
sudo useradd -m -g finance -s /bin/bash mitch

Granting Workspace Privileges

Granting workspace privileges for a user is analogous to ddding a user to the workspace group.

# Add the user 'mitch' to the 'finance' and 'padogrid' groups
sudo usermod -aG finance,padogrid mitch

Revoking Workspace Privileges

Revoking workspace previleges from a user is analogous to removing a user from the workspace group.

# Remove the user 'mitch' from the 'finance' group
sudo gpasswd -d mitch finance

To revoke privileges to all workspaces, remove the user from the padogrid group.

# Remove the user 'mitch' from the 'padogrid' group
sudo gpasswd -d mitch padogrid

Removing User Account

Removing a user account is analogous to removing a user from the OS.

sudo userdel -f -r mitch

Displaying Groups

# Display all defined groups
groups

Displaying User Groups

# Display all groups that 'mitch' belongs to
groups mitch
id mitch

Displaying Primary Group

# Display the primary group that 'mitch' belongs to
id -gn mitch

Workspace Administration

All PadoGrid administrative operations must be conducted by the padogrid user.

Creating RWE

create_rwe -rwe rwe-test -quiet
switch_rwe rwe-test
# Set permissions for the RWE directory itself
chmod 750 .
# Set permissions for env files
chmod 640 .addonenv.sh .*env.sh initenv.sh setenv.sh
# Set permissions for the default workspace. Allow only owner 'padogrid'.
chmod -R u=rwX,g-rwx,o-rwx,u+s myws
setfacl -L -R -d -m u::rwX -m g::000 -m o::000 myws

Enable group permissions in RWE's setenv.sh

vi setenv.sh

At the bottom of setenv.sh set GROUP_PERMISSIONS_ENABLED to "true":

# Enable group permissions for workspace owners. If "true" then RWX permissions are
# given to the user's secondary group.
GROUP_PERMISSIONS_ENABLED="true"

Creating Workspace

# Create workspace 'ws-finance'.
create_workspace -name ws-finance -quiet
sudo chgrp -R finance ws-finance
cd_rwe
# Set permissions for the workspace. Allow only owner and group.
chmod -R u=rwX,g=rwX,o-rwx,ug+s ws-finance
setfacl -L -R -d -m u::rwX -m g::rwX -m o::000 ws-fiance

Installing Bundle as Workspace

install_bundle -download -workspace bundle-hazelcast-4n5-cluster-session
cd_rwe
sudo chgrp -R finance bundle-hazelcast-4n5-cluster-session
# Set permissions for the workspace. Allow only owner and group.
chmod -R u=rwX,g=rwX,o-rwx,ug+s bundle-hazelcast-4n5-cluster-session
setfacl -L -R -d -m u::rwX -m g::rwX -m o::000 bundle-hazelcast-4n5-cluster-session

Viewing Workspace ACL Persmissions

cd_rwe
getfacl -R ws-finance

Renaming Workspace

switch_rwe
mv ws-finance ws-finance2

Removing Workspace

# Backup and remove workspace
remove_workspace -workspace ws-finance

# Remove without backup
cd_rwe
rm -rf ws-finance

Renaming RWE

switch_rwe rwe-test
cd ..
mv rwe-test rwe-test2

Removing RWE

# Remove without backup
cd_rwe rwe-test
cd ..
rm -rf rwe-test

Product Administration

Installing Product

If you have access to the Internet then run install_padogrid; otherwise, download each product and inflate them in the /opt/padogrid/products directory. See Installing PadoGrid for instructions.

install_padogrid

Updating Product

update_product

Removing Product

Product removals may disrupt workspaces. Before you remove products, notify all users and provide enough time for them to migrate to new product releases. Once the products are removed, the workspaces that depend on the removed products will be no longer operational.

First, install a new release and update RWE (and workspaces). The following example upgrades hazelcast-5.1.1 to hazelcast-5.1.3.

install_padogrid -product hazelcast-oss
update_product -product hazelcast-oss

The update_product interactively guides you to update RWE and/or workspaces. Since you are removing the product, update RWE and all workspaces with the new release.

...
Hazelcast OSS
   [0] (none)
   [1] 5.1.3
   [2] 5.1.1
Enter a version to add [0]: 1
Selected: [1] 5.1.3

Apply selected version(s) to RWE?
Enter 'r' for RWE, 'n' for no RWE, 'q' to quit this command: r

Apply selected version(s) to the current workspace or all workspaces?
Enter 'w' for current workspace, 'a' for all workspaces,
      'n' for no workspaces,     'q' to quit this command: a
...
Enter 'c' to continue, 'r' to re-enter, 'q' to quit: c

If you have more than one RWEs, then run update_product on each RWE.

You are now ready to remove the product. All products are installed in the /opt/padogrid/products directory. Find the ones you want to remove from that directory and remove them by running rm.

# Remove hazelcast-5.1.1
cd ~/products
rm -rf hazelcast-5.1.1

User Perspective

Each user would take the following steps to access their workspace.

  1. Login to the OS
  2. Switch to the workspace
  3. Start using the workspace. The workspace is completely isolated. The user can only view the workspaces owend by the groups he/she belongs to and cannot browse other workspaces. This applies to all RWEs.

:pencil2: The /opt/padogrid/downloads and /opt/padogrid/products directories are accessible by all users. The padogrid administrator can make product distributions and artifacts available in /opt/padogrid/downloads and install any products in /opt/padogrid/products.

Example:

Login via ssh:

ssh [email protected]

Or using the browser (JupyterLab):

Once logged in, from the shell execute PadoGrid commands:

# Display all RWEs and workspaces owne by 'mike'
padogrid

# Display all workspaces in the current RWE owned by 'mike'
show_rwe

# Switch to the 'ws-fiance' workspace in the current RWE owned by 'mike'
switch_workspace ws-finance

# Display current workspace (ws-finance) contents
show_workspace

Normally, workspaces run in a controlled environment accessible by only a select group of system administrators. Each administrator must strictly comply with the security governance established by the company across all the workspaces they are responsible for.

In an uncontrolled environment, the same security governance applies but the users are allowed to manage their own workspaces. This means workspaces would compete for system resources. To prevent this, each workspace should be installed on dedicated machines. This can be achieved by creating a VM-enabled workspace and auto-login users to one of the VMs. VMs can be any remote machines such as AWS EC2.


Branching Out Workspaces

A workspace can be branched out to a separate mount point by simply dedicating a mount point to the workspace or creating a symbolic link to a mount point path. Similarly, the entire RWE can also be branched out to a separate mount point. By having a workspace on a separate mount point, it limits the use of the host OS resources.

Adding Nodes to Workspaces

Each workspace is independently configured such that you can assign any number of nodes. The workspace owner can enable VMs in the workspace by simply listing the VM addresses in the configuration file. These VMs could be AWS EC2 VMs or on-prem VMs, for example.

Working with Docker Containers

Docker containers can be launched on the host OS or remote machines using the create_docker and install_bundle commands.

Working with Kubernetes

If Kubernetes is preferred then the workspace owner creates the k8s components using the create_k8s and install_bundle commands.


Creating Multiple Accounts - Example

The following example shows how to create multiple user accounts, groups, and workspaces. It 1) adds the users, mike, mary, matt, mia, and max in the secondary group finance, 2) creates the ws-finance workspace, and 3) updates each user's .bashrc file with a PadoGrid initialization command.

# Login as padogrid
su padogrid

# Create primary group to which the users will belong to
PRIMARY_GROUP="staff"
sudo groupadd $PRIMARY_GROUP

# Create user/secondary group. Users must belong to a secondary group.
PADOGRID_USERS="mike mary matt mia max"
GROUP="finance"
sudo groupadd $GROUP
for USER in $PADOGRID_USERS; do \
   sudo useradd -m -g $PRIMARY_GROUP -s /bin/bash $USER; \
   sudo usermod -aG $GROUP,padogrid $USER; \
done

# Optionally, add 'padogrid' to the seconary group so that it has access to that
# group's workspaces. Note that you must re-login for this to take effect.
# (Without this, commands such as 'show_rwe' may report incurate workspace information
# if padogrid does not have read permissions to certain artifacts.)
sudo usermod -aG $GROUP padogrid

# Create workspace. Always assign a secondary group to a workspace. Otherwise, you
# will not be able to revoke workspace privileges from users.
WORKSPACE=ws-$GROUP
create_workspace -name $WORKSPACE -quiet
sudo chgrp -R $GROUP $WORKSPACE
# Set permissions for the workspace. Allow only owner and group.
chmod -R u=rwX,g=rwX,o-rwx,ug+s $WORKSPACE
setfacl -L -R -d -m u::rwX -m g::rwX -m o::000 $WORKSPACE

# Login as root
su -

# Append PadoGrid initialization command in each user's .bashrc
RWE="rwe-test"
PADOGRID_USERS="mike mary matt mia max"
for USER in $PADOGRID_USERS; do \
   echo ". /opt/padogrid/workspaces/$RWE/initenv.sh -quiet" >> /home/$USER/.bashrc; \
done
exit

Removing Multiple Accounts - Example

We can remove all the accounts and the workspace we created in the previous example as follows.

# Login as padogrid
su padogrid

# Remove user/group
PADOGRID_USERS="mike mary matt mia max"
GROUP="finance"
for USER in $PADOGRID_USERS; do \
   sudo userdel -f -r $USER; \
done
sudo groupdel $GROUP

# Remove workspace
RWE="rwe-test"
switch_rwe $RWE
GROUP="finance"
WORKSPACE=ws-$GROUP
sudo rm -rf $WORKSPACE

◀️ VM-Enabled Pods :link: Multitenancy Best Practices ▶️