Microsoft Azure Guzzle Setup Using Script - ja-guzzle/guzzle_docs GitHub Wiki

Background

This document will help you to set up Guzzle inside Microsoft Azure Portal using an automated script.

User Assigned Managed Identity

Create one User Assigned Managed Identity.

  • this identity will be added inside Blob Storage Account and Virtual Machine.

Virtual Machine

Create one virtual machine of type UbuntuServer-18.04-LTS.

  • Configure virtual machine DNS name (virtual machine -> overview -> DNS name)
  • Add user-assigned managed identity to the virtual machine. (virtual machine -> identity -> user assigned -> add)
  • Add a firewall rule to allow access of Guzzle UI port 8082 and Guzzle API port 9090 (virtual machine -> Networking -> add inbound port rule)

* required properties in config.properties file

subscription_id=9d05****d7239
location=southeastasia
public_ip_dns=guzzlevm
admin_username=guzzle

Storage Account

Create one blob storage account to store guzzle binaries.

  • Create one container inside the storage account.
  • Add User Assigned Managed Identity with a contributor role (Storage Account -> Access Control (IAM) -> Add -> Add Role Assignment -> Role: contributer and select: created Managed Identity -> Add )

* required properties in config.properties file

storage_account_name=guzzlestorage
storage_account_container=guzzlehome
storage_account_resource_group_name=azure_rg1

SQL Server And Database

Create a SQL database server:

  • create one database guzzle to store guzzle related tables.
  • create one database hivemetastore to store hive metadata.
  • add firewall rules to allow azure services and resource to access SQL server. (SQL server -> Firewalls and virtual networks -> Allow azure services and resources to access this server -> ON)

* required properties in config.properties file

guzzle_database_type=sqlserver
guzzle_database_server_name=guzzledb
guzzle_database_name=guzzle
guzzle_database_username=demo
guzzle_database_password=clusterTest@123

metastore_database_type=external
metastore_database_server_name=guzzledb
metastore_database_name=hivemetastore
metastore_database_username=demo
metastore_database_password=clusterTest@123

Databricks Workspace

Create One Databricks Workspace

* required properties in config.properties file

databricks_workspace_name=guzzlews
databricks_workspace_region=southeastasia
databricks_org_id=5359*******61671
databricks_access_token=dapidb*********54b4ee600b7f59

Setup Guzzle

Download guzzle-setup.zip file from here.

  • copy the guzzle-setup.zip file inside a virtual machine and extract it inside guzzle-setup directory using below command.
scp guzzle-setup.zip [email protected]:~
ssh [email protected]
unzip guzzle-setup.zip -d guzzle-setup
  • update config.properties file
subscription_id=<Azure Subscription ID>
location=<Virtual Machine Location>
public_ip_dns=<Virtual Machine DNS Name>
admin_username=<Virtual Machine Admin User Name>
storage_account_name=<Blob Storage Account Name>
storage_account_container=<Blob Storage Container Home>
storage_account_resource_group_name=<Blob Storage Resource Group Name>
guzzle_database_type=<Guzzle Database Type ["sqlserver"]>
guzzle_database_server_name=<SQL Server Name>
guzzle_database_name=<Guzzle Database Name - to store guzzle releated tables>
guzzle_database_username=<Guzzle Database Username>
guzzle_database_password=<Guzzle Database Password>
metastore_database_type=external
metastore_database_server_name=<Hive Metastore Database Server Name>
metastore_database_name=<Hive Metastore Database Nanme>
metastore_database_username=<Hive Metastore Databaser Username>
metastore_database_password=<Hive Metastore Database Password>
databricks_workspace_name=<Databricks Workspace Name>
databricks_workspace_region=<Databrick Workspace Location or Region>
databricks_org_id=<Databricks Workspace Organization ID>
databricks_access_token=<Databricks Access Token>
enable_atlas=<To Enable Atlast [yes | no]>
enable_microsoft_oauth=<To Enable microsoft active directory authentication [yes | no] if Yes provide client id and secret >
microsoft_oauth_client_id=<Microsoft Active Directory client ID>
microsoft_oauth_client_secret=<Microsoft Active Directory Client Secret>
  • executed start-guzzle-deployment.sh from the guzzle-setup directory:
cd guzzle-setup/
sudo ./start-guzzle-deployment.sh
⚠️ **GitHub.com Fallback** ⚠️