Introduction

The app modules follow a strict structure. Developers must follow the structure to ensure coherence across the app as well as the dongcha platform. We describe the know-how for managing app modular entities and functional packages.

App Hierarchy

| app | -- has -- | modules | -- with -- |entities | -- that have -- | packages | -- performing -- |functions |

App [dongcha (base), mining, wrangler, visuals] | ------ Modules | ------------ Entities | ------------------ Packages | ----------------------------- Functional [classes]:

Structure

An entity folder embedded inside the app/modules folder holds a set of functional packages specific to that entity. For example, wrangler/modules/ota/scraper defines the Online-Travel-Agency (OTA) related functionality for housing classes (or packages) of scraper functions for scraping OTA website data. Central to configuring a particular app's environment variables is the app.cfg file and the dongcha.py class.

DONGCHA.PY

The class contains the main methods:

1. Setting up the app.ini for the functional packages
2. reading and writing the configuration data
3. reading and writing log files

APP.CFG

The app.cfg file follows the python ConfigParser taxonomy. Each app's (datamart, mining, utils, visuals, & wrangler) configuration information is described in this file. Information from the app.cfg will be copied into individual module functions app.ini files. Thereby, bringing deployment-independent coherence for all the packages to function easily. The relevant sections are discussed in the section below.

[MODULES]

The applicable module entities and functions must be defined in the app.cfg file. They are defined in the [MODULES] section.

##
## e.g. wrangler/app.cfg
## [MODULES]
## ota = scraper
##
[MODULES]
entity_X = function_X1, function_X2, function_X3
entity_Y = function_Y1

[DATASTORE]

Set the default data storage mode; i.e. local or cloud file system. Furthermore, specifies the root directory, which is also used to define the AWS Bucket name.

#   e.g., mode = AWS-S3-BUCKET & root = dongcha-wrangler-source-code (bucket name)
#   e.g., mode = LOCAL-FS & root = dongcha (directory name dongcha app is in)
mode = local-fs
root = dongcha

[HOSTS]

Defines the server IP. Required for establishing database and web connections. The application hosting location can be replaced with deployed URL.

#  default localhost
hostIP = 127.0.0.1

[DATABASE]

Defines the database connectivity information

#--database types: mysql, postgresql (default: postgres)
dbType = postgresql
#--port default 5432
dbPort = 5432
#--database driver
#  postgresql: 'org.postgresql.Driver'
dbDriver = org.postgresql.Driver
#--database name
dbName = rezstage
#--schema name
dbSchema = lakehouse
#--username and password to connect
#  default db_user=postgres, db_pswd = postgres
dbUser = postgres
dbPswd = postgres

[LOGGER]

All logging parameters are defined here.

#--directory path to store logs
#  defaulat logs/
Path = logs/
File = app.log
#--logging level to set DEBUG, ERROR, WARNING, INFO
Level = DEBUG
#--append or write over the logs
#  default: 'a' to append, 'w+' to overwrite 
Mode = w+
#--loggin format
#  default: %(asctime)s - %(name)s - %(levelname)s - %(message)s'
# logFormat = %(asctime)s - %(name)s - %(levelname)s - %(message)s
Format = asctime,name,levelname,message