• Brief
• Main entry
• Input Arguments
• return value

Brief

This document is for Data Engineer who writes Data Applications.

Main entry

def main(spark, input_args, sysops={}):
   ...

• spark: the saprk session
• input_args: the input arguments
• sysops: system options, normally user should ignore this.

Input Arguments

It looks like below

• app_args is what user specifies in UI

{
    "pipeline_group_context": {
        "dt": "2020-12-27",
        "xcom": {
            "import-trading-data-nasdaq": null
        }
    },
    "application_id": "0d8d2c02-0a32-4aa5-9301-505de05c30a5",
    "app_args": {
        "action": "import-data",
        "market": "NASDAQ",
        "data_root": "oci://BetaTradings@idrnu3akjpv5"
    },
    "dc_config": {
        "url_base": "http://dmdemo1:8888/api",
        "username": "stonezhong",
        "password": "****"
    }
}

return value