APIs ‐ StepExecutor - altoiddealer/ad_discordbot GitHub Wiki

Introduction

StepExecutor is the processing system used by both API Endpoint response_handling and Workflows

  • Streamlines many complicated functions via simple "Step" configurations:
    • Processes each defined "Step" sequentially
    • Can execute a "for-each" loop of steps from a list of data
    • Can even run a step "groups" in parallel
  • Uses a sophisticated context management system
    • Step results / extracted values can be stored via save_as
    • Context values, Bot Variables, even Websocket variables can be used anywhere in the system which will automatically resolve to their actual values

Meta Parameters

Any Step can include Meta parameters, set as top level items in each step dictionary (not nested in each "step config")

Meta Parameters are applied in the following order: on_error -> save_as -> returns -> log

Screenshot 2025-06-10 132104

save_as (string)

Saves the result into a Context dictionary using the given key name

Saved values can then be injected anywhere using {this_syntax}.

The script can also convert a string value like this into actual python value: "{some_key: {saved_value}}" - this will be turned into an actual dictionary object with the resolved value from context {'some_key': \<the value previously saved to context as 'saved_value'\>}

returns (string)

Can override what is typically returned by a Step.

  • "data" (default) the Step result
  • "input" the original input to the Step
  • "context" the entire stored context dictionary
  • "a key name" If the step result is a dict, you can return a top-level key value from it. For deeper values just use extract_key

log (bool)

Provides additional logging for a step. Useful for debugging

on_error (string)

Allows gracefully handling exceptions without aborting StepExecutor

  • "raise" (default) raises Exception (aborts StepExecutor)
  • "skip" returns the input value to the next step.
  • "default" A specific value to return for the next step.

Steps

Quick Reference

Step Name Description
for_each Runs a sub-StepExecutor for each item in a list or each key-value pair in a dict.
save Saves input data to a file
pass Does nothing. Can be useful when you simply want to use a meta parameter
extract_key Retrieves a nested value from a data structure (dicts/lists) based on a specified path
extract_values Applies extract_key to build a new dictionary using user-defined key names for the extracted values.
set_key Sets a value within a data structure (dicts/lists) based on a specified path
return Helpful for passing a specific variable from saved Context as input to the next step.
type Converts the input data to a specified target type.
cast Like type except it can convert multiple dictionary value types at once.
map Transforms each element of a list into a dict. The key name can be arbitrary, or set by a dynamic context value
regex Extracts text from an input string using a regular expression pattern. It returns the first captured group if one exists, or the entire match if not.
call_api Performs an API call, and returns the result.
poll_api Like call_api except that it repeatedly calls the API, collecting each result to a list which is returned after polling is stopped.
track_progress Polls an endpoint while sending a progress Embed to discord.
decode_base64 Decodes a Base64-encoded string into raw bytes

Iterative Steps

group

Executes multiple step sequences in parallel, each defined as a list of steps.

Each item in the config is a list of steps (a sub-workflow), which is executed in parallel with others.

Config: sub-lists of Steps sequences (any Steps)

Returns: A list of results from each parallel sub-sequence


for_each

Runs a sub-StepExecutor for each item in a list or each key-value pair in a dict.

Options Option Description
in The list object to be iterated over (A variable saved to context)
as A label to use for dynamic placeholders, based on the current index
steps list of Steps to be executed. Each value from the in list is the input data for each loop
Example Usage
    response_handling:
      # Capture 'images' list as context variable "image_list"
      - extract_key: images
        save_as: image_list
      # Execute a 'for loop' of steps on each value in 'image_list'
      - for_each:
          in: image_list
          as: image
          steps:
            - decode_base64: true          
            - save:
                file_format: png
                file_path: "images"
                file_name: "img_{image_index}"

save

Saves input data to a file

Parameter Value Type Description Default
file_format string (optional) Explicit format (e.g. 'json', 'jpg'). When not provided, script will try to infer the correct type
file_name string (optional) File name to save as (without extension)
file_path string (optional) Should be a path relative to /output. Can be a full path (path must be configured as "allowed" in config.yaml)
use_timestamp bool Whether to add a timestamp to the filename True

Returns dictionary

{"path": str(full_path),
"format": file_format,
"name": file_name,
"data": the_original_input_data}

Data Manipulation Steps

pass

Does nothing. Can be useful when you simply want to use a meta parameter


extract_key

Retrieves a nested value from a data structure (dicts and lists) based on a specified path. The path is expressed as a dot/bracket notation string (e.g., "a.b[0].c"), and can optionally return a default value if the path is invalid or inaccessible.

Parameter Value Type Description Default
path string The path to the value expressed as a dot/bracket notation string (ex: "user.name[1]" / "[0].images")
default string (optional) The default value to return if the path is not found. can be a "{context_variable}"

Returns: Any - The extracted value, or the default value

Example Usage

Scenario:

data = {
    "user": {
        "name": ["Alice", "Bob"],
        "profile": {
            "age": 30
        }
    }
}

Example 1:

path: "user.name[1]"

Returns: "Bob"

Example 2:

path: "user.profile.age"

Returns: 30

Example 3:

path: "user.address"
default: "Unknown"

Returns: "Unknown"


extract_values

Applies extract_key to build a new dictionary using user-defined key names for the extracted values.

The dictionary can be stored to Context. Each item can then be resolved in future steps using dot/bracket notation.

Example Usage:

- extract_values:
    image: data.image
    description: data.description
  save_as: image_data
  log: true

printed in cmd log: 'image_data': {'image': <the image data>, 'description': <the image description>}

The value for 'image' (Saved to context in the 'image_data' dict) can then be resolved into another field like this:

image: "{image_data.image}"


set_key

Sets a value within a data structure (dicts/lists) based on a specified path. The path is expressed using the same dot/bracket notation string as extract_key (e.g., "a.b[0].c").

Parameter Value Type Description Default
path string The path to the value expressed as a dot/bracket notation string (ex: "user.name[1]" / "[0].images")
value Any The value to set.

Returns: The updated data structure (dict or list)


return

Helpful for passing a specific variable from saved Context as input to the next step.

Parameter Value Type Description Default
return string The name of a context variable (not expressed in dot/bracket notation

type

Converts the input data to a specified target type. For "bool": "true", 1, and "yes" will return True otherwise False

Valid types: "int", "float", "str", "bool"

Example usage:

- type: "int" - Would convert the input value to an integer


cast

Like type except it can convert multiple dictionary value types at once.

Example Usage

Example scenario:

data = {"age": "30", "height": "5.8", "name": 123, "active": "yes"}

- cast:
    age: "int"
    height: "float"
    name: "str"
    active: "bool"

Output:

{
   "age": 30,           # converted to int
   "height": 5.8,       # converted to float
   "name": "123",       # converted to str
   "active": True       # converted to bool ("yes" → True)
 }


map

Transforms each element of a list into a dict. The key name can be arbitrary, or set by a dynamic context value:

Parameter Value Type Description Default
key string Defines the key to use in the resulting dictionary
Example Usage

data = ["apple", "banana", "cherry"]

- map:
    as: "dict"
    key: "fruit"

Returns: [{"fruit": "apple"}, {"fruit": "banana"}, {"fruit": "cherry"}]


regex

Extracts text from an input string using a regular expression pattern. It returns the first captured group if one exists, or the entire match if not.

Behavior: Uses re.search, which returns the first match anywhere in the string. Safely handles missing matches and avoids exceptions.

Returns: The first capturing group (group(1)) if the pattern contains any capturing groups. Otherwise, the full matched string (group(0)). If no match is found, logs a warning and returns the original data.

Example Usage

Example 1:

data = "Order #12345 received"

- regex: r"#(\d+)"

Returns: "12345" (captures group 1)


Example 2:

data = "Email: [email protected]"

- regex: r"\w+@\w+\.\w+"

Returns: "[email protected]" (no group, returns full match)


Example 3:

data = "Hello world"

- regex: r"ID: (\d+)"

Returns: "Hello world" (no match found, logs warning and returns original data)


API Related Steps

(Except for get_payload): For these Steps, any Endpoint Parameters can be included in addition to the inputs defined here, which will override the configured Endpoint behaviors for the Step interaction.

IMPORTANT For non-websocket API calls: Unless config input_data: null, the bot will send the current "result" / defined input.

get_api_payload

Gets the base_payload from an API Endpoint.

Parameter Value Type Description Default
client_name string (required) the name of the API client (in all_apis list). client may be used alternatively.
endpoint_name string (required) the name of the Endpoint. endpoint may be used alternatively.

Returns: The Endpoint's base_payload.


call_api

Performs an API call, and returns the result.

Parameter Value Type Description Default
input_data Any (optional) The input value to send to the API. If not specified, the current "Step Result" will be used as input. <last step result>
init_payload bool (optional) If True: overrides input_data; fetches the base_payload from the Endpoint; tries resolving internal variables before sending to API False
client_name string (required) the name of the API client (in all_apis list). client may be used alternatively.
endpoint_name string (required) the name of the Endpoint. endpoint may be used alternatively.

Returns: Result is dependent on the Endpoint configuration / extra API Endpoint parameters included.


poll_api

Like call_api except that it repeatedly calls the API, collecting each result to a list which is returned after polling is stopped.

Uses same parameters as call_api with a few additions

Parameter Value Type Description Default
All of the parameters for call_api are applicable here, additionally:
input_data Any (optional) The input value to send to the API. If not specified, the current "Step Result" will be used as input. <last step result>
return_values dict A dict where keys are output keys, and values are paths to extract values from the response, using the same mechanism/syntax as extract_key (dot/bracket notation)
interval float How frequently (in seconds) to call the API 1.0
duration int If specified, will stop polling after the given timeframe -1 (not factored)
num_yields int If specified, will stop polling after the given number of API calls -1 (not factored)

Returns: A list of the results from the API calls, dependent on the Endpoint configuration / extra API Endpoint parameters included.


track_progress

Polls an endpoint while sending a progress Embed to discord.

Like poll_api except specialized for purpose of tracking progress from a previous or concurrent call_api step.

If max_key is specified, progress is interpreted as a step count and normalized as (progress / max). Otherwise, progress is assumed to be a float between 0.0 and 1.0.

Uses same parameters as call_api and poll_api with a few additions

Parameter Value Type Description Default
All of the parameters for call_api and poll_api are applicable here, additionally:
input_data Any (optional) The input value to send to the API. If not specified, defaults to None None
progress_key string Uses same mechanism as extract_key (dot/bracket notation) to get the progress value from API response "progress"
max_key string (optional) Also uses extract_key. If applicable, should return the maximum value of "progress"
eta_key string (optional) Also uses extract_key. If applicable, discord embed will include the ETA value.
message string (optional) Message for the discord embed "Generating"
completion_condition dict A data structure which must be matched in a response to signify that the progress being tracked is completed
use_ws bool Controls whether the polling method is via http or the Client's Websocket (which must be properly configured in Client config) False
type_filter list of strings (Websocket only) Message types to be analyzed. Types not in the list are ignored. ["progress", "executed"]
data_filter dict (Websocket only) Required key-values in the data field. None
Complete Websocket example including a call_api step
- call_api:
    api_name: ComfyUI
    endpoint_name: Prompt
    payload_type: json
    input_data:
      prompt: "{prompt}"
  save_as: prompt_id
- track_progress:
    client_name: ComfyUI
    progress_key: data.value # Required. dot notation gets the value nested in "data"
    max_key: data.max        # Optional. If applicable, should return the maximum value of "progress".
    eta_key: null            # Optional. If applicable, discord embed will include the ETA value.
    message: Generating a video # message for the discord embed
    completion_condition:       # Optional: All key/values must be matched to stop fetching progress
      type: executed
      data: {prompt_id: "{prompt_id}"}
    # websocket exclusive values
    use_ws: True
    type_filter: [progress, executed]   # ignores other message types
    data_filter: {prompt_id: "{prompt_id}"} # must '.get()' these key/values

Other Steps

decode_base64

Decodes a Base64-encoded string into raw bytes. For convenience, checks for a data URI (e.g., "data:image/png;base64,...") and splits it, using only the part after the comma for decoding.

Returns: bytes

⚠️ **GitHub.com Fallback** ⚠️