APIs ‐ StepExecutor - altoiddealer/ad_discordbot GitHub Wiki
StepExecutor is the processing system used by both API Endpoint response_handling
and Workflows
- Streamlines many complicated functions via simple "Step" configurations:
- Processes each defined "Step" sequentially
- Can execute a "for-each" loop of steps from a list of data
- Can even run a step "groups" in parallel
- Uses a sophisticated context management system
- Step results / extracted values can be stored via
save_as
- Context values, Bot Variables, even Websocket variables can be used anywhere in the system which will automatically resolve to their actual values
- Step results / extracted values can be stored via
Any Step can include Meta parameters, set as top level items in each step dictionary (not nested in each "step config")
Meta Parameters are applied in the following order: on_error
-> save_as
-> returns
-> log

Saves the result into a Context dictionary using the given key name
Saved values can then be injected anywhere using {this_syntax}
.
The script can also convert a string value like this into actual python value: "{some_key: {saved_value}}"
- this will be turned into an actual dictionary object with the resolved value from context {'some_key': \<the value previously saved to context as 'saved_value'\>}
Can override what is typically returned by a Step.
-
"data"
(default) the Step result -
"input"
the original input to the Step -
"context"
the entire stored context dictionary -
"a key name"
If the step result is a dict, you can return a top-level key value from it. For deeper values just use extract_key
Provides additional logging for a step. Useful for debugging
Allows gracefully handling exceptions without aborting StepExecutor
-
"raise"
(default) raises Exception (aborts StepExecutor) -
"skip"
returns the input value to the next step. -
"default"
A specific value to return for the next step.
Step Name | Description |
---|---|
for_each | Runs a sub-StepExecutor for each item in a list or each key-value pair in a dict. |
save | Saves input data to a file |
pass | Does nothing. Can be useful when you simply want to use a meta parameter |
extract_key | Retrieves a nested value from a data structure (dicts/lists) based on a specified path |
extract_values | Applies extract_key to build a new dictionary using user-defined key names for the extracted values. |
set_key | Sets a value within a data structure (dicts/lists) based on a specified path |
return | Helpful for passing a specific variable from saved Context as input to the next step. |
type | Converts the input data to a specified target type. |
cast | Like type except it can convert multiple dictionary value types at once. |
map | Transforms each element of a list into a dict. The key name can be arbitrary, or set by a dynamic context value |
regex | Extracts text from an input string using a regular expression pattern. It returns the first captured group if one exists, or the entire match if not. |
call_api | Performs an API call, and returns the result. |
poll_api | Like call_api except that it repeatedly calls the API, collecting each result to a list which is returned after polling is stopped. |
track_progress | Polls an endpoint while sending a progress Embed to discord. |
decode_base64 | Decodes a Base64-encoded string into raw bytes |
Executes multiple step sequences in parallel, each defined as a list of steps.
Each item in the config is a list of steps (a sub-workflow), which is executed in parallel with others.
Config: sub-lists of Steps sequences (any Steps)
Returns: A list
of results from each parallel sub-sequence
Runs a sub-StepExecutor for each item in a list or each key-value pair in a dict.
Options | Option Description |
---|---|
in | The list object to be iterated over (A variable saved to context) |
as | A label to use for dynamic placeholders, based on the current index |
steps |
list of Steps to be executed. Each value from the in list is the input data for each loop |
Example Usage
response_handling:
# Capture 'images' list as context variable "image_list"
- extract_key: images
save_as: image_list
# Execute a 'for loop' of steps on each value in 'image_list'
- for_each:
in: image_list
as: image
steps:
- decode_base64: true
- save:
file_format: png
file_path: "images"
file_name: "img_{image_index}"
Saves input data to a file
Parameter | Value Type | Description | Default |
---|---|---|---|
file_format | string | (optional) Explicit format (e.g. 'json', 'jpg'). When not provided, script will try to infer the correct type | |
file_name | string | (optional) File name to save as (without extension) | |
file_path | string | (optional) Should be a path relative to /output . Can be a full path (path must be configured as "allowed" in config.yaml ) |
|
use_timestamp | bool | Whether to add a timestamp to the filename | True |
Returns dictionary
{"path": str(full_path),
"format": file_format,
"name": file_name,
"data": the_original_input_data}
Does nothing. Can be useful when you simply want to use a meta parameter
Retrieves a nested value from a data structure (dicts and lists) based on a specified path. The path is expressed as a dot/bracket notation string (e.g., "a.b[0].c"
), and can optionally return a default value if the path is invalid or inaccessible.
Parameter | Value Type | Description | Default |
---|---|---|---|
path | string | The path to the value expressed as a dot/bracket notation string (ex: "user.name[1]" / "[0].images" ) |
|
default | string | (optional) The default value to return if the path is not found. can be a "{context_variable}" |
Returns: Any
- The extracted value, or the default value
Example Usage
Scenario:
data = {
"user": {
"name": ["Alice", "Bob"],
"profile": {
"age": 30
}
}
}
Example 1:
path: "user.name[1]"
Returns: "Bob"
Example 2:
path: "user.profile.age"
Returns: 30
Example 3:
path: "user.address"
default: "Unknown"
Returns: "Unknown"
Applies extract_key
to build a new dictionary using user-defined key names for the extracted values.
The dictionary can be stored to Context. Each item can then be resolved in future steps using dot/bracket notation.
Example Usage:
- extract_values:
image: data.image
description: data.description
save_as: image_data
log: true
printed in cmd log: 'image_data': {'image': <the image data>, 'description': <the image description>}
The value for 'image' (Saved to context in the 'image_data' dict) can then be resolved into another field like this:
image: "{image_data.image}"
Sets a value within a data structure (dicts/lists) based on a specified path. The path is expressed using the same dot/bracket notation string as extract_key (e.g., "a.b[0].c"
).
Parameter | Value Type | Description | Default |
---|---|---|---|
path | string | The path to the value expressed as a dot/bracket notation string (ex: "user.name[1]" / "[0].images" ) |
|
value | Any | The value to set. |
Returns: The updated data structure (dict or list)
Helpful for passing a specific variable from saved Context as input to the next step.
Parameter | Value Type | Description | Default |
---|---|---|---|
return | string | The name of a context variable (not expressed in dot/bracket notation |
Converts the input data to a specified target type. For "bool": "true", 1
, and "yes" will return True
otherwise False
Valid types: "int", "float", "str", "bool"
Example usage:
- type: "int"
- Would convert the input value to an integer
Like type
except it can convert multiple dictionary value types at once.
Example Usage
Example scenario:
data = {"age": "30", "height": "5.8", "name": 123, "active": "yes"}
- cast:
age: "int"
height: "float"
name: "str"
active: "bool"
Output:
{
"age": 30, # converted to int
"height": 5.8, # converted to float
"name": "123", # converted to str
"active": True # converted to bool ("yes" → True)
}
Transforms each element of a list into a dict. The key name can be arbitrary, or set by a dynamic context value:
Parameter | Value Type | Description | Default |
---|---|---|---|
key | string | Defines the key to use in the resulting dictionary |
Example Usage
data = ["apple", "banana", "cherry"]
- map:
as: "dict"
key: "fruit"
Returns: [{"fruit": "apple"}, {"fruit": "banana"}, {"fruit": "cherry"}]
Extracts text from an input string using a regular expression pattern. It returns the first captured group if one exists, or the entire match if not.
Behavior: Uses re.search
, which returns the first match anywhere in the string. Safely handles missing matches and avoids exceptions.
Returns: The first capturing group (group(1)) if the pattern contains any capturing groups. Otherwise, the full matched string (group(0)). If no match is found, logs a warning and returns the original data.
Example Usage
Example 1:
data = "Order #12345 received"
- regex: r"#(\d+)"
Returns: "12345"
(captures group 1)
Example 2:
data = "Email: [email protected]"
- regex: r"\w+@\w+\.\w+"
Returns: "[email protected]"
(no group, returns full match)
Example 3:
data = "Hello world"
- regex: r"ID: (\d+)"
Returns: "Hello world"
(no match found, logs warning and returns original data)
(Except for get_payload
): For these Steps, any Endpoint Parameters can be included in addition to the inputs defined here, which will override the configured Endpoint behaviors for the Step interaction.
IMPORTANT For non-websocket API calls: Unless config input_data: null
, the bot will send the current "result" / defined input.
Gets the base_payload
from an API Endpoint.
Parameter | Value Type | Description | Default |
---|---|---|---|
client_name | string | (required) the name of the API client (in all_apis list). client may be used alternatively. |
|
endpoint_name | string | (required) the name of the Endpoint. endpoint may be used alternatively. |
Returns: The Endpoint's base_payload
.
Performs an API call, and returns the result.
Parameter | Value Type | Description | Default |
---|---|---|---|
input_data | Any | (optional) The input value to send to the API. If not specified, the current "Step Result" will be used as input. | <last step result> |
init_payload | bool | (optional) If True: overrides input_data ; fetches the base_payload from the Endpoint; tries resolving internal variables before sending to API |
False |
client_name | string | (required) the name of the API client (in all_apis list). client may be used alternatively. |
|
endpoint_name | string | (required) the name of the Endpoint. endpoint may be used alternatively. |
Returns: Result is dependent on the Endpoint configuration / extra API Endpoint parameters included.
Like call_api
except that it repeatedly calls the API, collecting each result to a list which is returned after polling is stopped.
Uses same parameters as call_api
with a few additions
Parameter | Value Type | Description | Default |
---|---|---|---|
All of the parameters for call_api are applicable here, additionally: |
|||
input_data | Any | (optional) The input value to send to the API. If not specified, the current "Step Result" will be used as input. | <last step result> |
return_values | dict | A dict where keys are output keys, and values are paths to extract values from the response, using the same mechanism/syntax as extract_key (dot/bracket notation) | |
interval | float | How frequently (in seconds) to call the API | 1.0 |
duration | int | If specified, will stop polling after the given timeframe | -1 (not factored) |
num_yields | int | If specified, will stop polling after the given number of API calls | -1 (not factored) |
Returns: A list of the results from the API calls, dependent on the Endpoint configuration / extra API Endpoint parameters included.
Polls an endpoint while sending a progress Embed to discord.
Like poll_api
except specialized for purpose of tracking progress from a previous or concurrent call_api
step.
If max_key
is specified, progress is interpreted as a step count and normalized as (progress / max). Otherwise, progress is assumed to be a float between 0.0 and 1.0.
Uses same parameters as call_api
and poll_api
with a few additions
Parameter | Value Type | Description | Default |
---|---|---|---|
All of the parameters for call_api and poll_api are applicable here, additionally: |
|||
input_data | Any | (optional) The input value to send to the API. If not specified, defaults to None
|
None |
progress_key | string | Uses same mechanism as extract_key (dot/bracket notation) to get the progress value from API response | "progress" |
max_key | string | (optional) Also uses extract_key. If applicable, should return the maximum value of "progress" | |
eta_key | string | (optional) Also uses extract_key. If applicable, discord embed will include the ETA value. | |
message | string | (optional) Message for the discord embed | "Generating" |
completion_condition | dict | A data structure which must be matched in a response to signify that the progress being tracked is completed | |
use_ws | bool | Controls whether the polling method is via http or the Client's Websocket (which must be properly configured in Client config) | False |
type_filter | list of strings | (Websocket only) Message types to be analyzed. Types not in the list are ignored. | ["progress", "executed"] |
data_filter | dict | (Websocket only) Required key-values in the data field. | None |
Complete Websocket example including a call_api step
- call_api:
api_name: ComfyUI
endpoint_name: Prompt
payload_type: json
input_data:
prompt: "{prompt}"
save_as: prompt_id
- track_progress:
client_name: ComfyUI
progress_key: data.value # Required. dot notation gets the value nested in "data"
max_key: data.max # Optional. If applicable, should return the maximum value of "progress".
eta_key: null # Optional. If applicable, discord embed will include the ETA value.
message: Generating a video # message for the discord embed
completion_condition: # Optional: All key/values must be matched to stop fetching progress
type: executed
data: {prompt_id: "{prompt_id}"}
# websocket exclusive values
use_ws: True
type_filter: [progress, executed] # ignores other message types
data_filter: {prompt_id: "{prompt_id}"} # must '.get()' these key/values
Decodes a Base64-encoded string into raw bytes. For convenience, checks for a data URI (e.g., "data:image/png;base64,..."
) and splits it, using only the part after the comma for decoding.
Returns: bytes