General: Project Variables - FlipsideCrypto/fsc-evm GitHub Wiki

Requires fsc-evm v4.0.0+

Querying Project Variables

Master Seeds: Config Tables

Source of truth for project variables

SELECT *
FROM fsc_evm.bronze.master_variable_keys;

SELECT *
FROM fsc_evm.bronze.master_variable_values;

Dimensional, Fact & EZ Tables

Database specific

SELECT *
FROM mantle.silver.dim_variables;

SELECT *
FROM mantle.silver.fact_variables;

SELECT *
FROM mantle.silver.ez_variables;

Adding, Updating or Removing Project Variables

  1. To modify the project variables, simply update the following seed files in the fsc_evm database;

    • bronze__master_variable_keys: Variable Keys and related configuration details
    • bronze__master_variable_values: Variable Values and related configuration details

  2. Then run the following command: dbt seed -m bronze__master_variable_keys bronze__master_variable_values

Note:

  • If modifying the column structure (including column order, new columns etc.), the seed files will first need to be dropped from the fsc_evm database prior to running dbt seed -m ....
  • Changes to the variables will be reflected in other databases immediately, as they are directly sourced from fsc_evm in each repo's sources.yml. New fsc_evm version tags are not required for seed file changes only.

DO NOT DELETE ANY COLUMNS, KEYS OR VALUES WITHOUT PRIOR AUTHORIZATION FROM FLIPSIDE'S EVM TEAM.

Standardized Naming Convention for Variable KEY

[PACKAGE]_[CATEGORY?]_[ENTITY?]_[PROPERTY]_[ACTION?]

  • ? == optional
  • PACKAGE: Required. Package name with _PACKAGE excluded. Use GLOBAL if spans multiple packages.
    • Examples: MAIN, DECODER, CURATED
  • CATEGORY: Optional. Functional area. Omit if variable applies across multiple categories.
    • Examples: PRICES, ABIS, SILVER, SL (streamline, see "Abbreviations" below)
  • ENTITY: Optional. Specific model/subcategory/component being configured. Omit if variable applies across multiple entities.
    • Examples: RECEIPTS, DECODED_LOGS, TRACES_COMPLETE
  • PROPERTY: Required. The column name, characteristic, feature or aspect of the package, category or entity being configured.
    • Examples: BLOB_BASE_FEE, NODE_SECRET_PATH, FULL_REFRESH, TESTING_LIMIT, PRODUCER_BATCH_SIZE
  • ACTION: Optional. The specific operation or modifier for the property. Omit when the PROPERTY itself fully describes the configuration without needing additional qualification.
    • Examples: ENABLED, DISABLED

Abbreviations:

  • Any abbreviations must be used sparingly, as Variables may be constructed or referenced dynamically and sometimes require full non-abbreviated names. Unapproved usage of abbreviations may result in breaking changes.
  • Approved abbreviations include;
    • Package names → _PACKAGE can be omitted. E.g. MAIN_PACKAGE → MAIN
    • STREAMLINE → SL
    • OBSERVABILITY → OBSERV
    • FULL_REFRESH → FR

Full Variable KEY Examples:

  • GLOBAL_NODE_SECRET_PATH
    • Comprised of: [PACKAGE]_[PROPERTY]
  • MAIN_SL_RECEIPTS_COMPLETE_FR_ENABLED
    • Comprised of: [PACKAGE]_[CATEGORY]_[ENTITY]_[PROPERTY]_[ACTION]
  • MAIN_SL_TRACES_REALTIME_TESTING_LIMIT
    • Comprised of: [PACKAGE]_[CATEGORY]_[ENTITY]_[PROPERTY]
  • DECODER_ABIS_EXPLORER_URL
    • Comprised of: [PACKAGE]_[CATEGORY]_[ENTITY]_[PROPERTY]

Note: These standards also apply to variables with PARENT_KEY functionality enabled. If PARENT_KEY is not NULL, KEY == some_value

  • Example:
    • PARENT_KEY :CURATED_VERTEX_TOKEN_MAPPING
    • KEY: USDC, wMNT, WETH

Macro: get_var(variable_key, default)

  • Variables can now be defined, managed and tracked via master seed data files, rather than listed out in dbt_project.yml. See Querying Project Variables above
    • IS_ENABLED value defaults to FALSE in silver.ez_variables to prevent variables that are not properly defined from being picked up by get_var()
  • Supports various data types including but not limited to: STRING, ARRAY, JSON, VARIANT, OBJECT, NUMBER, INTEGER, FIXED, FLOAT, DECIMAL, BOOLEAN
  • Mapping / Dictionary support for Parent <> Child Key relationships
  • Backwards Compatible with dbt native var() function
    • Variables can continue to be defined in dbt_project.yml if necessary, e.g. get_var() defaults to existing variables in dbt_project.yml.
    • get_var() utilizes similar input parameter structure and rules as var(). If variables do not exist in silver.ez_variables`, values will default to the default value set for the variable from the model/macro it is called in, as expected