Requires fsc-evm v4.0.0+

1. Overview
2. Upgrading Packages
3. Enabling Models & Packages
4. Snowflake Tasks & Github Actions Workflows
5. Project Specific Pipelines & One-off Models

This documentation provides guidance for maintaining existing EVM repos that depend on the fsc-evm framework. It covers essential maintenance tasks such as upgrading to newer fsc-evm package versions, configuring project-specific settings through dbt_project.yml and packages.yml files, and managing deployment workflows via Makefiles and GitHub Actions. The framework allows projects to leverage standardized EVM data processing patterns and shared resources like macros and documentation, while still providing flexibility to customize models, workflows, and deployment processes for specific blockchain requirements.

• packages.yml:
  • There's a handful of required dbt packages to install within the fsc-evm repo. These include fsc-utils, livequery and various others for usage in processes such as snowflake table tagging and dbt tests.
    • fsc-utils contains various Macros, UDFs (User Defined Functions) and UDTFs (User Defined Table Functions) that we leverage in our dbt models, such as udf_hex_to_int.
    • livequery-models is inherited through the fsc-utils package, and is required for powering Streamline, in addition to other helpful functions such as the Github UDTFs.
  • The majority, if not all, package installations will flow through to EVM project repos via fsc-evm. This means that there is only 1 major package we'll typically need to install in each project repo, and that's fsc-evm. Version tags follow traditional semantic versioning and can be found in either the releases or tags page on Github. Upgrade a project repo to the latest fsc-evm version by editing the revision: portion of the package.

{# fsc-evm packages.yml #}

packages:
  - package: calogica/dbt_expectations
    version: 0.8.2
  - package: dbt-labs/dbt_external_tables
    version: 0.8.2
  - package: dbt-labs/dbt_utils
    version: 1.0.0
  - git: https://github.com/FlipsideCrypto/fsc-utils.git
    revision: v1.35.1
  - package: get-select/dbt_snowflake_query_tags
    version: [">=2.0.0", "<3.0.0"]

{# project repo packages.yml #}

packages:
  - git: https://github.com/FlipsideCrypto/fsc-evm.git
    revision: v4.0.0

• requirements.txt:
  • Simple text file that establishes the dbt-snowflake version for dbt models to run with. Though rare, this file can also be used to pin a certain package or dependency to a specific version, which can help resolve 3rd party bugs.
  • Defined in both fsc-evm and project repos. Requirements may vary.

dbt-snowflake>=1.7,<1.8
protobuf==4.25.3

• Makefile:
  • Build automation for dbt: Automates complex dbt commands, eliminating the need to remember lengthy command sequences with multiple tags and variables
  • Includes phased deployment commands for new projects
  • Environment-aware: Allows different deployment targets (dev vs prod) with conditional logic for varying integrations such receipts or receipts_by_hash, external table configurations, and testing limits
  • Workflow automation: Dedicated tasks for deploying and managing GitHub Actions workflows and Snowflake Tasks

• dbt_project.yml: Example
  • Configuration settings for the repo. This file is crucial for accessing packages, file paths, enabling/disabling dbt models and establishing variables.
  • name: Project names should contain only lowercase characters and underscores. This typically matches the name of the repo, but with an underscore rather than a dash between <chain>-models
  • profile: Profile names should contain only lowercase characters. This setting configures which "profile" dbt uses for this project. This typically matches the name of the database / blockchain.
  • models: This is where we enable various packages and sub-packages from fsc-evm. Top level packages should be disabled by default (+enabled: false), while sub-packages or specific models can be enabled as necessary (+enabled: true). Failure to disable top level packages by default may result in compilation failures in the production environment. Please see the full dbt documentation for more details.
  • vars: This section is used to define the values of any variables that use dbt's native var() function. For the majority of fsc-evm variables, we use a custom implementation through the get_var() function. However, var() is still valid and may be defined necessary, such as for Streamline configs and other non-fsc-evm variables. See the project variable docs for more details.
  • DBT Context Windows: In rare cases, get_var() may not function properly due separate context windows in a DBT Model. Specifically, variables defined via get_var() and return_vars() may not always process correctly when used in the {{ config() }} portion of a dbt model if the variables are sourced from project_vars/. Please reach out to the EVM engineering team for questions or concerns.
• sources.yml:
  • All sources used in any fsc-evm model or macro MUST be defined in fsc-evm's sources.yml file and the project repo's sources.yml file. Failure to do so may result in compilation errors.
• data/:
  • This directory contains .csv seed files for use in dbt models. Examples include observability__exclusion_list.csv, which is used for excluding blocks from various tests if they do not fit the testing structure.
• docs:
  • Documentation for dbt models no longer lives in project specific repos. Instead, all EVM documentation is centralized to fsc-evm doc_descriptions. This enables a more accountable, standardized and easier to manage approach to documentation across EVMs. In order to access fsc-evm defined docs, simple add to the docs-path in dbt_project.yml.

docs-paths:
  ["dbt_packages/fsc_evm/doc_descriptions", "models/doc_descriptions", "models"]

• Fsc-evm:
  • Contains github actions workflow templates, for use in project repos. These templates utilize the project name and profile name defined in dbt_project.yml, alongside other inputs, to establish the env vars and secrets required to run a workflow.
  • Other workflows in fsc-evm may include those needed for alerting, adhoc runs, and the Variable Management System.
• Project Repos:
  • Contain various workflows for with dedicated dbt run commands, dependent on the models that need to be run or tested. These workflows all leverage the fsc-evm workflow templates, which have input fields that allow for customization, which can be modified as needed for each chain. Schedules are defined in the github_actions schema, and can be overridden via project_vars/.

• While fsc-evm is powerful for centralized code management across all EVM chains, it may not always be the best, most resource effective solution for our needs. In cases where a pipeline or model only applies to one blockchain/database/repo (and will always only apply to one), it may make sense for it to live outside of fsc-evm. Examples of this include the Ethereum Beacon Chain and the Avalanche subnet Dexalot. These models may still leverage fsc-evm variables, macros and models to a degree, but for the most part are considered stand-alone pipelines with custom implementations. Please reach out to the EVM engineering team for questions or concerns.