Bronze models in our dbt project serve as the initial layer of data transformation, creating views on top of our raw data sources. These models are designed to be flexible and configurable, allowing for easy adaptation to different data sources and chain specifications.

Currently, we have the two types of models for each source:

• _fr: these views will look at all files in the external table
• views without the _fr suffix only look at files inserted in the last three days
• sources include:
  • blocks
  • transactions
  • traces
  • receipts / receipts_by_hash
    • table is set based on the presence of the GLOBAL_USES_RECEIPTS_BY_HASH variable.
  • decoded_logs

Each model can be customized using various variables and macros.

Each model can be customized using the following variables:

Replace <MODEL_NAME> with the specific model name (e.g., BLOCKS_TRANSACTIONS, TRACES).

To override the default variables for a bronze model, you can set variables in your dbt_project.yml file. For example:

vars:
...
  GLOBAL_USES_RECEIPTS_BY_HASH: True
  BLOCKS_SOURCE_NAME: 'BLOCKS_V2'
  TRACES_BLOCK_NUMBER: False
  RECEIPTS_BY_HASH_PARTITION_FUNCTION: "CAST(SPLIT_PART(SPLIT_PART(file_name, '/', 4), '_', 1) AS INTEGER)"
...

This allows you to adapt the bronze models to different data sources and chain specifications without modifying the core model code.

To run a bronze model in another repo, you can do so with something like:

For a single model:

dbt run -m core_package.bronze.bronze__blocks

For all bronze models:

dbt run -m core_package.bronze

For a tag:

dbt run -m "fsc_evm,tag:core"

The bronze models use several macros to set default variables and log details:

• set_default_variables_bronze: Sets default variables for the bronze model, including the partition function, partition join key, and block number extraction.
• log_bronze_details: Logs details about the bronze model for debugging purposes. This is optional but useful for debugging. It will only log when you compile the model.

Here's a breakdown of the bronze__blocks.sql model:

1. Set source name and model type:

   {% set source_name = var('BLOCKS_SOURCE_NAME', 'BLOCKS') %}
   {% set model_type = '' %}

2. Set default variables:

   {%- set default_vars = set_default_variables_bronze(source_name, model_type) -%}

3. Log model details:

   {{ log_bronze_details(...) }}

4. Configure the model:

   {{ config (
       materialized = 'view',
       tags = ['streamline_core_complete', 'bronze_external']
   ) }}

5. Create metadata CTE and join with source data:

   WITH meta AS (...)
   SELECT
       s.*,
       b.file_name,
       b._inserted_timestamp,
       ...
   FROM
       {{ source("bronze_streamline", source_name.lower()) }} s
       JOIN meta b
       ON b.file_name = metadata$filename
       AND b.partition_key = s.partition_key
   WHERE
       b.partition_key = s.partition_key
       AND DATA :error IS NULL
       AND DATA IS NOT NULL

1. Always use the provided macros for setting default variables and logging details.
2. When adding new bronze models, follow the existing naming conventions and structure.
3. Document any new variables or customizations in this wiki page.
4. Test thoroughly when changing partition functions or join keys.