Streamline enables us to rapidly scale our data processing and ingestion capabilities with the use of AWS Lambdas, Snowflake External Tables, and DBT Models. Please follow this guide for new Streamline deployments only.

• Thorough testing in STG/DEV is required before deploying to PROD. There are cost and other downstream implications to each deployment and invocation of streamline.
• Depending on the integration or blockchain, there may be nuances to the implementation process. This is intended to be a general guide, please reach out to the Analytics Engineering team if you have additional questions or concerned.

1. Define variables in dbt_project.yml;

   • STREAMLINE_INVOKE_STREAMS: Allows Streamline Macros to execute if set to TRUE
   • STREAMLINE_USE_DEV_FOR_EXTERNAL_TABLES: Points models to source the DEV database if set to TRUE and defined in sources.yml e.g. conjunction with {{ "<DATABASE>_DEV" if var("STREAMLINE_USE_DEV_FOR_EXTERNAL_TABLES", False) else "<DATABASE>" }}
   • To determine the name for the API_INTEGRATION, run SHOW INTEGRATIONS; in Snowflake and apply the following grant;

   GRANT USAGE ON INTEGRATION <INTEGRATION NAME> TO ROLE DBT_CLOUD_<DATABASE>;

   • Additionally, add the following section and update accordingly.

   #### STREAMLINE 2.0 BEGIN ####
   
   API_INTEGRATION: '{{ var("config")[target.name]["API_INTEGRATION"] if var("config")[target.name] else var("config")["dev"]["API_INTEGRATION"] }}'
   EXTERNAL_FUNCTION_URI: '{{ var("config")[target.name]["EXTERNAL_FUNCTION_URI"] if var("config")[target.name] else var("config")["dev"]["EXTERNAL_FUNCTION_URI"] }}'
   ROLES: |
     ["INTERNAL_DEV"]
   config:
   # The keys correspond to dbt profiles and are case sensitive
     dev:
       API_INTEGRATION: AWS_<DATABASE>_API_STG_V2
       EXTERNAL_FUNCTION_URI: <INSERT_URI>
       ROLES:
         - AWS_LAMBDA_<DATABASE>_API
         - INTERNAL_DEV
   
     prod:
       API_INTEGRATION: AWS_<DATABASE>_API_PROD_V2
       EXTERNAL_FUNCTION_URI: <INSERT_URI>
       ROLES:
         - AWS_LAMBDA_<DATABASE>_API
         - INTERNAL_DEV
         - DBT_CLOUD_<DATABASE>
   
   #### STREAMLINE 2.0 END ####

2. Ensure the latest fsc-utils and fsc-evm packages are installed. See Release Versions and/or tags for details. Run dbt clean && dbt deps to install the latest packages.

3. Create the relevant DBT models, dependent on the intended integration. See the Core, Decoder, Balances etc. Streamline Setup Guides in the fsc-evm Wiki for more details.

4. Deploy relevant External Tables in streamline-snowflake

   • New External Tables may be required, dependent on the intended structure and/or latest Streamline Release Version.
   • Please see the streamline-snowflake README for deployment details. Additional privileges may be required.

5. Follow the Streamline Deployment Steps below;

   • Note. Please use --target prod for PROD deployments and --target dev for DEV deployments. Additional privileges may be required.

   • Drop the _live schema with DROP SCHEMA <DATABASE>._live; prior to deploying the latest livequery core package

   • Deploy the latest livequery core package with dbt run -m livequery_models.deploy.core --vars '{UPDATE_UDFS_AND_SPS: true}'

   • Grant usage on schema/all functions to DBT_CLOUD_<DATABASE> and AWS_LAMBDA_<DATABASE>_API roles

     GRANT USAGE ON SCHEMA <DATABASE>._live TO AWS_LAMBDA_<DATABASE>_API; 
     GRANT USAGE ON ALL FUNCTIONS IN SCHEMA <DATABASE>._live TO AWS_LAMBDA_<DATABASE>_API;
     GRANT USAGE ON SCHEMA <DATABASE>._live TO DBT_CLOUD_<DATABASE>; 
     GRANT USAGE ON ALL FUNCTIONS IN SCHEMA <DATABASE>._live TO DBT_CLOUD_<DATABASE>; 
     
     -- if insufficient privilege or function not found errors occur on _utils then also apply the following grants...
     GRANT USAGE ON SCHEMA <DATABASE>._utils TO AWS_LAMBDA_<DATABASE>_API; 
     GRANT USAGE ON ALL FUNCTIONS IN SCHEMA <DATABASE>._utils TO AWS_LAMBDA_<DATABASE>_API;
     GRANT USAGE ON SCHEMA <DATABASE>._utils TO DBT_CLOUD_<DATABASE>; 
     GRANT USAGE ON ALL FUNCTIONS IN SCHEMA <DATABASE>._utils TO DBT_CLOUD_<DATABASE>; 

   • Deploy the EVM streamline UDFs with dbt run-operation fsc_utils.create_evm_streamline_udfs --vars '{UPDATE_UDFS_AND_SPS: true}'

   • (Optional) Full-refresh the Streamline Bronze layer and/or Streamline Complete models. If building in the DEV environment, set STREAMLINE_USE_DEV_FOR_EXTERNAL_TABLES to TRUE.

• Vars in dbt_project.yml
• streamline-snowflake (External Tables)