runbook getGoogleAnalytics - willsql4food/LakehouseToolkit GitHub Wiki

Runbook

getGoogleAnalytics ADF Pipeline

See GA4 Ingestion Pipeline for details of the data flow.

Support Notes

Common Failures

Get_gbqObjectDetail

  • Required parameter is missing: query

    Failure happened on 'Source' side. ErrorCode=UserErrorOdbcOperationFailed,
    'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,
    Message=ERROR [HY000] [Microsoft][BigQuery] (100)
    HTTP Error 400: required (Required parameter is missing: query).,
    Source=Microsoft.DataTransfer.Runtime.GenericOdbcConnectors,''
    Type=System.Data.Odbc.OdbcException,
    Message=ERROR [HY000] [Microsoft][BigQuery] (100)
    HTTP Error 400: required (Required parameter is missing: query).,
    Source=Microsoft ODBC Driver for Google BigQuery,'

    Response Action

    • Ignore - No new data exists at this time in Gooble Big Query

BatchToParquet

  • GOAWAY error from Google BigQuery driver

    Failure happened on 'Source' side.
    ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,
    Message=ERROR [HY000] [Microsoft][BigQuery] (310)
    Storage API Error: Error: { code: UNAVAILABLE, message: GOAWAY received },
    Source=Microsoft.DataTransfer.ClientLibrary.Odbc.OdbcConnector,'' Type=System.Data.Odbc.OdbcException,Message=ERROR [HY000]
    [Microsoft][BigQuery] (310) Storage API Error:
    Error: { code: UNAVAILABLE, message: GOAWAY received },Source=Microsoft ODBC Driver for Google BigQuery,'

    Response Action

    • If pipeline fails, rerun pipeline without modification.
      • This should be a transient error handled automaticaly by retry logic.

  • Stream removed error from Google BigQuery driver

    Failure happened on 'Source' side. ErrorCode=UserErrorOdbcOperationFailed,
    'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,
    Message=ERROR [HY000] [Microsoft][BigQuery] (310) Storage API Error:
    Error: { code: UNKNOWN, message: Stream removed },
    Source=Microsoft.DataTransfer.ClientLibrary.Odbc.OdbcConnector,''
    Type=System.Data.Odbc.OdbcException,
    Message=ERROR [HY000] [Microsoft][BigQuery] (310) Storage API Error:
    Error: { code: UNKNOWN, message: Stream removed },
    Source=Microsoft ODBC Driver for Google BigQuery,'

    Response Action

    • Rerun pipeline without modification.

  • On Repeated Failure

    • Modify Loop Batches activity to run sequentially
    • Rerun pipeline

Note

Explanation
Google limits the total row count which can be returned by a query (1M as of 2024-01)
This indicates that the limit was exceeded, either in a single query or multiple large queries running simultaneously

⚠️ **GitHub.com Fallback** ⚠️