Metrics and Scores Configuration

The config.toml file allows users to customize various aspects of metric calculations without modifying the core scoring logic. This ensures consistency and reliability in the computed scores.

Important Note ⚠️

Do not modify the scores entries within the metrics configurations. These entries define the specific metrics the application calculates and are integral to its functionality. Altering them may lead to incorrect computations or application errors.

Configurable Parameters

For each metric section (e.g., [metrics.conversion], [metrics.engagement]), you can adjust the following parameter:

1. group_by

• Purpose: Specifies the dimensions by which data should be grouped before metric calculation.
• Example:

  group_by = ['Day', 'Channel', 'CustomerType']

• Usage: Modify this list to change how data is segmented in report.

2. filter

• Purpose: Applies a global filter to the dataset, allowing you to include or exclude specific data point.
• Example:

  filter = """pl.col("ModelControlGroup").is_in(["Test", "Control"])"""

• Usage: Use this to focus on particular subsets of data relevant to your analysis.

3. positive_model_response and negative_model_response

• Purpose: Defines which values in the dataset are considered positive or negative responses for model evaluation.
• Example:

  positive_model_response = ["Clicked"]
  negative_model_response = ["Impression", "Pending"]

• Usage: Adjust these lists to align with the specific response indicators in your dataset.

4. columns (Specific to [metrics.descriptive])

• Purpose: Lists the data columns to be included in descriptive statistical analysis.
• Example:

  columns = ['Outcome', 'Propensity', 'FinalPropensity', 'Priority']

• Usage: Modify this list to include additional columns for descriptive statistics.

5. use_t_digest (Specific to [metrics.model_ml_scores] and [metrics.descriptive])

• Purpose: Determines whether to use t-digest for calculating ROC AUC and average precision score.
• Example:

  use_t_digest = "true"

• Usage: Set to "true" to enable t-digest calculations, which can be beneficial for large datasets.

6. experiment_name and experiment_group (Specific to [metrics.experiment])

• Purpose: IH property where where name of the experiment and correspondent control group name are stored.
• Example:

  experiment_name = "ExperimentName"
  experiment_group = "ExperimentGroup"

• Usage: Set to IH properties used in your case.

7. CLV-Specific Parameters

Within [metrics.clv]:

• Purpose: Defines parameters specific to Customer Lifetime Value calculation.
• Example:

  order_id_col = "HoldingID"
  customer_id_col = 'CustomerID'
  monetary_value_col = 'OneTimeCost'
  purchase_date_col = 'PurchasedDateTime'
  lifespan = 3

• Usage: Adjust these parameters to match the structure and semantics of your customer transaction data.

---

Best Practices

• Backup Before Changes Always create a backup of the original config.toml file before making modification.
• Consult Documentation Refer to the application's documentation or seek advice if unsure about any configuration change.