How To Read Reports - grishasen/proof_of_value GitHub Wiki

Reports Overview

This document provides overview of the reports, available as templates in the config template

These reports are designed to analyze and visualize business performance across engagement, conversion, model/ML, and customer value (CLV) metrics. They help answer:

  • How well NBA is performing?
  • Which channels/placements/models/segments/issues and groups drive the best results?
  • How do ML models and CDH as recommendation system perform?
  • What is the value and segmentation of the customer base?
  • Are experimental changes statistically significant?

They are useful for:

  • Marketing and Product Teams: To optimize NBA strategy, placements, and targeting.
  • Data Analysts: For deep-dive analysis of engagement, conversion, or model performance.
  • Business Stakeholders: To get quick visual insights for decisions.

How to Read These Reports

Most reports are interactive visualizations. The general principles:

  • Axes, colors, and grouping help slice data by different business dimensions (time, channel, placement, etc).
  • Color is often used for a key variable (e.g., Issue, Group, CTR) to make patterns visible.
  • Facet rows/columns split charts by additional dimensions for side-by-side comparisons.
  • Treemaps, gauges, and funnels give at-a-glance summaries or breakdowns.

For each report, pay attention to:

  • What is being measured? (Look at y-axis or value parameter)
  • How is it grouped? (group_by shows segmentation)
  • What does color represent? (color parameter gives the “category” or “score” you see as colored blocks or lines)
  • Facets? (facet_row/facet_column—if present—show mini-charts by each split)
  • References? (Benchmarks in some gauges, for comparison)

What Report Parameters Do

Below are the key parameters found in most templates:

Parameter What It Controls
------------------------------------ ---------------------------------------------------------------------------------------------------
metric Main topic: "engagement", "conversion", "model_ml_scores", "clv", etc.
type Visualization type: "line", "bar_polar", "gauge", "treemap", "heatmap", "scatter", "boxplot", etc.
description Short report description (shows what is measured and why)
group_by How data is split for analysis: by day, month, channel, placement, issue, group, etc.
x, y Which fields are on X and Y axes (for charts that use axes)
color Variable that determines color coding in the chart (makes grouping or scores visible)
size In scatter plots, variable for bubble size
facet_row, facet_column Extra splits: each value creates a separate row or column of mini-charts
value For gauge or single-number reports: what value is shown
reference Benchmark values (for gauges), for quick context
animation_frame, animation_group For animated/scatter plots: how the animation or grouping works
stages For funnel charts: list of steps in a process
score In descriptive reports, which statistic to display (e.g., "Mean")
log_x, log_y Whether axes are on log scale (scatter plots)

Example: How to Interpret a Report

  • heatmap_month_group_ctr Monthly CTR Heatmap

    • Shows: Click-Through Rate (CTR) as a heatmap by Month (X) and Group (Y).
    • Use: Spot trends—Which groups perform best over time?
    • Color: Indicates the CTR value for each (Month, Group) pair.

  • gauge_channel_placement_ctr Click-through rate (Channel/Placement)

    • Shows: CTR for each (Channel, Placement) as a gauge.
    • Use: Quick check vs. benchmarks.
    • Reference: Shows if current CTR is above/below typical levels.

  • treemap_channel_placement_taxonomy_ctr Click-through rate treemap

    • Shows: Treemap of CTR across channel, placement, issue, group.
    • Use: See the biggest contributors/drivers at a glance.
    • Color: Higher CTR = more prominent color.

  • daily_model_roc_auc_place Daily model ROC AUC By Placement

    • Shows: Model ROC AUC (quality of predictions) by day, broken down by channel/placement/source.
    • Use: Track model performance and stability over time.

  • experiment_z_score Experiment Significance Z Test

    • Shows: Z-score (significance) for experiments.
    • Use: Quickly see if an experimental change is statistically significant.

  • clv_frequency_hist Recency-Frequency-Money (RFM) analysis

    • Shows: Histogram of customer frequency by control group.
    • Use: Analyze distribution of engagement or value segments.

  • describe_dataset_propensity Univariate analysis of various IH properties

    • Shows: Visualizes how the propensity scores (i.e., the model’s estimate of likelihood to act/convert) and other if properties changes over time or across business segments (by Day, Channel, Placement, Issue, Group, etc.).
    • Use: Spotting trends, seasonal patterns, or outliers in model scores across business slices.

  • reports.describe_dataset_boxplot Quartiles of IH data

    • Shows: Distribution of propensity scores using quartiles (median, interquartile range, and outliers) across different groups or over time.
    • Use:
      • Detecting skewed, bimodal, or unexpected distributions in scores.
      • Comparing variation and outliers across groups or time.

  • reports.describe_outcome_funnel Action Outcome Funnel

    • Shows: Step-by-step drop-off from Impressions → Clicked → Conversion for each Channel/PlacementType.
    • Use:
      • Quickly identifying at which stage most users drop off.
      • Comparing funnel performance across channels or placements.

Quick Tips

  • Choose your report by what you want to answer (CTR, Conversion, ML, CLV, etc).
  • Adjust parameters (date range, channels, issues, etc) to focus your analysis.
  • Look at colors, facets, and axes to spot differences or trends.
  • Gauges/References help you see if results are “good” or “bad” versus expectations.
  • Use treemaps/funnels for summary and line/heatmap/scatter for trends or details.