12.2 Evidence Sufficiency Index (ESI) - ravkorsurv/kor-ai-core GitHub Wiki

12.2 Evidence Sufficiency Index (ESI)

The Evidence Sufficiency Index (ESI) is a metric developed by Kor.ai to complement the Bayesian risk score. While the Bayesian model outputs a probabilistic risk value, ESI provides a measure of how well-supported that score is, based on input diversity, quality, and distribution. This enables analysts to differentiate between alerts that are high-risk and well-evidenced versus those driven by sparse or noisy data.


๐ŸŽฏ Purpose

ESI enables:

  • Trust calibration for analysts and reviewers.

  • Filtering and triage based on evidence quality.

  • Better prioritisation in high-volume alert environments.

  • Enhanced explainability and defensibility in audit and STOR reviews.


๐Ÿงฎ Core Inputs for ESI Calculation

Component Description
node_activation_ratio Proportion of active (populated) nodes in the BN.
mean_confidence_score Average confidence level of inputs (mapped to numeric).
fallback_ratio Proportion of nodes relying on priors or latent defaults.
contribution_entropy Entropy of node contributions โ€” measures distribution evenness.
cross_cluster_diversity Evidence spread across distinct node groups (e.g. trade, comms, PnL).

Example Calculation (score between 0 and 1):

ESI = W1 * node_activation_ratio +
      W2 * mean_confidence_score +
      W3 * (1 - fallback_ratio) +
      W4 * contribution_entropy +
      W5 * cross_cluster_diversity

๐Ÿ“Š Output Example

{
  "evidence_sufficiency_index": 0.84,
  "node_count": 6,
  "mean_confidence": "High",
  "fallback_ratio": 0.0,
  "contribution_spread": "Balanced",
  "clusters": ["PnL", "MNPI", "TradePattern"]
}

๐Ÿ–ฅ UI Integration Strategy

Dual Display (Recommended)

  • Primary: Bayesian Risk Score

  • Secondary: ESI Score + Badge (Strong, Moderate, Sparse)

Analyst UI Features:

  • Badges on Alert Cards (e.g. ESI: Strong Evidence)

  • Sort/Filter Controls:

    • Filter alerts with ESI > 0.7

    • Sort by ESI descending

  • Explain Panel Tooltips:

    • โ€œThis alert is supported by evidence from 5 distinct factors with high confidence and no fallback reliance.โ€


๐Ÿ› ๏ธ Use in Tuning & Backtesting

  • ESI can be used as a multiplier in simulation:

    Adjusted Risk Score = Risk Score * ESI
    
  • This can help:

    • Evaluate impact of noisy nodes

    • Filter low-ESI alerts in QA workflows

    • Simulate precision/recall tradeoffs


๐Ÿ” Future Enhancements

  • Analyst Feedback Learning: Use confirmed STORs or dismissals to adjust ESI weightings.

  • Sensitivity Maps: Show how much ESI would drop if a key input was removed.

  • Percentile Labels: Show where an alertโ€™s ESI sits within a weekly/monthly percentile range.


โœ… Summary

The Evidence Sufficiency Index is a key enabler for moving beyond opaque alert scores. It allows Kor.ai to explain not just why an alert was scored as risky, but how trustworthy and complete the supporting evidence is. This separation of risk and sufficiency enables a new standard of transparency in AI-powered surveillance.

โš ๏ธ **GitHub.com Fallback** โš ๏ธ