Copilot Usage Analytics GPT 5 - NiclasOlofsson/remember-mcp-vscode GitHub Wiki

Copilot Usage Analytics Whitepaper

A developer’s playbook for measuring, visualizing, and improving effectiveness and cost-efficiency across tasks and models

Audience: Hands-on developers who switch models depending on the task (architecture & design, code generation, refactoring/cleanup, project maintenance) and want data-driven guidance on when to use which model, and how to continuously improve.

Version: 1.0

1) Why Track Copilot Statistics?

  • Make model choices data-driven, not habitual.
  • Improve flow by reducing latency where speed matters and investing in quality where outcomes matter more.
  • Control costs by matching model capability to task complexity and monitoring request volume.
  • Build a personal feedback loop to learn which patterns and setups deliver the best results for you.

2) Data Sources and What We Derive

Our pipeline combines:

  • Raw session events (chat sessions, turns, tool call rounds, metadata).
  • Aggregations that transform these events into actionable statistics.

Key raw fields available per turn or request (derived via mapping logic similar to the one used by analytics and usage panels):

  • Time, session/workspace identifiers
  • Agent (chat/edit/workspace/inline)
  • Model identifier/label
  • File path(s) and inferred language
  • Latency
  • Requests per turn via toolCallRounds (backend LLM calls)
  • Turn type (ask vs edit-like modes)

Ground rules for counting:

  • One backend LLM call = one toolCallRound. If missing, fall back to 1/request for compatibility. This is how we track real “model usage.”

3) Core KPIs (with formulas)

Use these foundational metrics everywhere (KPI chips, summaries, and filters). All periods are filterable by time range and workspace.

  • Sessions: unique chat sessions in the period.
  • Turns: total user turns.
  • Requests: sum of backend LLM calls (toolCallRounds).
  • Files: unique files referenced.
  • Models: unique models used.
  • Agents: unique agents used.
  • Median latency (ms): median end-to-end request latency.
  • Edits: count of edit-type turns.
  • Edit ratio: share of turns that were edits.

Formulas (KaTeX):

  • Edit ratio: $edit_ratio = \frac{edits}{turns}$
  • Requests: $requests = \sum_{t \in turns} \left|toolCallRounds(t)\right|$ (fallback: 1 if rounds missing)
  • Median latency: $median({latency_ms_i})$

Optional cost-aware extensions (future-ready if you add cost metadata):

  • Cost per request: $cpr = \frac{\sum cost}{requests}$
  • Cost-efficiency index (example): $CEI = \frac{edit_ratio}{cpr}$

4) Visualizations That Matter

Make insights obvious with a small set of high-signal views:

  1. KPI Chips (top-level snapshot)
  • Sessions, Turns, Requests, Files, Models, Agents, Median Latency, Edits, Edit Ratio.
  • Purpose: A quick “health check” for the selected time range.
  1. Daily Requests Trend (bar/line chart)
  • Requests/day for the chosen period with a 7-day rolling average.
  • Purpose: See spikes, streaks, regressions; correlate with project phases or deadlines.
  1. Model Distribution (ranked list or stacked bar)
  • Requests by model with trend arrows (up/down/stable) period-over-period.
  • Purpose: Reveal overreliance on a single model; identify candidates for cheaper/faster alternatives.
  1. Agent Mix (ranked list)
  • Requests by agent (chat, edit, workspace, inline).
  • Purpose: Match workflow patterns to productivity strategies (e.g., heavy inline usage → optimize latency).
  1. Activity Feed (recent interactions)
  • A compact feed of recent turns with agent/model labels and basic context.
  • Purpose: Debug behaviors, confirm scanning works, spot anomalies fast.
  1. Language/File Heat (optional)
  • Requests by language and by top files/folders.
  • Purpose: Identify hotspots in the codebase; measure refactor campaigns or documentation sprints.

5) Task-Driven Model Selection Guide

Different tasks value different trade-offs. Use stats to route requests to the right model.

A) Architecture & Design (deep reasoning, long context)

  • Priorities: quality, coherence, fewer but more capable calls
  • Ideal signals: higher edit ratio on complex files, acceptable higher latency
  • Guidance: Prefer premium models for long-form reasoning. Accept higher $latency$ if outcomes (edit ratio, reduced rework) are strong.

B) Code Generation (implementation, boilerplate)

  • Priorities: speed and decent accuracy, volume throughput
  • Ideal signals: high requests with low latency, steady edit ratio
  • Guidance: Use fast, cost-efficient models when median latency spikes or cost per request rises.

C) Refactoring & Cleanup (multi-file hygiene, correctness)

  • Priorities: balanced reasoning and speed, consistent acceptance of edits
  • Ideal signals: stable edit ratio, improved file touch rate with minimal rework
  • Guidance: If refactors stall (low edit ratio, high rework), step up to a more capable model for these turns only.

D) General Project Activities (reviews, small fixes, browsing)

  • Priorities: frictionless flow, low latency, predictable responses
  • Ideal signals: quick round-trip times, minimal context needs
  • Guidance: Default to fast models; escalate only when the task reveals deeper complexity.

Practical heuristics (rules of thumb):

  • If $latency_{median}$ > target for simple tasks, switch to a faster model.
  • If $edit_ratio$ falls below a threshold for complex tasks, try a higher-capability model.
  • If model usage concentrates >80% on one model across all tasks, experiment with a cascade (start cheap/fast, escalate only when needed).

6) Decision Framework (How to choose per request)

At the moment of asking Copilot, apply a short checklist:

  1. What task type is this?
  • simple completion, non-critical → prefer fast/cheap
  • refactor/migration, correctness-critical → prefer balanced/higher capability
  • architecture/design → prefer premium reasoning
  1. What does my recent data say?
  • For this task type, which model gave higher $edit_ratio$ under acceptable $latency$?
  • Did the daily trend show cost/latency spikes for my default choice?
  1. Choose model:
  • “Baseline”: fastest viable model
  • “Escalate”: if quality indicators lag (low acceptance, high rework), move to next tier for this turn only
  • “Fallback”: low-cost option when nearing daily budgets
  1. Log and learn:
  • Watch how KPIs respond; iterate on thresholds weekly.

7) Alerts and Thresholds (Actionable, not noisy)

Recommended thresholds (tune to your tolerance):

  • High Latency (for fast loops): $latency_{median} > 2000\text{ ms}$ for code-gen/inline work
  • Low Effectiveness (complex work): $edit_ratio < 0.5$ over last 10+ turns of refactoring/design
  • Cost Spike (if tracked): daily cost > 2× rolling average
  • Context Pressure (if tracked): repeated context overflow events in a session

Each alert should recommend an action:

  • “High latency on inline edits” → switch to faster model for short turns
  • “Low effectiveness on refactors” → escalate to higher-capability model
  • “Cost spike” → reduce capability for low-stakes tasks today

8) Day-to-Day Workflow

Morning (setup)

  • Review KPI chips for your default range (e.g., 7d).
  • Scan the daily requests chart—are we trending up/down? Any anomalies yesterday?

During work

  • For each turn, use the decision framework (Section 6).
  • Watch the activity feed for quick validation of agent/model routing.

End of day

  • Check model distribution—are you overusing a single model?
  • If latency spiked for routine tasks, adjust defaults to a faster model tomorrow.

Weekly review (30 minutes)

  • Compare edit ratio and latency across your top 2–3 models.
  • Capture lessons learned: “For refactors in Typescript, Model X consistently yields higher acceptance with ~+0.5s latency—worth it.”
  • Update personal model cascade playbook.

9) Privacy and Compliance (minimal and aggregated)

Track the least you need:

  • DO: counts, latency, filenames (or redacted paths), languages, agent/model labels.
  • DO NOT: persist source code or PII unless explicitly allowed.

Retention and redaction:

  • Keep aggregated metrics for 30–90 days.
  • Redact sensitive file paths on export or share aggregated-only data.

10) Implementation Mapping (How this aligns with your extension)

You already have the building blocks to deliver the analytics and views described above:

Data and KPIs

  • KPI computation and filters come from (AnalyticsService.getKpis), with helper accessors like (AnalyticsService.getDailyRequests), (AnalyticsService.getAgents), (AnalyticsService.getModels), and (AnalyticsService.getActivity).
  • The turn mapping that feeds these metrics includes sessionId, agent, model, filePath, language, latencyMs, and modelRequests (toolCallRounds).

Accurate request counting

  • The real backend call counting via toolCallRounds is already leveraged in the usage model: model usage is computed from raw session results.

Panels and components (how to present)

  • History panel model/view orchestrate component-level updates:
    • Usage History Model
    • Usage History View
  • Recommended components match your micro-architecture:
    • KPI chips → Kpis from getKpis
    • Daily requests chart → getDailyRequests
    • Agents/models lists → getAgents, getModels
    • Activity feed → getActivity

Filters and ranges

  • Use the panel’s global filters (time range, workspace) to scope metrics. History model applies a user-friendly time range mapping and refreshes downstream view models.

11) Practical Tips That Move the Needle

  • Default to fast models for short-turn tasks; escalate by exception.
  • For design/refactor work, quality beats speed—measure by edit ratio and reduced rework.
  • Don’t fight your trends: if a model is consistently slow in your environment, switch for latency-sensitive tasks.
  • Separate “thinking” turns (higher-capability) from “doing” turns (faster/cheaper). Your stats will reflect clean gains.
  • Keep your daily chart and KPI chips in view—behavior follows what you measure.

12) Roadmap Ideas (Optional Enhancements)

  • Cost tracking per model (adds cost-per-request and cost-efficiency indices).
  • Task tagging per turn (architecture, code-gen, refactor) for per-task model benchmarking.
  • Rolling percentiles (P50/P90/P95) for latency stability insights: visualize $P50, P90, P95$ side-by-side.
  • Anomaly hints (sudden latency spikes, dips in edit ratio) with automatic “try-faster/try-stronger” suggestions.
  • Team view (if shared): compare model preferences and identify best practices to broadcast.

13) Summary

  • Measure a compact set of KPIs that tie directly to choices you can make at the keyboard.
  • Visualize them in a way that makes “what to do next” obvious.
  • Apply a lightweight decision framework per request to route to the right model for the task at hand.
  • Review trends weekly; codify your personal model cascade and evolve it as projects change.

Outcome: Higher effectiveness at lower cost, with fewer regressions and a clearer understanding of where each model shines for your work.