Customize Tracing - microsoft/MSO-Scripts GitHub Wiki

Event Tracing for Windows (ETW) can collect data from any of thousands of different 'providers' originating from:

  • Certain Applications (Office, Edge, Chrome, ...)
  • Windows
  • The CPU itself

MSO-Scripts enables specific ETW providers to analyze the behavior and resource usage of Windows Applications.

You can easily customize this in a variety of ways.

ETW Providers

Q. How do I even know which ETW Providers (data sources) are available?
A. ETW has these classes of data providers:

Available System Providers are listed here, and their corresponding stackwalk options.

Each of these commands creates a list of Event Providers registered on the current system:

  • WPR -providers
  • XPerf -providers I
    XPerf.exe is often adjacent to WPA.exe, or part of the ADK.
  • LogMan query providers
    Details of a specific ETW provider: LogMan query providers <Name of Provider>
  • wevtutil ep
    Details of a specific ETW provider: wevtutil gp <Name of Provider>
  • PowerShell: Get-NetEventProvider -ShowInstalled | Format-Table -Property Name,GUID

Note

Certain applications make available many additional ETW Event Providers by way of TraceLogging. These are discovered via documentation. See: Windows Tracing Mechanisms

Environment Variables

MSO-Scripts can enable extra ETW providers (without modifying the scripts) by way of two environment variables:

WPT_XPERF = a list of ETW Event Provider definitions in 'XPerf -on' format, separated by '+'

Format: Name|GUID : KeywordFlags : Level : Stack (with additional definitions separated by '+')

  • Name|GUID:  Name or GUID of an ETW provider (Mandatory)
  • KeywordFlags:  64-bit numeric value indicating which subsets of events to enable (default = all) (Optional)
  • Level:  Logging level (0-255): 1=Critical, 2=Error, 3=Warning, 4=Information, 5=Verbose (default) (Optional)
  • Stack:  If present, a call stack will be collected for each event. (Optional)

Example: Microsoft-Windows-TCPIP + Microsoft-Windows-RPC:::Stack + e53c6823-7bb8-44bb-90dc-3f86090d48a6:0x00A4:4:Stack
Interpretation: Add 3 ETW providers: TCPIP + RPC with stackwalk + Winsock-AFD, keyword 0x00A4, level 4, with stackwalk

Note

MSO-Scripts uses the WPT_XPERF environment variable to automatically create a WPR Recording Profile: WPR_XPERF_Profile.wprp

See: Create a Custom Recording Profile

WPT_WPRP = a list of WPR Recording Profiles; semicolon-separated, in any of these formats:

  • Full path with profile name: c:\MyProfiles\MyProfile.wprp!ProfileName
    To see a list of available Profile Names, run: WPR -Profiles c:\MyProfiles\MyProfile.wprp
  • Relative path with profile name: .\WPRP\OtherProfile.wprp!ProfileName (relative to the script's folder)
    or ..\WPRP\OtherProfile.wprp!ProfileName (relative to the root script folder)
  • The name of a Built-in Recording Profile, eg. Registry
    To see a list of available Built-in Recording Profiles, run: WPR -Profiles
    These Built-in Recording Profiles optionally take a suffix: .Light or .Verbose
    To see the contents of a Recording Profile, run: WPR -ProfileDetails Registry.Light

Tip

Every WPR Profile has a Logging Mode: File and/or Memory. (MSO-Scripts defaults to File Logging Mode.)
To trace in Memory Logging Mode (Circular Buffer), run the Trace* Start script with -Loop: TraceCPU Start -Loop ...

Note

Separate WPRP Recording Profiles can create separate event collectors, which may record duplicate copies of various events, and could max out WPR's session limit. MSO-Scripts avoids this by giving all <EventCollector> elements the same name: "MSO Event Collector"
See: Controlling the Event Session Name

To list the collectors in a running MSO-Scripts profiling session, use Status -verbose, for example:
    .\TraceCPU Status -verbose
To list the collectors defined by a set of WPR Profiles, run WPR with -ProfileDetails, for example:
    WPR -ProfileDetails ".\WPRP\CPU.wprp!CPU-Dispatch" + ".\WPRP\FileDiskIO.wprp!FileAndDiskIO"
In each case look for "Collector Name". Typically there will be a System Collector and an Event Collector, and under the System Collector there should be at least: Loader and ProcessThread

See also: Create a Custom Windows Recording Profile (.WPRP file)

Modify the Scripts (Advanced)

The MSO-Scripts' PowerShell and XML code is formatted to be relatively easy to modify:

  • Add a new WPR Recording Profile (.WPRP file):
    In the Trace*.ps1 script, find the definition of $TraceParams.RecordingProfiles and add a recording profile description:

    • Full path with profile name: c:\MyProfiles\MyProfile.wprp!ProfileName
      To see a list of available Profile Names, run: WPR -Profiles c:\MyProfiles\MyProfile.wprp
    • Relative path with profile name: .\WPRP\OtherProfile.wprp!ProfileName (relative to the script's folder)
      or ..\WPRP\OtherProfile.wprp!ProfileName (relative to the root script folder)
    • The name of a Built-in Recording Profile, eg. Registry
      To see a list of available Built-in Recording Profiles, run: WPR -Profiles

    [See: Authoring Custom WPR Profiles]
    [See the Technical Note above about event duplication when multiple Recording Profiles enable the same providers.]

  • Modify an existing WPR Recording Profile (WPRP\*.wprp file):

    • Find and add to the <EventProvider> and <EventProviderId> records.
    • Find and add to the <SystemProvider> and <SystemProviderId> records.

    See: System and Event Provider Definitions and ETW Providers (above)

CPU Monitor Counters (Advanced)

Modern CPUs have many internal event counters, which ETW is able to expose.
These event counters include: Instructions Retired, Cache Misses, Branch Mispredicts, and numerous others.

SHORT STORY

ETW has two mechanisms for capturing CPU event counters:

  1. SAMPLING:  Every N CPU events, capture an ETW event with stackwalk.
    Over time, this will generate a statistical picture of where those specific CPU events are happening.
  2. GATHERING:  Gather the various CPU counters whenever a specific Windows Kernel event occurs.
    That specific Windows Kernel event is usually a Thread Context Switch (CSWITCH), so that the CPU counts are associated with the thread which was executing on that logical CPU.

Quick Start:

  • To gather-trace Cycles per Instruction Retired (CPI), run one of:

    • MSO-Scripts\BETA\TracePMC Start -CPI
    • MSO-Scripts\BETA\PMC.bat

  • To sample-trace Branch Mispredicts and Cache Misses, run one of:

    • MSO-Scripts\BETA\TracePMC Start -PMC *
    • MSO-Scripts\BETA\PMCs.bat

  • To view the resulting trace, run:

    • MSO-Scripts\BETA\TracePMC View [-Path <path>\<file>.etl]

    WPA has special views for CPI, Branch Mispredicts, and Cache Misses.


LONG STORY

Windows is pre-configured with a variety of CPU Counters. Others can be added.

  • To list the currently available CPU Counters, run one of:
    • WPR -PMCSources
    • XPerf -PMCSources
      If there is only one counter listed then your machine is probably configured for HYPER-V.

Additional CPU-specific Counters can be registered via the Windows Registry.
For Intel CPUs, run:
    MSO-Scripts\BETA\Generate-PmuRegFile -Description
and follow the instructions at the end.

  • To sample-trace specific CPU counters, run one of:
    • MSO-Scripts\BETA\TracePMC Start -PMC Counter1[,Counter2[,...]]
    • MSO-Scripts\BETA\PMCs.bat Counter1 [Counter2 [...]]
  • To gather-trace specific CPU counters at each Thread Context Switch (CSWITCH) event, run:
    • MSO-Scripts\BETA\PMC.bat Counter1 [Counter2 [...]]
  • To view the resulting trace, run one of:
    • MSO-Scripts\BETA\TracePMC View [-Path <path>\<file>.etl]
    • WPA <path>\<file>.etl -profile MSO-Scripts\BETA\WPAP\CPUCounters.wpaProfile

FURTHER READING


WINDOWS PERFORMANCE RESOURCES

⚠️ **GitHub.com Fallback** ⚠️