Collect a Network Trace - microsoft/CSS_SQL_Networking_Tools GitHub Wiki

Collect a Network Trace

Choosing a Capture Method

The main tools we will be discussing are NETMON and NETSH, though WireShark is a good alternative and is also available on Linux systems. Your company may have a preferred capture tool already approved or in-place and it would be best to use that.

Depending on the scenario you want, different tools may offer some advantages over other tools.

Scenario Recommendation
Easily Reproducible Issue If the issue is reliably reproducible, testing using WireShark or NETMON using the GUI is a relatively simple method for performing the capture.
Long-Running Capture Use NMCAP or WireShark with chain files. You can also use a circular capture, but have to terminate the capture quickly to prevent relevant data from being overwritten. Chain files are preferred. You may have to monitor how much disk space they take up and delete older files if your system is low on storage. WireShark can limit the number of chain files, NMCAP does not.
Cannot Install On the Server NETSH is installed on every version of Window since 2008 R2. You can capture a trace without having to install another tool.
Winsock Details By using SCENARIO=NETCONNECTION, NETSH will also log WinSock and other low-level TCP events to the capture file.
Linux or MAC Systems You can use WireShark on these systems or any other capture tool that is compatible with the PCAP format or other format that WireShark can open.
VPN Traffic Use WireShark for best results.
Capture loopback/local network traffic Installing NPCAP as part of the WireShark installation allows you to select the loopback adapter for capturing local network traffic, i.e. client and server are on the same machine.

Note: Chain files are additional files generated once the capture has recorded a predetermined amount of data. Instead of having 6GB in a single file, which most tools won't open, you could have 12 x 500MB or 30 x 200MB capture files. SQL Network Analyzer can analyze all the chained trace files at once as if they were a single larger file. The report will tell you which file the issue is in, and the smaller files open and filter a lot more quickly than the larger files.

Note: To read NETSH files, you need to use NETMON. You can also convert from ETL to PCAPNG using this utility: https://github.com/microsoft/etl2pcapng

Note: Do not capture to the SQL Server data disk. Do not capture to a network share. If possible, avoid capturing to the C: drive as it could impact paging performance, etc.

Note: You must run the application or command prompt "As Administrator" in order to perform a packet capture. To analyze a captured file, you can run as an ordinary user.

Download Locations

Tool Download Location
NETMON http://www.microsoft.com/en-us/download/details.aspx?id=4865
Perform a complete install.
WireShark www.wireshark.org
NETSH Built into Windows 7 and Windows 2008 R2 and later.

Sample NETSH Commands

Note: NETSH must be run from the Admin command prompt.
Note: NETSH does not allow for chained captures. See batch commands below for a workaround.
Note: Unless explicitly specified, the trace file ends up in %LocalAppData%\temp\NetTraces\NetTrace.etl
Note: Using the SCENARIO=NETCONNECTION command-line option results in a CAB file with the ETL trace file in it, plus some other files containing the basic network configuration of the machine.

Description Command
Stop the capture NETSH TRACE STOP
Some post processing must be performed and may take a minute or two, so do not close the window until the command prompt appears again.
Show if a trace is running NETSH TRACE SHOW STATUS
Basic capture, unlimited growth NETSH TRACE START CAPTURE=YES MAXSIZE=0
Capture with socket events NETSH TRACE START SCENARIO=NETCONNECTION CAPTURE=YES TRACEFILE=c:\temp\%computername%.etl
Circular capture and truncate packets NETSH TRACE START SCENARIO=NETCONNECTION CAPTURE=YES TRACEFILE=c:\temp\%computername%.etl FILEMODE=CIRCULAR MAXSIZE=2048 PACKETTRUNCATEBYTES=250
NETSH has a larger header than other capture formats, so do not truncate much below the amount suggested.
Filter the data during capture NETSH TRACE START CAPTURE=YES IPv4.Address=10.10.10.10 Protocol=!DNS MAXSIZE=0
Note the use of '=' instead of '==' in the filter. For more help on filtering options: NETSH TRACE SHOW CAPTUREFILTERHELP.

NETSH does not allow for a chained capture, but LOGMAN does. LOGMAN cannot initiate a network capture, but can log the same provider as NETSH. By combining the commands into a batch file, you can get the best of both worlds:

StartChainedCap.cmd
These commands clear various caches before starting the trace. A TASKLIST helps correlate process ID's in the trace to application names.
Pasting the commands directly into the command window won't work because of the % symbols in the command are treated differently.

cd /d %~dp0
IPCONFIG /flushdns
NBTSTAT -R
Powershell -command "Get-WmiObject Win32_LogonSession | Where-Object {$_.AuthenticationPackage -ne 'NTLM'} | ForEach-Object {c:\windows\system32\klist.exe purge -li ([Convert]::ToString($_.LogonId, 16))}"
netsh trace start capture=yes maxsize=1 TRACEFILE=.\deleteme.etl
logman start ndiscap -p Microsoft-Windows-NDIS-PacketCapture -mode newfile -max 200 -o .\%computername%%%d.etl -ets
tasklist > .\TasklistAtStart.txt

Note: The CD command moves the "current directory" to the location of the cmd file.
The NETSH command starts the capture into a small circular file called deleteme.etl. LOGMAN piggy-backs this and logs to the computername file with a numeric suffix every 200MB (-max 200). Keep this size small to allow faster opening and filtering. Both files are stored in the current directory.

StopChainedCap.cmd
Delete the temporary etl file created by NETSH and give the CAB file a better name.

cd /d %~dp0
tasklist > .\TasklistAtEnd.txt
logman stop ndiscap -ets
netsh trace stop
del .\deleteme.etl
rename .\deleteme.cab network_settings.cab

Additional NETSH and LOGMAN command ideas

Sample NMCAP Commands

Note: NMCAP gets installed with NETMON.
Note: NMCAP must be run from an Admin command prompt.

Description Command
Stop the capture CTRL+C
Do not close the window until the command-prompt reappears. Doing so will prevent the frame table from being written and the file will contain no identifiable frame data. If you capture a chained trace and the last file is bad, closing the window too soon is usually the cause.
Basic non-chained capture with Process names NMCAP /network * /capture /file c:\data\%computername%.cap /captureprocesses
Basic chained capture NMCAP /network * /capture /file c:\data\%computername%.chn:100M
Note CHN file extension in the command. Produces multiple files of 100MB each.
Chained capture with packet truncation NMCAP /network * /capture /file c:\data\%computername%.chn:100M /maxframelength 180
Auto-stop after 4 hours NMCAP /network * /capture /file c:\data\%computername%.chn:100M /maxframelength 180 /stopwhen /timeafter 240 min
Auto-stop at a set time NMCAP /network * /capture /file c:\data\%computername%.chn:100M /maxframelength 180 /stopwhen /time 3:00:00 PM 9/10/2020
Auto-start at 3 AM and stop 2 hours later NMCAP /network * /startwhen /time 3:00:00 AM 9/10/2016 /capture /file c:\data%computername%.chn:100M /stopwhen /timeafter 120Min
Filter packets during capture NMCAP /network * /capture (!ARP AND !ICMP AND !NBTNS AND !BROWSER) /file NoNoise.cap
Filter words in Frame description NMCAP /network * /capture contains(.Property.Description, "Continuation") /File TCPContinuations.cap
Start and stop on a filter condition NMCAP /network * /startwhen /frame tcp.flags.syn==true and ipv4.address=10.10.10.10 /capture /file c:\data\%computername%.chn:100M /stopwhen /frame (tcp.flags.fin==true or tcp.flags.reset==true) and ipv4.address==10.10.10.10

Note: In Auto-start scenarios, NMCAP will begin capturing immediately, but will throw away packets until the capture start condition is met.

Using NMCAP to Extract a Conversation

NMCAP can also be used to process existing files, such as joining several captures into one file or splitting a large file into several smaller files.

Description Command
Extract a conversation NMCAP /inputcapture mytrace.cap /capture ipv4.address==10.10.10.10 AND tcp.port==1433 /file filteredoutput.cap
Extract a conversation in several files NMCAP /inputcapture mytrace.cap mytrace(1).cap mytrace(2).cap /capture ipv4.address==10.10.10.10 AND tcp.port==1433 /file filteredoutput.cap
Wildcards are not supported for /inputcapture, files must be specified individually.
Split a large file into smaller ones NMCAP /inputcapture mybigtrace.cap /capture /file mysmallfiles.chn:100M

PowerShell Script Using NMCAP to Extract From Multiple Files

The PowerShell script below automatically builds the NMCAP command above that extracts a conversation across multiple input files. This is especially useful if there are a large number of input files.

#------------------------------- 
# 
# Extract-NetConversation.ps1 
# 
# A PowerShell script to extract a conversation using NMCAP that is spread across multiple chained files.
#
# Requirements: NETMON must be installed to use the NMCAP tool.
#
# Arguments:
#
#    File Filter:    *.cap - avoid using this if you re-run the script as it will also gather the filtered files.
#                    Use something more specific, such as sqlprod01(*).cap or even sql*.cap
#
#    Capture Filter: Any valid NETMON filter. Surround with "quotes" if the filter string contains spaces.
#                    Use the client's IP address and port number for the most narrow filter.
#
#    Output File:    Any file name. You can incorporate the IP address and port number to distinguish from other filtered files.
#
#
# USAGE: .\Extract-NetConversation sql*.cap "IPV4.Address==10.10.10.10 AND tcp.port==1433" filtered.10.10.10.10.1433.cap
# 
#------------------------------- 
param ([string]$FileFilter = "*.cap", [string]$CaptureFilter = "", [string]$OutFileName = "filteredoutput.cap")
Set-ExecutionPolicy Unrestricted -Scope CurrentUser

# Get the list of files and concatenate the names into a string

$filestring = ""
$direntries = get-childitem -filter $FileFilter | sort-object -property LastWriteTime
foreach ($direntry in $direntries) { $filestring += "$($direntry.Name) " }

# Build and execute the command string - cannot just execute it directly because of the concatenation

$command = "NMCAP /inputcapture $filestring /capture $CaptureFilter /file $OutFileName"
$command | cmd       # Executes the command via cmd.exe

Using WireShark's EDITCAP.EXE to Split Large Trace Files

The EDITCAP command does not split on file size, but rather on frame count, so the files may come out a bit uneven in size. A frame count of 200,000 should keep the split files under 1GB in size. If using Jumbo Frames, then you may want to lower this number.

Note: Omit the -F pcap argument if splitting PCAPNG files. pcap must be in lower case and switches are case-sensitive.

"C:\Program Files\Wireshark\editcap.exe" -c 200000 -F pcap BigCapture.pcap SplitCapture.pcap

Using WireShark's TSHARK.EXE extract a conversation from Very Large Trace Files ( > 4GB)

Note: Omit the -F pcap argument if splitting PCAPNG files. pcap must be in lower case and switches are case-sensitive.
Note: The filter string must be quoted if it contains any spaces.

You should include some aspect the filter in the output file name to be able to distinguish between files.

"C:\Program Files\Wireshark\tshark.exe" -r sourcetracefile.pcap
                                        -Y "ip.addr==10.10.10.10 and tcp.port==50288"
                                        -w filteredoutput.10.10.10.10.50288.pcap -F pcap

Using TCPDUMP for Linux captures.

Where to get TCPDUMP if it is not installed.

Distribution Command
Debian, Ubuntu, Linux Mint sudo apt install tcpdump
CentOS, Fedora, AlmaLinux, Red Hat sudo dnf install tcpdump
Arch Linux, Manjaro sudo pacman -S tcpdump

TCPDUMP command options

Option Description Example
-i (Interface) Select the interface you want to trace -i eth0, -i any (for all interfaces)
-s #number Set packet truncation 0=no truncation, max truncation=65535
-C #number Sets the File size in MB -C 200
-W #number Sets the number if files -W 20
-w $path Sets the path and file name for trace -w /tmp/trace.pcap

Examples:

Consistent issue

tcpdump -i any -s 0 -w /tmp/MSTS/networktrace.pcap

Intermittent issue Issues that require a longer duration for the traces to be run without overwriting data: This command will create a chain trace of 50 files at 200MB.

tcpdump -i any -s 0 -C 200 -W 50 -w /tmp/MSTS/networktrace.pcap

For the full list of command option: https://www.tcpdump.org/manpages/tcpdump.1.html

⚠️ **GitHub.com Fallback** ⚠️