TEX (Traffic EXtractor) Documentation - opelbn/SPIF GitHub Wiki
Whether you’re debugging network issues, analyzing protocol behavior, or hunting for anomalies, TEX
turns raw PCAP data into actionable insights. This command-line tool extracts packet features, applies custom filters, and labels suspicious packets—perfect for network engineers, security analysts, and researchers working with protocols like Modbus or DNS.
- Extract standard and custom packet features (e.g., packet size, IP addresses, payload bytes).
- Filter packets with BPF syntax, including payload-specific extraction.
- Label packets for anomaly detection using JSON suspicion profiles.
- Support for live capture and offline PCAP analysis.
- Cross-platform: Windows (with Npcap) and POSIX systems (with libpcap).
- C++17 Compiler: GCC, Clang, or MSVC.
- libpcap: Packet capture library (or Npcap on Windows).
- nlohmann/json: JSON parsing library (single header file).
- cxxopts: Command-line parsing library (single header file).
-
Ubuntu/Debian:
sudo apt-get install libpcap-dev g++
- Windows:
- Place
json.hpp
andcxxopts.hpp
in your project directory.
-
Linux/macOS:
g++ -o TEX TEX.cpp -lpcap
-
Windows (MSVC):
cl TEX.cpp -I. -link ws2_32.lib wpcap.lib
-
Note: Ensure Npcap’s
wpcap.lib
is in your library path on Windows.
Basic syntax:
TEX --in <pcap_file> --out <csv_file> [options]
Or for live capture:
TEX --live <interface> --out <csv_file> [options]
Option | Description | Default | Example |
---|---|---|---|
-i, --in |
Input PCAP file | N/A | --in test.pcap |
-o, --out |
Output CSV file | N/A | --out output.csv |
--live |
Live capture from interface (name/index) | N/A | --live eth0 |
Option | Description | Default | Example |
---|---|---|---|
-f, --features |
Comma-separated feature list | None | --features packetsize,srcip |
--profile |
JSON profile (overrides -f , -b ) |
None | --profile modbus.json |
--full_payload |
Output full payload (not 64-byte limit) | False | --full_payload |
Option | Description | Default | Example |
---|---|---|---|
-b, --bpf |
Named BPF filter (e.g., name=filter ) |
None | --bpf tid=payload[0:2] |
--bpf_pre_filter |
Pre-applied BPF filter | None | --bpf_pre_filter "tcp port 502" |
--min_payload_len |
Minimum payload length | 0 | --min_payload_len 8 |
--payload_format |
Payload format (hex or raw ) |
hex |
--payload_format raw |
Option | Description | Default | Example |
---|---|---|---|
--label |
JSON suspicion profile | None | --label suspicion.json |
--reference_ip |
IP for direction (inbound /outbound ) |
None | --reference_ip 192.168.1.1 |
-v, --verbose |
Enable verbose logging | False | --verbose |
--list-features |
List valid features and exit | N/A | --list-features |
--list_interfaces |
List network interfaces and exit | N/A | --list_interfaces |
--packet_limit |
Stop live capture after N packets | 0 (unlimited) | --packet_limit 1000 |
--time_limit |
Stop live capture after X seconds | 0 (unlimited) | --time_limit 60 |
-h, --help |
Show usage and exit | N/A | --help |
-
packetsize
: Total packet length in bytes (e.g., 66). -
arrivalinterval
: Time between packets in milliseconds (e.g., 1.234). -
timestamp
: Capture time in seconds with microsecond precision (e.g., 1612345678.123456). -
payload
: Packet payload (hex by default, e.g.,000100000006
).
-
protocol
: IP protocol number (e.g., 6 for TCP, 17 for UDP). -
srcip
: Source IP address (e.g.,192.168.1.10
). -
dstip
: Destination IP address (e.g.,10.0.0.10
). -
dstport
: Destination port (e.g., 502) or 0 for non-TCP/UDP. -
ip_ttl
: Time-to-live value (e.g., 64). -
ip_tos
: Type of service value (e.g., 0). -
tcp_window
: TCP window size (e.g., 65535). -
tcp_syn
,tcp_ack
,tcp_fin
,tcp_rst
,tcp_psh
,tcp_urg
: TCP flags (1 or 0).
-
direction
: Packet direction based on--reference_ip
(inbound
,outbound
,unknown
). -
label
: Suspicion label (0 or 1) from--label
profile.
- Defined via
--bpf
orprofile["bpf_filters"]
(e.g.,modbus_tid
for Modbus transaction ID). - Use
--list-features
to see all available features, including custom ones.
BPF (Berkeley Packet Filter) filters let you target specific packet data, from entire protocols to individual bytes.
- Limits which packets are processed.
- Example:
--bpf_pre_filter "tcp port 502"
(only Modbus TCP packets).
- Extracts or compares data, adding named columns.
-
Syntax:
- Standard BPF:
tcp port 80
(filters packets). - Header:
tcp[13]=2
(TCP SYN flag). - Payload:
payload[0:2]
(extracts 2 bytes from payload).
- Standard BPF:
-
Endianness: Add
little
for little-endian (e.g.,payload[0:2] little
); default is big-endian. -
Note: Use with
--bpf_pre_filter
to avoid processing irrelevant packets (e.g.,tcp
forpayload[]
).
--bpf modbus_tid=payload[0:2] --bpf_pre_filter "tcp port 502"
JSON profiles configure features and filters, ideal for complex setups.
{
"features": ["packetsize", "dstport", "modbus_tid"],
"bpf_pre_filter": "tcp port 502",
"bpf_filters": [
{"name": "modbus_tid", "filter": "payload[0:2]", "endian": "big"}
]
}
./TEX --in test.pcap --out output.csv --profile modbus.json
Suspicion profiles label packets (0 or 1) for anomaly detection.
{
"threshold": 1,
"rules": [
{"feature": "dstport", "condition": "equals", "value": 502},
{"feature": "modbus_tid", "condition": "greater_than", "value": 0}
]
}
./TEX --in test.pcap --out output.csv --profile modbus.json --label suspicion.json
./TEX --in test.pcap --out output.csv --features packetsize,srcip
Output: output.csv
packetsize,srcip
66,192.168.1.10
54,10.0.0.10
./TEX --in test.pcap --out output.csv --features packetsize,my_field --bpf my_field=payload[4:2] --bpf_pre_filter "tcp port 1234"
Output: output.csv
packetsize,my_field
70,256
./TEX --in test.pcap --out output.csv --profile modbus.json --label suspicion.json
Output: output.csv
packetsize,dstport,modbus_tid,label
66,502,1,1
./TEX --live eth0 --out output.csv --features packetsize,timestamp --time_limit 10
Output: Captures for 10 seconds.
CSV with headers matching selected features:
packetsize,srcip,dstip,dstport,label
66,192.168.1.10,10.0.0.10,502,1
- "Permission denied": Run as root/admin or check Npcap/libpcap installation.
-
"BPF compilation failed": Validate filter with
tcpdump -d "filter"
. -
No output: Ensure PCAP file is valid (
tcpdump -r test.pcap
). -
Payload[] warning: Add
--bpf_pre_filter
(e.g.,tcp
) to scope packets.
- No real-time GUI; output is CSV only.
- Payload truncated to 64 bytes unless
--full_payload
is used. - Live capture requires sufficient permissions.
Feedback or contributions? Open an issue at [GitHub repo TBD] or email [[email protected]].
[Specify license, e.g., MIT, GPL, or TBD]