TEX (Traffic EXtractor) Documentation - opelbn/SPIF GitHub Wiki

Traffic EXtractor (TEX)

Whether you’re debugging network issues, analyzing protocol behavior, or hunting for anomalies, TEX turns raw PCAP data into actionable insights. This command-line tool extracts packet features, applies custom filters, and labels suspicious packets—perfect for network engineers, security analysts, and researchers working with protocols like Modbus or DNS.

Features

  • Extract standard and custom packet features (e.g., packet size, IP addresses, payload bytes).
  • Filter packets with BPF syntax, including payload-specific extraction.
  • Label packets for anomaly detection using JSON suspicion profiles.
  • Support for live capture and offline PCAP analysis.
  • Cross-platform: Windows (with Npcap) and POSIX systems (with libpcap).

Installation

Prerequisites

  • C++17 Compiler: GCC, Clang, or MSVC.
  • libpcap: Packet capture library (or Npcap on Windows).
  • nlohmann/json: JSON parsing library (single header file).
  • cxxopts: Command-line parsing library (single header file).

Install Dependencies

  • Ubuntu/Debian:
    sudo apt-get install libpcap-dev g++
  • Windows:
    • Install Npcap (ensure WinPcap compatibility mode is enabled).
    • Download json.hpp from GitHub.
    • Download cxxopts.hpp from GitHub.
  • Place json.hpp and cxxopts.hpp in your project directory.

Build

  • Linux/macOS:
    g++ -o TEX TEX.cpp -lpcap
  • Windows (MSVC):
    cl TEX.cpp -I. -link ws2_32.lib wpcap.lib
  • Note: Ensure Npcap’s wpcap.lib is in your library path on Windows.

Usage

Basic syntax:

TEX --in <pcap_file> --out <csv_file> [options]

Or for live capture:

TEX --live <interface> --out <csv_file> [options]

Command-Line Options

Input/Output

Option Description Default Example
-i, --in Input PCAP file N/A --in test.pcap
-o, --out Output CSV file N/A --out output.csv
--live Live capture from interface (name/index) N/A --live eth0

Feature Extraction

Option Description Default Example
-f, --features Comma-separated feature list None --features packetsize,srcip
--profile JSON profile (overrides -f, -b) None --profile modbus.json
--full_payload Output full payload (not 64-byte limit) False --full_payload

Filtering

Option Description Default Example
-b, --bpf Named BPF filter (e.g., name=filter) None --bpf tid=payload[0:2]
--bpf_pre_filter Pre-applied BPF filter None --bpf_pre_filter "tcp port 502"
--min_payload_len Minimum payload length 0 --min_payload_len 8
--payload_format Payload format (hex or raw) hex --payload_format raw

Miscellaneous

Option Description Default Example
--label JSON suspicion profile None --label suspicion.json
--reference_ip IP for direction (inbound/outbound) None --reference_ip 192.168.1.1
-v, --verbose Enable verbose logging False --verbose
--list-features List valid features and exit N/A --list-features
--list_interfaces List network interfaces and exit N/A --list_interfaces
--packet_limit Stop live capture after N packets 0 (unlimited) --packet_limit 1000
--time_limit Stop live capture after X seconds 0 (unlimited) --time_limit 60
-h, --help Show usage and exit N/A --help

Supported Features

Basic Features

  • packetsize: Total packet length in bytes (e.g., 66).
  • arrivalinterval: Time between packets in milliseconds (e.g., 1.234).
  • timestamp: Capture time in seconds with microsecond precision (e.g., 1612345678.123456).
  • payload: Packet payload (hex by default, e.g., 000100000006).

IP and Transport Features

  • protocol: IP protocol number (e.g., 6 for TCP, 17 for UDP).
  • srcip: Source IP address (e.g., 192.168.1.10).
  • dstip: Destination IP address (e.g., 10.0.0.10).
  • dstport: Destination port (e.g., 502) or 0 for non-TCP/UDP.
  • ip_ttl: Time-to-live value (e.g., 64).
  • ip_tos: Type of service value (e.g., 0).
  • tcp_window: TCP window size (e.g., 65535).
  • tcp_syn, tcp_ack, tcp_fin, tcp_rst, tcp_psh, tcp_urg: TCP flags (1 or 0).

Contextual Features

  • direction: Packet direction based on --reference_ip (inbound, outbound, unknown).
  • label: Suspicion label (0 or 1) from --label profile.

Custom Features

  • Defined via --bpf or profile["bpf_filters"] (e.g., modbus_tid for Modbus transaction ID).
  • Use --list-features to see all available features, including custom ones.

BPF Filters

BPF (Berkeley Packet Filter) filters let you target specific packet data, from entire protocols to individual bytes.

Pre-Filters (--bpf_pre_filter)

  • Limits which packets are processed.
  • Example: --bpf_pre_filter "tcp port 502" (only Modbus TCP packets).

Feature Filters (--bpf)

  • Extracts or compares data, adding named columns.
  • Syntax:
    • Standard BPF: tcp port 80 (filters packets).
    • Header: tcp[13]=2 (TCP SYN flag).
    • Payload: payload[0:2] (extracts 2 bytes from payload).
  • Endianness: Add little for little-endian (e.g., payload[0:2] little); default is big-endian.
  • Note: Use with --bpf_pre_filter to avoid processing irrelevant packets (e.g., tcp for payload[]).

Example

--bpf modbus_tid=payload[0:2] --bpf_pre_filter "tcp port 502"

JSON Profiles

JSON profiles configure features and filters, ideal for complex setups.

Structure

{
  "features": ["packetsize", "dstport", "modbus_tid"],
  "bpf_pre_filter": "tcp port 502",
  "bpf_filters": [
    {"name": "modbus_tid", "filter": "payload[0:2]", "endian": "big"}
  ]
}

Example

./TEX --in test.pcap --out output.csv --profile modbus.json

Suspicion Profiles

Suspicion profiles label packets (0 or 1) for anomaly detection.

Structure

{
  "threshold": 1,
  "rules": [
    {"feature": "dstport", "condition": "equals", "value": 502},
    {"feature": "modbus_tid", "condition": "greater_than", "value": 0}
  ]
}

Example

./TEX --in test.pcap --out output.csv --profile modbus.json --label suspicion.json

Examples

1. Basic Extraction

./TEX --in test.pcap --out output.csv --features packetsize,srcip

Output: output.csv

packetsize,srcip
66,192.168.1.10
54,10.0.0.10

2. Custom Payload Extraction

./TEX --in test.pcap --out output.csv --features packetsize,my_field --bpf my_field=payload[4:2] --bpf_pre_filter "tcp port 1234"

Output: output.csv

packetsize,my_field
70,256

3. Modbus Analysis with Labeling

./TEX --in test.pcap --out output.csv --profile modbus.json --label suspicion.json

Output: output.csv

packetsize,dstport,modbus_tid,label
66,502,1,1

4. Live Capture

./TEX --live eth0 --out output.csv --features packetsize,timestamp --time_limit 10

Output: Captures for 10 seconds.

Output Format

CSV with headers matching selected features:

packetsize,srcip,dstip,dstport,label
66,192.168.1.10,10.0.0.10,502,1

Troubleshooting

  • "Permission denied": Run as root/admin or check Npcap/libpcap installation.
  • "BPF compilation failed": Validate filter with tcpdump -d "filter".
  • No output: Ensure PCAP file is valid (tcpdump -r test.pcap).
  • Payload[] warning: Add --bpf_pre_filter (e.g., tcp) to scope packets.

Limitations

  • No real-time GUI; output is CSV only.
  • Payload truncated to 64 bytes unless --full_payload is used.
  • Live capture requires sufficient permissions.

Contributing

Feedback or contributions? Open an issue at [GitHub repo TBD] or email [[email protected]].

License

[Specify license, e.g., MIT, GPL, or TBD]

⚠️ **GitHub.com Fallback** ⚠️