MCP Across an Estate - NormB/sipnab GitHub Wiki

MCP deployment gets one sipnab answering one agent. This page is what comes after that:

• several SIP servers feeding one capture host,
• an agent that has to reach it from somewhere else,
• one agent holding several capture hosts at once,
• the hardest of them: following a single call as it crosses an SBC, a proxy and a PBX that each see a different Call-ID.

These four build on each other in that order. Each one names the SINGLE sipnab it starts from. So you can stop at whichever answers your question. Every scenario assumes you have already done Install sipnab.

Central capture host fed by HEP.

You usually can't (and shouldn't) run a packet-capturing debugger on every production SIP proxy.

Instead the proxies mirror signaling to one capture host via HEP. OpenSIPS, Kamailio, and FreeSWITCH all speak it. One sipnab MCP service then sees calls from the whole estate. The HEP listener is a plain UDP socket: no capture privileges, no setcap, fully unprivileged.

• [server] Do Install sipnab and create the user as in the create-the-unprivileged-user step of Keep a capture running between agent sessions, without the setcap.

• [server] Install /etc/systemd/system/sipnab-mcp.service.

  • This variant listens for HEP on udp/9063 and serves MCP on loopback.
  • Pair it with the SSH tunnel.
  • For the token shape instead, use its --mcp-bind/token/allowed-host lines:

  [Unit]
  Description=sipnab MCP server (HEP listener)
  After=network-online.target
  Wants=network-online.target
  
  [Service]
  Type=simple
  ExecStart=/usr/local/bin/sipnab --mcp -N --mcp-transport http \
      --mcp-bind 127.0.0.1:8731 \
      -L 0.0.0.0:9063 --hep-allow 192.0.2.0/24
  User=sipnab
  Group=sipnab
  NoNewPrivileges=true
  ProtectSystem=strict
  PrivateTmp=true
  Restart=on-failure
  RestartSec=5
  
  [Install]
  WantedBy=multi-user.target

  Replace 192.0.2.0/24 with the addresses your proxies send HEP from. sipnab refuses to start a HEP listener on a routable address unless it has one of these:

  • a source allowlist (--hep-allow),
  • a shared secret (--hep-auth-file).

  sudo systemctl daemon-reload && sudo systemctl enable --now sipnab-mcp

• [server] Open udp/9063 from the proxies' addresses in the host firewall.

• [proxy] Point each proxy's HEP mirror at the capture host.

  OpenSIPS (3.x) — opensips.cfg:

  loadmodule "proto_hep.so"
  loadmodule "tracer.so"
  modparam("tracer", "trace_id",
      "[sipnab]uri=hep:capture01.example.net:9063;version=3;transport=udp")
  

  and in the main route (traces the dialog's SIP both ways):

  trace("sipnab", "t", "sip");
  

  Kamailio: use the siptrace module with duplicate_uri pointed at sip:capture01.example.net:9063 and HEP mode enabled. See the siptrace docs for the handful of modparams.

• [server] Verify HEP is arriving. Place a test call through a proxy, then:

  curl -sS http://127.0.0.1:8731/mcp \
    -H "Content-Type: application/json" \
    -H "Accept: application/json, text/event-stream" \
    -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}'

  Then watch journalctl -u sipnab-mcp -f. The listener warns when it admits nothing for 30 seconds. The wording tells you which of two problems you have:

  • no packets for 30s means nothing arrived at all.
    • The HEP sender isn't reaching the -L port (firewall, wrong port, wrong host).
  • no packets admitted for 30s ... every one was refused means the sender IS reaching the port and sipnab is turning it away.
    • The line names the reason, such as auth_mismatch for a wrong shared secret.
    • It also names the peer it came from.
    • Fix the sender's key, --hep-auth-mode or the --hep-allow list rather than the network.

  --hep-silence-warn changes the 30 seconds.

  To see each proxy on its own, ask the MCP hep_senders tool. It lists:

  • every sender, with its packet count and whether it went silent,
  • every address the listener refused, with the reason.

• [laptop] Wire up exactly as the open-the-tunnel and register-against-localhost steps of Keep a capture running without exposing a port (or the copy-the-token and verify-the-client-connected steps of Keep a capture running between agent sessions for the token shape). Then ask across the estate: "search all proxies' traffic for Call-ID X and render the ladder."

A Homer / heplify-server box is a natural home for sipnab. The HEP plumbing from the proxies already exists there. Two things to arrange:

• Port: heplify-server owns udp/9060. So sipnab takes its own (udp/9063 above), and each proxy mirrors to both destinations.
  • OpenSIPS: add a second trace_id and a second trace() call.
  • Kamailio: add a second duplicate destination.
  • If a sender can't dup, put a small UDP fan-out (e.g. socat) in front.
• Budget: cap sipnab's footprint with [limits] (see Understand the load on a busy server). Then the box's primary tenant keeps its headroom.

Anything else on the host is simply a neighbor process. Examples are an OpenTelemetry collector, Prometheus, etc. sipnab neither speaks OTLP nor conflicts with it.

Internet-exposed endpoint with nginx TLS in front.

Use this when agents connect from outside your network and SSH isn't an option. Keep sipnab on loopback and let nginx own the public endpoint.

• [server] Build the loopback service with a token.

  • Use the first three steps of Keep a capture running between agent sessions.
  • Then add a unit whose ExecStart binds loopback but keeps auth and allows the public hostname. nginx forwards the client's Host: header:

  ExecStart=/usr/local/bin/sipnab --mcp -N --mcp-transport http \
      --mcp-bind 127.0.0.1:8731 \
      --mcp-token-file /etc/sipnab/mcp.token \
      --mcp-allowed-host capture.example.com \
      -d eth0

• [server] Install nginx and certbot:

  sudo apt install nginx certbot python3-certbot-nginx

  Then issue the certificate. certbot does these things:

  • it needs capture.example.com already resolving to this host,
  • it asks for a contact address,
  • it rewrites the nginx config in place.

  Run it on its own so you can answer it:

  sudo certbot --nginx -d capture.example.com

• [server] Site config (/etc/nginx/sites-available/sipnab-mcp, symlink into sites-enabled, sudo nginx -t && sudo systemctl reload nginx):

  server {
      listen 443 ssl;
      server_name capture.example.com;
      ssl_certificate     /etc/letsencrypt/live/capture.example.com/fullchain.pem;
      ssl_certificate_key /etc/letsencrypt/live/capture.example.com/privkey.pem;
  
      location /mcp {
          proxy_pass http://127.0.0.1:8731;
          proxy_set_header Host $host;
          proxy_buffering off;          # SSE responses must stream
          proxy_read_timeout 3600s;
      }
  }

  proxy_buffering off is load-bearing. The streamable-HTTP transport answers with text/event-stream, and buffering proxies stall it.

• [server] Firewall: open tcp/443 only. Port 8731 stays closed to the outside.

• [laptop] Copy the token as in the copy-the-token step of Keep a capture running between agent sessions, then:

  claude mcp add --transport http \
    --header "Authorization: Bearer $(cat ~/.config/sipnab/capture.token)" \
    sipnab-prod https://capture.example.com/mcp

• [laptop] Verify with claude mcp list. On failure, test the path layer by layer:

  • curl against https://capture.example.com/mcp from the laptop,
  • then the loopback curl from the verify-locally step of Keep a capture running between agent sessions on the server.

Run one of these on each capture host:

• the token service,
• the SSH tunnel,
• the nginx TLS endpoint.

Then give each its own entry in one client config:

• [each server] Any persistent wiring above (the token service shown here).

• [laptop] Register them all. The loop is one command and does nothing unless you take the whole of it:

  # Run all of these, in order.
  for h in nyc1 chi1 lax1; do
    claude mcp add --transport http \
      --header "Authorization: Bearer $(cat ~/.config/sipnab/$h.token)" \
      "sipnab-$h" "https://$h.example.net/mcp"
  done

  Then confirm all three registered:

  claude mcp list

• [laptop] Ask cross-fleet questions. Tool names are namespaced per server (mcp__sipnab-nyc1__list_dialogs, mcp__sipnab-chi1__list_dialogs, …). So the agent can fan out:

  the caller says the 14:02 UTC call dropped. Find its Call-ID on all three sites and compare the ladders.

Federated tracing. Each node keeps its own capture. The agent does the joining.

flowchart LR
    AG["agent or script (laptop)"]
    SBC["sipnab on the SBC<br/>node: sbc-edge-1"]
    PX["sipnab on the proxy<br/>node: proxy-1"]
    P1["sipnab on the PBX<br/>node: pbx-1"]

    AG -->|"1. find_correlated(access leg)"| SBC
    SBC -.->|"a Call-ID, and the strategy that tied it"| AG
    AG -->|"2. get_dialog(whatever the SBC named)"| PX
    AG -->|"3. and again, one hop further in"| P1

The previous section registers several servers. This one is about two things:

• the ORDER to ask them in,
• what the answer is worth.

Three hops means three MCP servers and one client.

• Nothing about the wiring is different from a single node.
  • You register each one.
  • The client namespaces the tools per server.
• What changes is only how the laptop reaches each server.

The laptop reaches all three directly. Nothing is between them. So use the SSH tunnel shape on each node.

• That is a loopback bind plus an SSH tunnel per node.
• It needs no token and opens no port.

Open the tunnels in this order:

• Open the tunnel to the SBC on local port 8811:

  ssh -f -N -L 8811:127.0.0.1:8731 sbc-edge-1.example.net

• Open the tunnel to the proxy on local port 8812:

  ssh -f -N -L 8812:127.0.0.1:8731 proxy-1.example.net

• Open the tunnel to the PBX on local port 8813:

  ssh -f -N -L 8813:127.0.0.1:8731 pbx-1.example.net

The local port differs per node. The remote port does not, because each node binds its own loopback. Then register the three, in this order:

• Register the SBC:

  claude mcp add --transport http sipnab-sbc   http://127.0.0.1:8811/mcp

• Register the proxy:

  claude mcp add --transport http sipnab-proxy http://127.0.0.1:8812/mcp

• Register the PBX:

  claude mcp add --transport http sipnab-pbx   http://127.0.0.1:8813/mcp

Give each node a --node-name that matches how you think of it.

• That string is what comes back in capture_identity.node.
• It is the only thing that attributes a fact to a box:

ExecStart=/usr/local/bin/sipnab --mcp -N --mcp-transport http \
    --mcp-bind 127.0.0.1:8731 --node-name sbc-edge-1 -d eth0

The nodes are behind NAT. The tunnel commands above still work.

• SSH dials outward from the laptop.
• The tunnel carries the MCP traffic back.
• So there is no inbound rule, no port forward, and no change to the MCP wiring.

One thing does not work against a node behind NAT: the token service (HTTP plus a token). There is no address to put in the URL.

SSH itself may not reach the node. Then the node has to reach out instead. That means the HEP shape: Collect captures from several SIP servers in one place. Then you have one server, not three.

A jump host sits in the way. Put the hop in your SSH config rather than in the sipnab wiring. Every command above then still works word for word:

Host sbc-edge-1.example.net proxy-1.example.net pbx-1.example.net
    ProxyJump bastion.example.net

For the SSH stdio shape, ssh -J bastion.example.net sbc-edge-1.example.net /usr/local/bin/sipnab --mcp -N … does the same thing inline.

Not measured. Everything in this subsection above the --node-name line is the same wiring that the SSH stdio shape, the token service and the SSH tunnel document, applied three times.

• The tunnel, NAT and ProxyJump commands were not run against three real hosts for this page.
• There was one machine available.

The behavior that was measured is everything below.

• The measurement used three sipnab servers on one host.
• Each had its own --node-name and its own capture.
• The transcripts say which build produced them.

The SBC is the only box that saw both sides of the call. So its find_correlated result names the core-side Call-ID to look up next.

• Querying the PBXes first means guessing which one took the call.
• It usually means asking all of them.

That ordering also matters for cost.

• Answering a query is cheap for the server.
• But every agent round trip spends seconds of model time.
• Following one pointer beats fanning out.

Ask it first even when you expect the box to stay a proxy on this call.

• If it did, find_correlated returns nothing. You carry the same Call-ID inward, which costs one query.
• If it did not, you now hold the identifier the next hop knows the call by.

Read what matched, because the topology is not fixed is about telling those two apart.

Here is the thing that makes this hard, and it is not a corner case: an SBC or proxy may run back-to-back on one call and stay a proxy on the next.

• Whether it re-originates depends on:
  • the call type,
  • which endpoints take part,
  • configuration that can change while you are watching.
• So you cannot pick a correlation strategy in advance.
• Any procedure that says "our SBC is a B2BUA, therefore look for X" is right until the day it quietly is not.

The way out is to stop predicting and start reading.

• You ask the same question every time: find_correlated on the leg you have.

• The answer tells you which topology you were in:

• A call that stayed in proxy mode keeps its Call-ID across the hop.

  • Nothing correlates, because there is nothing to correlate to.
  • The same identifier is simply present on the next node.
  • find_correlated returning zero legs is not a failure here. It is the finding.

• A call that went back-to-back gets a new Call-ID, and strategy names what tied the two together, or admits to a guess.

Every leg carries the strategy that matched it. The strategies are not degrees of one thing:

Listed in evaluation order, which is descending score.

• The loop stops at the first match, then sorts by score.
• So a leg that satisfies two strategies comes back as the stronger one.

Four fields decide how much of that tree you should act on:

sipnab omits timing_clock when every leg matched on an identifier. It does not send a healthy-looking default.

• A clock reading beside an identifier match invites you to weigh one against the other.
• They do not trade off.

Check the clock before believing a timing_heuristic match across nodes. The window is two seconds unless --leg-correlation-window says otherwise. The failure is silent in both directions:

• a fast clock misses legs that belong together,
• a slow one pulls unrelated calls in.

timing_clock reports the answering node's NTP discipline at the moment of the query (synchronized, max_error_us, est_error_us, available). capture_health reports the same under clock, for any node you want to check without running a correlation.

• synchronized: false means treat the tree as a hypothesis.
• synchronized: true with a max_error_us approaching the two-second window means the same thing.
  • The flag says a time daemon is disciplining the clock.
  • It does not say the clock is accurate to within the window you are matching in.

Both transcripts below are real output from sipnab 0.5.97.

• They come from the script in Drive it from a script.
• They ran against sipnab servers reading tests/pcap-samples/.

Same command shape, opposite evidence.

The hop stayed a proxy. Zero legs, and the Call-ID turns up unchanged one node in. That is what proxy mode looks like, not what a lost call looks like:

[proxy] sipnab 0.5.97 node=proxy-1
[pbx] sipnab 0.5.97 node=pbx-2
[proxy] 0 leg(s) correlated to [email protected]
  (nothing correlated: a call that stayed in proxy mode keeps its
   Call-ID, so ask the other nodes for the SAME id.)
[pbx] holds [email protected]  (node=pbx-2, state=Completed, 11 msgs)

The hop went back-to-back. A leg comes back. The strategy names a guess from a 3 ms gap and a shared endpoint. No identifier crossed the box:

[sbc] sipnab 0.5.97 node=sbc-edge-1
[sbc] 1 leg(s) correlated to [email protected]:5060
  [email protected]
      via timing_heuristic [GUESS] score 50, gap 3ms
  !! every leg was a timing guess, not an identifier match.
     clock on sbc: synchronized=True max_error_us=295000
     The window is 2s. Skew larger than that invents legs and hides legs.

Read the second one carefully. It is the case operators act on wrongly.

• A leg came back. It has a Call-ID, a score, and a plausible-looking 3 ms gap.
• It is still a guess: identifier_match is false and heuristic_only is true.
• Two unrelated calls through the same SBC inside the same window produce output that looks exactly like this.

Now read max_error_us, and do not read it once. It is a live reading, not a constant.

• On this one host it has reported 0.295 s (the run above), 1.944 s and 2.38 s.

• That is an order of magnitude apart.

• All three came while saying synchronized=True.

• The last of them is past the 2 s correlation window entirely.

• At the high end the clock could account for the entire match on its own.

• At the low end it could not.

• Nothing in the output tells you which run you are looking at except the number itself. So read yours each time.

• A figure quoted from another run says nothing about your box. That includes the ones on this page.

Take a box that emits no Session-ID and no X-Call-ID, and re-originates SDP. Then nothing in the signaling proves the two legs are one call.

• The honest answer is that they may be.
• sipnab says so rather than drawing a tree on a timing guess.
• Configuring the box to insert a correlation identifier is the fix. Choose a correlation identifier covers the options.
• sipnab watches a wire and cannot add one. The SBC forwards its own message to the far side regardless of what sipnab saw.

Attribution needs one more step than you might expect. Not every response carries capture_identity. Read the table rather than a rule of thumb.

• The rule "whole-store answers carry it, per-dialog ones do not" is close but wrong.
• media_diagnostics is per-dialog and carries it:

So call capture_status once per node and hold the name. Do not expect every answer to carry it.

It matters.

• "answered 407" is incomplete until you know which box answered.
• With three servers registered, the agent has three places that sentence could have come from.
• Per-message answers carry a frame pointer instead (tests/pcap-samples/sip-proxy.pcap#0@a57665bcdb62f03a).
  • It names the capture the message came out of rather than the node.

The transcript above ends in a guess. The durable fix is not a better guess. It is an identifier that survives the hop. Three options get raised, and they are not equally good.

1. Configure the SBC, proxy and PBX to insert one. Do this. RFC 7989 Session-ID exists for exactly this problem.

• It is a pair of UUIDs, one contributed by each endpoint.
• Each side reports the pair from its own point of view.
• So it survives a box that rewrites Call-ID, From tag and Via.

sipnab already reads it.

• src/sip/session_id.rs parses the header.
• It intersects the non-nil halves rather than comparing strings.
  • The halves swap direction across a B2BUA.
  • So string equality would find nothing and look exactly like "unrelated calls".
• Correlation on it reports strategy: session_id with identifier_match: true.

Nothing on the sipnab side needs changing. The work is one config line per box. It converts every future trace from a guess into evidence.

2. Use an identifier the network already carries. This is what x_call_id, sdp_origin and the two P-Charging-Vector strategies are. sipnab is already looking for them. You get an identifier match today with no configuration at all if any of these is true:

• your SBC emits X-Call-ID by vendor convention,
• it forwards SDP untouched so the RFC 8866 origin tuple survives,
• it sits in a carrier network where RFC 7315 charging headers are on the wire anyway.

Check before you plan work. Run find_correlated on a known B2BUA call and see what strategy comes back.

Three caveats worth knowing:

• sdp_origin identifies the media session rather than the dialog. So it goes away the moment anything re-originates SDP.
• via_branch is an identifier match, but it never crosses a B2BUA. A back-to-back user agent opens a new transaction by definition.
• charging_vector_icid is the weaker of the two charging strategies. The reason is the one the note below gives.
  • A conformant B2BUA gives each of its two dialogs its own icid.
  • So plain equality across one is a vendor behavior rather than something the RFC promises.

3. Have sipnab compute its own identical id on each node. Do not. The appeal is obvious: no config change on any SIP box. But it is the wrong trade.

• sipnab is a passive wire observer. It cannot inject a header.
• So "the same id on both nodes" would have to be computed from what each node independently sees.

Across a re-originating B2BUA there is no guaranteed invariant to compute it from.

• Call-ID, From tag, Via branch, Contact and usually the SDP are all legitimately new on the far side.
• That is what re-origination means. It is not a defect to work around.

Any id derived from the remainder is a heuristic dressed as an identifier. That is worse than the labeled heuristic already in the output.

• timing_heuristic announces itself as a guess and sets heuristic_only.
• A computed id would arrive looking like proof.
• It would correlate two unrelated calls with no field left to catch it.

That is the same judgement this codebase makes elsewhere.

• sipnab records the RFC 7329 legacy Session-ID form as an interop notice, not a violation.
  • A single message cannot distinguish a legacy implementation from a broken one.
  • So the finding states what arrived on the wire and declines to assert which.
• A computed cross-node id would be the opposite move. It would assert an identity the wire never established.

For IMS and carrier readers: sipnab reads P-Charging-Vector in two strategies, and the difference between them decides whether it helps you.

• RFC 7315 section 4.6 says the ICID identifies a dialog.
• A B2BUA is two dialogs.
• So a conformant B2BUA emits a different icid-value on each side.
• So plain icid-value equality does not solve the re-origination case.

What crosses that hop is the separate related-icid parameter (RFC 7315 section 4.6.4.1).

• Its value is the icid of the original dialog.
• The B2BUA MAY emit it rather than must.

So:

• charging_vector_related_icid (95): the intermediary declared the link. This is the one that crosses a B2BUA, and only when the box chose to send it.
• charging_vector_icid (85): the two legs carry the same icid-value. Useful where it happens, and it means some intermediary copied a per-dialog identifier onto a second dialog. No RFC grants that.

Two further limits, both from the RFC rather than from the implementation:

• The first proxy generates the icid (RFC 7315 section 5.6).
  • So the leg arriving from an endpoint carries none.
  • So this is useless at the access edge.
• RFC 7315 section 4.6.2.2 permits the next hop to "modify the contents".
  • RFC 7315 section 6.6 calls that normal behavior.
  • There is no end-to-end constancy requirement of any kind.
  • So this is not a substitute for Session-ID.

Full argument, including what is still unverified: docs/design/icid-correlation.md.

Both work. The choice is about where packet data lives rather than which is newer.

Centralizing needs no new code. See Collect captures from several SIP servers in one place. It is also what Homer does, at the scale of a whole enterprise system. sipnab is one binary. It can feed Homer rather than replace it.