Expectations - NormB/sipnab GitHub Wiki

A change to a dialplan, a proxy script or a trunk can break calls that nobody places until a customer does. An expectation suite states what a test call capture must show: how many calls, what share answered, how many conformance errors, how good the audio was. sipnab --expect judges a capture against it and exits non-zero when a rule fails, so the step that runs it fails the build.

What you need

  • A sipnab build with the default features. Judging a capture file needs no privileges.
  • A capture of the traffic you want to judge. In a pipeline that is usually a test run: SIPp or a test phone places calls through a staging server while sipnab or tcpdump records them.
  • A suite file in your repository, sipnab.expect.toml by convention.
  • No change on any other system. The suite reads the capture only, so nothing on your SIP servers changes.

Write a suite

A suite is a list of rules. Each rule measures one figure over the capture's calls and compares it with a threshold:

[[rules]]
name = "calls captured"
metric = "count"
op = ">="
value = 5

[[rules]]
name = "answer-seizure ratio"
metric = "asr"
op = ">="
value = 50

[[rules]]
name = "no busy answers"
metric = "count"
op = "=="
value = 0
scope = "filter:response_code == 486"

This is tests/fixtures/expect/sip-problem-call.expect.toml in a source checkout.

Key Required What it takes
metric yes count: the calls in scope. asr: the percentage of INVITEs with a final response that the far end answered. lint_errors: the conformance findings at or above a severity. mos_p<N>: the Nth percentile of the streams' MOS, from mos_p0 (the worst stream) to mos_p100 (the best)
op yes >=, >, <=, <, == or !=
value yes The threshold, a number
name no A label for the output. Without it, the output names the rule by its position, such as #1
scope no filter:<expression> narrows the rule to the calls a filter selects; an alias such as filter:problems works too. severity:<info, notice, warning or error> sets the lowest severity a lint_errors rule counts. Without it, the rule reads the whole capture
min_sample no The fewest calls (or streams, for MOS) the rule needs. Below it, the rule is skipped instead of judged
grounded_only no MOS rules only. true (the default) reads only streams whose codec has a published impairment value. false admits the others, and the result says so

An asr or MOS rule that measured nothing fails: an asr rule whose scope holds no INVITE with a final response, or a MOS rule over a capture with no RTP, does not pass by default. Give it min_sample when an empty population is acceptable. A count rule always measures something: a scope that selects no call counts 0, so count == 0 passes.

Run it

sipnab -I tests/pcap-samples/sip-problem-call.pcap --expect tests/fixtures/expect/sip-problem-call.expect.toml --no-cli-print --quiet
Expectations: fail, 1 of 3 rules passed, 2 failed
Rule                  Metric  Op  Threshold  Observed  Sample  Verdict
----------------------------------------------------------------------
calls captured        count   >=  5          5         5       pass
answer-seizure ratio  asr     >=  50         20        5       fail
no busy answers       count   ==  0          1         5       fail
answer-seizure ratio: 20 is NOT >= 50 over 5 seizure(s)
no busy answers: 1 is NOT == 0 over 5 dialog(s)

The first line is the suite's verdict. Each rule's row gives what it observed and over how many calls, and each failed rule repeats on a line of its own.

The run exits with the verdict:

Exit Meaning
0 Every rule that ran passed
3 At least one rule failed
2 No rule ran, because each one's population was below its min_sample. Or sipnab could not read the suite, or the suite holds a rule sipnab refuses
1 sipnab could not read the capture

sipnab reads and checks the suite before it reads the capture, so a mistake in the file stops the run at once.

Check that it worked

Print the exit status after the run. The sample suite has two failing rules, so the status is 3:

sipnab -I tests/pcap-samples/sip-problem-call.pcap --expect tests/fixtures/expect/sip-problem-call.expect.toml --no-cli-print --quiet > /dev/null; echo $?
3

A status of 3 with the Expectations: fail line means the suite ran and judged the capture. A status of 2 or 1 means it did not judge it: When it does not work lists the causes.

The rules judge every call in the capture. --filter and --since do not narrow them. Narrow a rule with its own scope.

Run it in a pipeline

Run the suite as its own step, after the step that records the capture. The step fails when sipnab exits non-zero:

sipnab -I test-calls.pcap --expect sipnab.expect.toml --no-cli-print --quiet

To keep a machine-readable record of the verdict, write the JSON report as well. exit_code is the run's exit status:

sipnab -I tests/pcap-samples/sip-problem-call.pcap --expect tests/fixtures/expect/sip-problem-call.expect.toml --json --no-cli-print --quiet | jq -c '{verdict, exit_code, failed: [.results[] | select(.verdict == "fail") | .name]}'
{"verdict":"fail","exit_code":3,"failed":["answer-seizure ratio","no busy answers"]}

A pipe hands the shell jq's exit status, not sipnab's. In a script, read exit_code from the JSON, or use set -o pipefail.

Check conformance in the same suite

A lint_errors rule counts the conformance linter's findings, so one suite can fail a build on non-conformant SIP as well as on failed calls. The scope sets the lowest severity counted:

[[rules]]
name = "no conformance errors"
metric = "lint_errors"
op = "=="
value = 0
scope = "severity:error"

With that rule in lint.expect.toml, against the capture that ships with sipnab to show what the linter finds:

sipnab -I tests/pcap-samples/sip-lint-findings.pcap --expect lint.expect.toml --no-cli-print --quiet
Expectations: fail, 0 of 1 rule passed, 1 failed
Rule                   Metric       Op  Threshold  Observed  Sample  Verdict
----------------------------------------------------------------------------
no conformance errors  lint_errors  ==  0          2         3       fail
no conformance errors: 2 is NOT == 0 over 3 dialog(s)

The rule honors the same suppressions as --lint: a .sipnablint file beside the capture, or the file --lint-suppress-file names. See Suppress a rule.

Ask from an agent

The MCP tool evaluate_expectations judges the capture the server holds. It takes the rules inline as JSON, or a suite file's text in rules_toml, and returns the same report. See evaluate_expectations.

When it does not work

A metric sipnab does not know stops the run before it reads the capture, with exit 2. The message names the suite file you gave, here sipnab.expect.toml, and lists the metrics sipnab takes:

error: invalid value 'sipnab.expect.toml' for '--expect <FILE>': sipnab.expect.toml: rule 0: unknown metric 'asr2'. Known metrics: count, asr, lint_errors, mos_p<N>. A percentile is spelled mos_p10, mos_p50, mos_p90 and so on, from mos_p0 (worst stream) to mos_p100 (best)

A rule that measured nothing fails, and says why. Here a MOS rule over sip-problem-call.pcap, which holds no RTP:

Expectations: fail, 0 of 1 rule passed, 1 failed
Rule  Metric   Op  Threshold  Observed  Sample  Verdict
-------------------------------------------------------
#1    mos_p10  >=  3.50       -         0       fail
#1: unevaluable: no scored stream was in scope, so 'mos_p10 >= 3.5' rests on nothing. Declare a min_sample of 1 or more to accept an empty population in writing; a gate does not pass on data it never judged

With min_sample = 1 sipnab skips the same rule instead, and a suite in which sipnab skipped every rule exits 2:

Expectations: not evaluated, 0 of 0 rules passed, 1 skipped
Rule  Metric   Op  Threshold  Observed  Sample  Verdict
-------------------------------------------------------
#1    mos_p10  >=  3.50       -         0       skipped
#1: skipped: 0 scored stream(s) is below the declared min_sample of 1
What you see Why What to do
Exit 2 before sipnab reads the capture, naming a rule The suite has a metric, operator, scope or key sipnab does not take, or a filter that does not parse Fix the rule the message names
Exit 2 and not evaluated Every rule was below its min_sample Capture more calls, or lower min_sample
A rule fails with unevaluable and Sample 0 An asr or MOS rule measured nothing: its scope held no INVITE with a final response, or no stream Check the filter with --report --filter. Give min_sample if an empty capture is acceptable
A MOS rule fails on a capture with audio grounded_only excluded streams whose codec has no published impairment value See MOS and codecs, or set grounded_only = false and read the caveat
The step passes although a rule failed A pipe returned the exit status of the last command Run sipnab on its own, or use set -o pipefail
⚠️ **GitHub.com Fallback** ⚠️