Evaluation and Results

Synthetic data, controlled single-factor flaws. These numbers show pipeline behaviour, not performance on human speech.

Method

• Predicted regions are matched one-to-one to ground-truth spans of the same dimension by IoU (greedy, highest IoU first).
• Reported: recall at IoU 0.3 and 0.5, mean IoU of matches, mean absolute start/end error, false-positive regions (unmatched predictions), flags on near-perfect mirrors, and whether the dimension score falls as severity rises.
• Label tolerance: exact by construction (not human-reviewed).
• Split: gettysburg is the dev speech (thresholds were set a priori and adjusted once after inspecting it). inaugural2 and fdr1933 are held-out and were not used for tuning.
• Reproduce with make evaluate (outputs in results/).

Held-out (2 speeches, 42 items)

Dimension	GT spans	[email protected]	[email protected]	Mean IoU	Start err (s)	End err (s)
Pacing	14	0.857	0.857	0.919	0.174	0.113
Pauses	20	1.000	1.000	0.952	0.012	0.012
Pitch contour	8	0.875	0.875	0.663	0.808	0.645
Energy	6	1.000	0.833	0.886	0.284	0.239

False-positive regions: 4. Flags on 4 near-perfect mirrors: 0. Severity ordering: 12/12 series.

Dev (1 speech, 21 items)

Dimension	GT spans	[email protected]	[email protected]	Mean IoU	Start err (s)	End err (s)
Pacing	7	1.000	0.857	0.851	0.337	0.317
Pauses	10	1.000	1.000	0.951	0.014	0.011
Pitch contour	4	0.750	0.750	0.644	0.976	0.768
Energy	3	1.000	1.000	0.992	0.023	0.013

False-positive regions: 1. Near-perfect flags: 0 of 2 items. Severity ordering: 6/6.

All 63 items

Dimension	GT spans	[email protected]	[email protected]	Mean IoU
Pacing	21	0.905	0.857	0.894
Pauses	30	1.000	1.000	0.951
Pitch contour	12	0.833	0.833	0.657
Energy	9	1.000	0.889	0.921

False-positive regions: 5. Near-perfect flags: 0 of 6 items. Severity ordering: 18/18.

Reading the results

• Pause and energy flaws are located almost exactly. Pause timing comes from silence detection, which is clean in this data.
• Pitch boundaries are loose (about 0.7 s), because the detector uses a 5-word window that smears edges. The mildest pitch flattening (60% compression) was missed in the dev speech.
• Pacing misses and boundary error depend on alignment accuracy inside chunks.
• The noise-free, single-factor setup flatters every number. Real recordings will be harder.

Raw numbers: results/evaluation.json, per-item output: results/per_item_results.json.