icassp2027-experiment/results/easycom_egospeak_three_level_eval_2026-09-16/README.md

EasyCom EgoSpeak three-level evaluation

Protocol erratum. The dense and raw-boundary results in this record remain valid. The opportunity-conditioned ENTRY/INT values were produced by the superseded v1 loader, which could cross an intervening ENTRY competitor and selected the latest onset in ambiguous multi-competitor INT cases. Do not cite those opportunity values; rerun protocol v2 and create a new result record.

This record freezes the first Sessions 1--3 evaluation of the released EgoSpeak MiniROADA audio-visual checkpoint under three deliberately separate questions. The values below were produced on the remote experiment host; the aggregate console output was copied into this record on 2026-09-16. Generated prediction and per-event artifacts remain on that host and are not committed.

Protocol and split

• Test split: EasyCom Sessions 1--3.
• Model: released EgoSpeak MiniROADA audio-visual checkpoint.
• Dense forecasting: framewise AP for background, wearer, and aggregate others at ten horizons from 0.2 to 2.0 seconds.
• Raw boundary anticipation: threshold-free AP for wearer OFF->ON and ON->OFF changes. These are timing-derived diagnostics, not semantic EOT or INT claims.
• Opportunity-conditioned behavior: historical v1 activity-derived score per reviewed ENTRY or INT/YIELD opportunity. Labels are provisional two-pass LLM-assisted reviews and have not been human reviewed.
• No test-set threshold is fitted for any AP reported here. Session 4 is needed only for threshold-dependent operating points, which are intentionally not summarized as headline findings in this record.

Findings

Dense forecasting reaches 74.124% mAP at 0.2 s and 61.155% mean mAP over ten horizons. The wearer class alone reaches 63.750% AP at 0.2 s and 53.493% averaged over horizons. Restricting evaluation to causally valid rows changes mean mAP by only +0.038 percentage points (61.193% versus 61.155%).

Raw wearer onset and offset anticipation obtain 24.491% and 38.024% mean AP, respectively. These lower values do not contradict the dense result: dense AP rewards correct future state over sustained regions, whereas a raw boundary event is counted once and has many harder non-boundary comparisons.

The superseded v1 run reported ENTRY AP of 52.487% for 163 positives and 335 hard negatives, versus 32.731% positive prevalence (+19.756 points; 1.60x prevalence). INT/YIELD AP is 20.135% for 28 positives and 109 hard negatives, essentially equal to its 20.438% prevalence. These values are retained only for provenance and must be replaced by a v2 rerun. The original pattern was consistent with the output contract: wearer probability can rank many entry opportunities, but 1-p(wearer) does not encode whether an interruption should cause a yield. The ENTRY and INT loaders additionally exclude 81 and 5 invalid/post-outcome cases; these exclusions require an audit before any final paper claim.

Relation to EgoSpeak Figure 7

Figure 7 of EgoSpeak plots a continuous target/wearer-speaking confidence over ground-truth speaking regions. It therefore corresponds most directly to the dense wearer AP, not to raw onset/offset AP or semantic ENTRY/YIELD AP. A visually convincing dense curve can coexist with weak boundary or semantic selectivity. The figure caption does not identify one forecast horizon, and it uses the paper's Transformer variants rather than proving the aggregate result of this MiniROADA run.

For our eventual qualitative figure, the continuous wearer curve can be kept, but it should also show the causal opportunity anchor, event boundary, competing-risk censor, and the single score assigned to each positive or hard negative.

Interpretation limits

• The three metric layers answer different questions and their similar numeric values must not be equated.
• Aggregate others activity cannot identify a particular speaker, addressee, semantic EOT, or interruption intent.
• The opportunity result is provisional until label/exclusion audit and human review; it is useful for baseline diagnosis, not a final confirmatory claim.
• This is one released checkpoint, not a multi-seed uncertainty estimate.

Machine-readable curves and counts are in summary.json and the compact comparison is in metrics.csv. The opportunity protocol and reproduction command are documented in docs/easycom_opportunity_event_evaluation.md.