icassp2027-experiment/docs/easycom_opportunity_event_evaluation.md

EasyCom opportunity-conditioned ENTRY/YIELD evaluation

evaluate_egospeak_opportunity_events.py uses LLM-assisted or human-reviewed semantic labels to define matched conversational opportunities while scoring every model only from its original EgoSpeak-compatible wearer probabilities. It does not train a semantic event head.

ENTRY

A positive must have a reviewer-selected preceding non-wearer release. Its opportunity starts at that release and resolves at the reviewed wearer entry. If a new non-wearer onset occurs before or exactly at that entry, the release has already been resolved by a competing event and the positive is excluded. The same applies when wearer activity already occurred between the selected release and the reviewed onset: the selected release is then a stale, post-outcome anchor. Both cases are counted separately in review_diagnostics.skipped. Core hard negatives start at the same type of non-wearer release and resolve when the same speaker resumes (ENTRY_OTHER_MIDTURN_PAUSE) or a different non-wearer takes the next floor (ENTRY_OTHER_TO_OTHER). The score is the maximum projected wearer probability up to the competing resolution, capped at 2.0 seconds.

ENTRY_WEARER_BACKCHANNEL is excluded from headline ENTRY FPR. Predicting wearer activity for a real backchannel is correct under EgoSpeak's dense target; ENTRY-versus-backchannel remains a separate semantic-selectivity diagnostic.

YIELD

An opportunity starts at a non-wearer onset during wearer speech. The score is the maximum 1 - p_wearer over 0.2--0.6 seconds, censored strictly at that non-wearer's offset. Reviewed INT_YIELD items are positive; other backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.

Current INT reviews do not select one competing onset. A unique supplied other interval that crosses the wearer offset within the annotation tolerance is used. Cases with multiple eligible intervals are excluded and counted rather than being assigned a deterministic latest-onset anchor.

Provisional EgoSpeak run

Threshold-free AP on the official test sessions can be run without fitting a test threshold:

python3 evaluate_egospeak_opportunity_events.py \
  --reviews reports/easycom_entry_int_sessions1_4_v16/easycom_eot_int_provisional_llm_reviews.jsonl \
  --run egospeak=results/egospeak_miniroada_floor_eval \
  --sessions Session_1 Session_2 Session_3 \
  --allow-provisional-llm-reviews \
  --print-event-examples 10 \
  --output results/easycom_egospeak_opportunity_events/sessions1_3_provisional_summary.json

The output JSON records label coverage and every exclusion reason. The metrics CSV contains threshold-free ENTRY/YIELD AP. The event CSV contains anchors, causal source cutoffs, censor times, horizons, and wearer probabilities for timeline auditing.

Do not request matched-recall operating points on Sessions 1--3 without disjoint Sessions 4--12 calibration reviews and out-of-fold predictions. Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD cap, positive anchor-selection policy, and multiple-competitor handling using training-side data only.

Superseded v1 Sessions 1--3 baseline

The first threshold-free run used the superseded v1 loader and reported ENTRY AP 52.487% (163 positive, 335 hard negative; 32.731% prevalence) and INT/YIELD AP 20.135% (28 positive, 109 hard negative; 20.438% prevalence). It excluded 81 ENTRY and 5 INT cases as invalid or post-outcome. These opportunity values must not be reported: v1 allowed a selected ENTRY release to cross an intervening other onset and selected the latest onset in ambiguous multi-competitor INT cases. Dense and raw-boundary metrics are unaffected. Rerun the opportunity evaluation with protocol v2 before updating the manuscript. The historical output is retained in the 2026-09-16 result record.

The expanded v1 Omni screen found no opportunity-ranking gain from the current F1 Frame-set Attention model: its three-seed ENTRY/INT AP is 41.561%/22.991%, below the historical pooled-V seed-0 values 46.022%/24.266%. Because F1 and that pooled run use different train/dev model- selection protocols and used the superseded opportunity loader, only its dense and raw-boundary findings remain current. Per-seed and provenance details are in the Omni floor screening record.