icassp2027-experiment/docs/easycom_opportunity_event_evaluation.md

EasyCom opportunity-conditioned ENTRY/YIELD evaluation

evaluate_egospeak_opportunity_events.py uses LLM-assisted or human-reviewed semantic labels to define matched conversational opportunities while scoring every model only from its original EgoSpeak-compatible wearer probabilities. It does not train a semantic event head.

ENTRY

A positive must have a reviewer-selected preceding non-wearer release. Its opportunity starts at that release and resolves at the reviewed wearer entry. Core hard negatives start at the same type of non-wearer release and resolve when the same speaker resumes (ENTRY_OTHER_MIDTURN_PAUSE) or a different non-wearer takes the next floor (ENTRY_OTHER_TO_OTHER). The score is the maximum projected wearer probability up to the competing resolution, capped at 2.0 seconds.

ENTRY_WEARER_BACKCHANNEL is excluded from headline ENTRY FPR. Predicting wearer activity for a real backchannel is correct under EgoSpeak's dense target; ENTRY-versus-backchannel remains a separate semantic-selectivity diagnostic.

YIELD

An opportunity starts at a non-wearer onset during wearer speech. The score is the maximum 1 - p_wearer over 0.2--0.6 seconds, censored strictly at that non-wearer's offset. Reviewed INT_YIELD items are positive; other backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.

Current INT reviews do not select one competing onset. The loader therefore uses the latest supplied other interval that crosses the wearer offset within the annotation tolerance. It records how many intervals were eligible so multi-competitor cases can be audited before the protocol is frozen.

Provisional EgoSpeak run

Threshold-free AP on the official test sessions can be run without fitting a test threshold:

python3 evaluate_egospeak_opportunity_events.py \
  --reviews reports/easycom_entry_int_sessions1_4_v16/easycom_eot_int_provisional_llm_reviews.jsonl \
  --run egospeak=results/egospeak_miniroada_floor_eval \
  --sessions Session_1 Session_2 Session_3 \
  --allow-provisional-llm-reviews \
  --print-event-examples 10 \
  --output results/easycom_egospeak_opportunity_events/sessions1_3_provisional_summary.json

The output JSON records label coverage and every exclusion reason. The metrics CSV contains threshold-free ENTRY/YIELD AP. The event CSV contains anchors, causal source cutoffs, censor times, horizons, and wearer probabilities for timeline auditing.

Do not request matched-recall operating points on Sessions 1--3 without disjoint Sessions 4--12 calibration reviews and out-of-fold predictions. Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD cap, positive anchor-selection policy, and multiple-competitor handling using training-side data only.

Recorded Sessions 1--3 baseline

The first threshold-free run gives ENTRY AP 52.487% (163 positive, 335 hard negative; 32.731% prevalence) and INT/YIELD AP 20.135% (28 positive, 109 hard negative; 20.438% prevalence). It excludes 81 ENTRY and 5 INT cases as invalid or post-outcome. These are provisional LLM-assisted labels, not human-reviewed confirmatory results. Dense, raw-boundary, and opportunity metrics and their Figure 7 interpretation are frozen together in the 2026-09-16 result record.