icassp2027-experiment/docs/easycom_competing_risk_floor_evaluation.md

EasyCom competing-risk floor-event evaluation

This runbook evaluates whether a candidate model reduces false wearer-entry triggers and false wearer-yield triggers relative to a baseline. It reads the saved pred/ and valid/ arrays; no GPU is required.

The protocol is implemented in evaluate_floor_state_changes.py. It is a timing-derived behavior evaluation, not semantic EOT/INT, addressee, or individual-speaker ground truth: EasyCom's others target is aggregate activity.

Raw EgoSpeak MiniROADA predictions use a different source-indexed layout. Use prepare_egospeak_floor_eval.py to align them and create an explicit valid mask before including the GRU A+V baseline in the same comparison:

python3 prepare_egospeak_floor_eval.py \
  --prediction-root /path/to/miniroad/raw/eval \
  --target-root /path/to/easycom_target_frame \
  --output-dir results/egospeak_miniroada_floor_eval \
  --sessions Session_12 Session_3 Session_2 Session_1

The converter requires raw arrays shaped [T-10,10,3] and fails rather than guessing if their shape, range, or target length differs. Do not transpose or pad MiniROADA output by hand.

Event attribution

At each horizon from 0.2 to 2.0 seconds, every eligible aggregate-other boundary is counted once.

• Response candidate: others changes from active to inactive while the wearer is inactive. A wearer onset is a response only if it occurs before the next others onset.
• Yield candidate: others changes from inactive to active while the wearer is active. A wearer offset is a yield only if it occurs before the next others offset.
• The competing boundary itself is excluded. A simultaneous wearer transition is therefore not assigned to the earlier candidate.
• An observed wearer outcome before the recording ends is retained. An unresolved candidate without its full horizon is right-censored and omitted.
• Invalid future-score paths are omitted and counted separately.

For each event, predictions made from the same source time are max-pooled only over the observable event window. This is a ranking score, not a calibrated event probability.

Confirmatory test command

Select each model's operating threshold on training-side predictions, then apply that frozen threshold to official test Sessions 1--3. The run names in --calibration-run must exactly match the names in --run.

python3 evaluate_floor_state_changes.py \
  --target-root /path/to/easycom_target_frame \
  --run baseline=/path/to/baseline/test_result \
  --run candidate=/path/to/candidate/test_result \
  --calibration-run baseline=/path/to/baseline/calibration_result \
  --calibration-run candidate=/path/to/candidate/calibration_result \
  --calibration-sessions Session_12 \
  --sessions Session_3 Session_2 Session_1 \
  --matched-recalls 0.7 0.8 0.9 \
  --midturn-pause-max-seconds 2.0 \
  --print-event-timelines 3 \
  --output reports/easycom_floor_competing_risk/test_summary.json

If the calibration and test predictions already coexist under each --run directory, omit --calibration-run. Official-test evaluation rejects missing calibration and rejects calibration on Sessions 1--3. The --allow-in-sample-test-thresholds option exists only for explicitly marked exploratory audits and must not be used for a paper result.

Session 12 is shown as the current development convention. If the final model has been trained on Session 12, prefer out-of-fold training-side predictions or a pre-registered cross-fitting scheme; record exactly how calibration predictions were produced.

How to decide whether errors decreased

The command writes:

• test_summary.json: full protocol metadata, thresholds, counts, and metrics;
• test_summary_events.csv: one row per run and horizon for the primary event evaluation;
• test_summary_framewise.csv: the older endpoint diagnostic, retained only for secondary analysis.

At the headline recall (normally 0.8), compare the following event-CSV fields:

Question	Primary field	Supporting checks
Did false entry decrease?	no_response_trigger_rate_at_recall_0.8	count, response_recall_0.8, response/no-response denominators
Did false entry during a short other-speaker restart decrease?	short_resume_trigger_rate_at_recall_0.8	trigger count and subgroup count
Did false yield/stop decrease?	continue_false_stop_rate_at_recall_0.8	count, yield_recall_0.8, yield/continue denominators

Lower is better for all three error rates. A lower error rate is not evidence of improvement if the candidate also obtains materially lower test response or yield recall. The requested recall selects the threshold on calibration data; response_recall_0.8 and yield_recall_0.8 report the recall actually achieved on test and may differ from 0.8.

Before comparing models, confirm that their event denominators and invalid window counts match. Then report both absolute percentage-point change and relative error reduction, for example (baseline_rate - candidate_rate) / baseline_rate. Event AP is a useful threshold-free secondary result, but it does not replace the error-rate comparison at a pre-registered operating point.

Required manual audit before protocol freeze

Use --print-event-timelines 3 first, then inspect a larger random sample with an independent script or annotation view. For every printed timeline, verify:

1. the anchor row is the intended aggregate-other boundary;
2. no outcome at or after C/excluded is attributed to the earlier event;
3. the fixed-source horizon score and valid mask line up with physical time;
4. session/recording boundaries do not create transitions;
5. right-censored and invalid counts are plausible.

The current evaluator does not yet provide third_party, addressed, or group_or_unknown strata, nor false entries per minute. Those require a separately frozen addressee/scenario annotation contract and remain P0 TODOs.