icassp2027-experiment/docs/easycom_competing_risk_floor_evaluation.md

EasyCom competing-risk floor-event evaluation

This runbook evaluates whether a candidate model reduces false wearer-entry triggers and false wearer-yield triggers relative to a baseline. It reads the saved pred/ and valid/ arrays; no GPU is required.

The protocol is implemented in evaluate_floor_state_changes.py. It is a timing-derived behavior evaluation, not semantic EOT/INT, addressee, or individual-speaker ground truth: EasyCom's others target is aggregate activity.

Event attribution

At each horizon from 0.2 to 2.0 seconds, every eligible aggregate-other boundary is counted once.

• Response candidate: others changes from active to inactive while the wearer is inactive. A wearer onset is a response only if it occurs before the next others onset.
• Yield candidate: others changes from inactive to active while the wearer is active. A wearer offset is a yield only if it occurs before the next others offset.
• The competing boundary itself is excluded. A simultaneous wearer transition is therefore not assigned to the earlier candidate.
• An observed wearer outcome before the recording ends is retained. An unresolved candidate without its full horizon is right-censored and omitted.
• Invalid future-score paths are omitted and counted separately.

For each event, predictions made from the same source time are max-pooled only over the observable event window. This is a ranking score, not a calibrated event probability.

Confirmatory test command

Select each model's operating threshold on training-side predictions, then apply that frozen threshold to official test Sessions 1--3. The run names in --calibration-run must exactly match the names in --run.

python3 evaluate_floor_state_changes.py \
  --target-root /path/to/easycom_target_frame \
  --run baseline=/path/to/baseline/test_result \
  --run candidate=/path/to/candidate/test_result \
  --calibration-run baseline=/path/to/baseline/calibration_result \
  --calibration-run candidate=/path/to/candidate/calibration_result \
  --calibration-sessions Session_12 \
  --sessions Session_3 Session_2 Session_1 \
  --matched-recalls 0.7 0.8 0.9 \
  --midturn-pause-max-seconds 2.0 \
  --print-event-timelines 3 \
  --output reports/easycom_floor_competing_risk/test_summary.json

If the calibration and test predictions already coexist under each --run directory, omit --calibration-run. Official-test evaluation rejects missing calibration and rejects calibration on Sessions 1--3. The --allow-in-sample-test-thresholds option exists only for explicitly marked exploratory audits and must not be used for a paper result.

Session 12 is shown as the current development convention. If the final model has been trained on Session 12, prefer out-of-fold training-side predictions or a pre-registered cross-fitting scheme; record exactly how calibration predictions were produced.

How to decide whether errors decreased

The command writes:

• test_summary.json: full protocol metadata, thresholds, counts, and metrics;
• test_summary_events.csv: one row per run and horizon for the primary event evaluation;
• test_summary_framewise.csv: the older endpoint diagnostic, retained only for secondary analysis.

At the headline recall (normally 0.8), compare the following event-CSV fields:

Question	Primary field	Supporting checks
Did false entry decrease?	no_response_trigger_rate_at_recall_0.8	count, response_recall_0.8, response/no-response denominators
Did false entry during a short other-speaker restart decrease?	short_resume_trigger_rate_at_recall_0.8	trigger count and subgroup count
Did false yield/stop decrease?	continue_false_stop_rate_at_recall_0.8	count, yield_recall_0.8, yield/continue denominators

Lower is better for all three error rates. A lower error rate is not evidence of improvement if the candidate also obtains materially lower test response or yield recall. The requested recall selects the threshold on calibration data; response_recall_0.8 and yield_recall_0.8 report the recall actually achieved on test and may differ from 0.8.

Before comparing models, confirm that their event denominators and invalid window counts match. Then report both absolute percentage-point change and relative error reduction, for example (baseline_rate - candidate_rate) / baseline_rate. Event AP is a useful threshold-free secondary result, but it does not replace the error-rate comparison at a pre-registered operating point.

Required manual audit before protocol freeze

Use --print-event-timelines 3 first, then inspect a larger random sample with an independent script or annotation view. For every printed timeline, verify:

1. the anchor row is the intended aggregate-other boundary;
2. no outcome at or after C/excluded is attributed to the earlier event;
3. the fixed-source horizon score and valid mask line up with physical time;
4. session/recording boundaries do not create transitions;
5. right-censored and invalid counts are plausible.

The current evaluator does not yet provide third_party, addressed, or group_or_unknown strata, nor false entries per minute. Those require a separately frozen addressee/scenario annotation contract and remain P0 TODOs.