Add annotated fixed-horizon floor evaluation

docs/easycom_annotated_floor_evaluation.md

@@ -0,0 +1,101 @@
+# EasyCom human-annotated fixed-horizon floor-event evaluation
+
+This protocol evaluates saved EgoSpeak-layout dense predictions against
+human-confirmed `WEARER_ENTRY`, `INT_YIELD`, and hard-negative opportunities.
+It is separate from the timing-derived proxy in
+[`evaluate_floor_state_changes.py`](../evaluate_floor_state_changes.py).
+
+## Scoring contract
+
+At each nominal 0.2--2.0 second horizon:
+
+- a positive entry is one corrected 5 Hz behavior-anchor bin, scored by wearer
+  activity probability;
+- a positive INT-yield is one corrected 5 Hz behavior-anchor bin, scored by one
+  minus wearer activity probability;
+- a hard negative is one opportunity sample, scored by the maximum erroneous
+  action probability over its annotated half-open region;
+- a positive timestamp is never expanded into a tolerance window;
+- an excluded or unreviewed candidate never enters scoring.
+
+The prediction layout is `[target_time, class, future_horizon]`. Event frames
+are rasterized from the original 20 Hz annotation with the same `frame // 4`
+rule used by the EasyCom event derivation. A human `-0.2/0/+0.2` second boundary
+correction shifts the resulting target bin; for hard negatives it shifts the
+whole region.
+
+The reported horizon is the nominal EgoSpeak forecast-bin horizon. Because an
+acoustic boundary can occur anywhere inside a 200 ms target bin, do not
+reinterpret it as an exact continuous-time lead without separately reporting
+the quantization convention.
+
+## Development smoke run
+
+Use only training-side sessions while the annotation and region rules are
+being audited:
+
+```bash
+python3 evaluate_annotated_floor_events.py \
+  --reviews reports/easycom_event_reviews/reviews.jsonl \
+  --run egospeak=/path/to/egospeak/result \
+  --sessions Session_12 \
+  --matched-recalls 0.7 0.8 0.9 \
+  --print-event-examples 3 \
+  --output reports/easycom_annotated_floor_events/session12_summary.json
+```
+
+This selects thresholds on the same development events and labels the threshold
+source as exploratory. It is not a confirmatory test command.
+
+## Confirmatory test run
+
+Thresholds must be selected independently for entry and INT at every horizon
+from training-side out-of-fold predictions:
+
+```bash
+python3 evaluate_annotated_floor_events.py \
+  --reviews reports/easycom_event_reviews/test_reviews.jsonl \
+  --run baseline=/path/to/baseline/test_result \
+  --run candidate=/path/to/candidate/test_result \
+  --sessions Session_3 Session_2 Session_1 \
+  --calibration-reviews reports/easycom_event_reviews/oof_train_reviews.jsonl \
+  --calibration-run baseline=/path/to/baseline/oof_result \
+  --calibration-run candidate=/path/to/candidate/oof_result \
+  --calibration-sessions Session_4 Session_5 Session_6 Session_7 Session_8 Session_9 Session_10 Session_11 Session_12 \
+  --matched-recalls 0.7 0.8 0.9 \
+  --print-event-examples 3 \
+  --output results/easycom_annotated_floor_events_YYYY-MM-DD/summary.json
+```
+
+The CLI rejects evaluation involving official Sessions 1--3 unless calibration
+reviews and calibration sessions are supplied. The
+`--allow-in-sample-thresholds` override is exploratory only.
+
+Outputs:
+
+- `summary.json`: protocol metadata, counts, ten horizons per family, AP and
+  operating points;
+- `summary_metrics.csv`: one row per run, family, horizon, and requested recall;
+- `summary_events.csv`: one scored row per run, annotated opportunity, and
+  horizon for alignment audits and paired analysis.
+
+Before comparing models, verify identical positive, hard-negative, and invalid
+counts. The headline manuscript operating point is the nominal 0.2 second bin
+at matched recall 0.8; ten-horizon entry/INT AP and all horizon curves are
+supporting results.
+
+## Required audit before freezing
+
+1. Print examples and verify the 20 Hz boundary, 5 Hz target row, horizon, and
+   valid mask against the source timeline.
+2. Confirm that every positive occupies one target row.
+3. Confirm that each hard-negative region contributes one max-pooled score,
+   not one negative per frame.
+4. Audit continuous timeline segments without candidate hints to estimate
+   candidate recall.
+5. Freeze the annotation guide, region endpoints, boundary-correction policy,
+   and out-of-fold calibration procedure before inspecting Sessions 1--3.
+
+This evaluator does not emit committed actions or detection latency. Those
+require a separately frozen online controller with threshold, persistence,
+refractory period, matching interval, and competing-risk rules.