Add EgoSpeak dense event proxy evaluation

This commit is contained in:
Xin Wang
2026-09-15 13:35:35 +08:00
parent b3dc02b293
commit a8a09d8fc5
8 changed files with 826 additions and 24 deletions

View File

@@ -34,7 +34,8 @@ tests.
| [`easycom_competing_risk_floor_evaluation.md`](easycom_competing_risk_floor_evaluation.md) | Running and auditing the competing-risk entry/yield evaluator with training-side threshold calibration |
| [`easycom_floor_event_diagnostics.md`](easycom_floor_event_diagnostics.md) | Separating boundary observability, event-objective mismatch and addressee information with remote commands |
| [`easycom_event_review_tool.md`](easycom_event_review_tool.md) | Reviewing automatically mined EasyCom floor-event candidates against local video and transcripts |
| [`easycom_annotated_floor_evaluation.md`](easycom_annotated_floor_evaluation.md) | Scoring human-confirmed point events and hard-negative opportunity regions at fixed EgoSpeak horizons |
| [`easycom_egospeak_event_proxy.md`](easycom_egospeak_event_proxy.md) | Scoring the EgoSpeak wearer trajectory over equal-width causal decision regions as an activity-derived ENTRY/INT proxy |
| [`easycom_annotated_floor_evaluation.md`](easycom_annotated_floor_evaluation.md) | Reproducing the legacy asymmetric point-positive/region-negative fixed-horizon diagnostic |
## Design history and implementation options

View File

@@ -1,4 +1,10 @@
# EasyCom human-annotated fixed-horizon floor-event evaluation
# Legacy EasyCom fixed-horizon floor-event diagnostic
> **Status:** retained for reproducibility, not the current manuscript event
> protocol. Its one-bin positives and variable-duration hard-negative regions
> are asymmetric. Use the equal-width activity-derived protocol in
> [`easycom_egospeak_event_proxy.md`](easycom_egospeak_event_proxy.md) for the
> current EgoSpeak baseline.
This protocol evaluates saved EgoSpeak-layout dense predictions against
human-confirmed `WEARER_ENTRY`, `INT_YIELD`, and hard-negative opportunities.

View File

@@ -0,0 +1,48 @@
# EgoSpeak dense-trajectory event proxy
`evaluate_egospeak_event_proxy.py` provides a deliberately limited bridge
between EgoSpeak-compatible dense future-activity predictions and reviewed
wearer ENTRY/INT opportunities. It does **not** claim that wearer voice
activity is a semantic floor-event detector.
## Score
For every positive event, the default decision region contains the three 5-Hz
source rows immediately before its corrected behavior anchor. An independently
mined hard negative uses the first three rows of its labeled opportunity. A raw
wearer onset/offset relabeled as a hard negative uses the same pre-boundary
region as a positive. Regions shorter than 0.6 seconds are excluded and
counted; longer annotation evidence is not scored.
At each source row, the script recovers the target-aligned wearer trajectory
at 0.6, 0.8, ..., 2.0 seconds. ENTRY uses its mean and INT uses one minus its
mean. Max-pooling over the three decision rows produces one score per reviewed
opportunity. The output must be called an `activity-derived event proxy`.
## Threshold-free provisional run
This reports AP without fitting thresholds on Sessions 1--3:
```bash
python3 evaluate_egospeak_event_proxy.py \
--reviews reports/easycom_entry_int_sessions1_4_v16/easycom_eot_int_provisional_llm_reviews.jsonl \
--run egospeak=results/egospeak_miniroada_floor_eval \
--sessions Session_1 Session_2 Session_3 \
--allow-provisional-llm-reviews \
--print-event-examples 5 \
--output results/easycom_egospeak_event_proxy/sessions1_3_provisional_summary.json
```
The command writes JSON, a compact metrics CSV, and an event CSV containing the
three decision-frame scores and the winning selected-horizon wearer trajectory.
## Recall-matched operating points
`--matched-recalls` on official test sessions additionally requires disjoint
Sessions 4--12 reviews and predictions via `--calibration-reviews`,
`--calibration-sessions`, and optionally `--calibration-run`. Do not use
`--allow-in-sample-thresholds` for reported results.
The default 0.6-second decision window and 0.6--2.0-second trajectory range
remain protocol choices to freeze using training-side data. Report shorter or
longer windows only as predeclared sensitivity analyses.