Add opportunity-conditioned EgoSpeak event evaluation
This commit is contained in:
@@ -34,6 +34,7 @@ tests.
|
||||
| [`easycom_competing_risk_floor_evaluation.md`](easycom_competing_risk_floor_evaluation.md) | Running and auditing the competing-risk entry/yield evaluator with training-side threshold calibration |
|
||||
| [`easycom_floor_event_diagnostics.md`](easycom_floor_event_diagnostics.md) | Separating boundary observability, event-objective mismatch and addressee information with remote commands |
|
||||
| [`easycom_event_review_tool.md`](easycom_event_review_tool.md) | Reviewing automatically mined EasyCom floor-event candidates against local video and transcripts |
|
||||
| [`easycom_opportunity_event_evaluation.md`](easycom_opportunity_event_evaluation.md) | Current opportunity-conditioned ENTRY/YIELD evaluation from EgoSpeak-compatible wearer forecasts |
|
||||
| [`easycom_egospeak_event_proxy.md`](easycom_egospeak_event_proxy.md) | Scoring the EgoSpeak wearer trajectory over equal-width causal decision regions as an activity-derived ENTRY/INT proxy |
|
||||
| [`easycom_annotated_floor_evaluation.md`](easycom_annotated_floor_evaluation.md) | Reproducing the legacy asymmetric point-positive/region-negative fixed-horizon diagnostic |
|
||||
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
# EgoSpeak dense-trajectory event proxy
|
||||
|
||||
> **Status:** boundary-centered diagnostic retained for reproducibility. The
|
||||
> manuscript's current main behavioral protocol is the matched-opportunity
|
||||
> evaluator in
|
||||
> [`easycom_opportunity_event_evaluation.md`](easycom_opportunity_event_evaluation.md).
|
||||
|
||||
`evaluate_egospeak_event_proxy.py` provides a deliberately limited bridge
|
||||
between EgoSpeak-compatible dense future-activity predictions and reviewed
|
||||
wearer ENTRY/INT opportunities. It does **not** claim that wearer voice
|
||||
|
||||
59
docs/easycom_opportunity_event_evaluation.md
Normal file
59
docs/easycom_opportunity_event_evaluation.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# EasyCom opportunity-conditioned ENTRY/YIELD evaluation
|
||||
|
||||
[`evaluate_egospeak_opportunity_events.py`](../evaluate_egospeak_opportunity_events.py)
|
||||
uses LLM-assisted or human-reviewed semantic labels to define matched
|
||||
conversational opportunities while scoring every model only from its original
|
||||
EgoSpeak-compatible wearer probabilities. It does not train a semantic event
|
||||
head.
|
||||
|
||||
## ENTRY
|
||||
|
||||
A positive must have a reviewer-selected preceding non-wearer release. Its
|
||||
opportunity starts at that release and resolves at the reviewed wearer entry.
|
||||
Core hard negatives start at the same type of non-wearer release and resolve
|
||||
when the same speaker resumes (`ENTRY_OTHER_MIDTURN_PAUSE`) or a different
|
||||
non-wearer takes the next floor (`ENTRY_OTHER_TO_OTHER`). The score is the
|
||||
maximum projected wearer probability up to the competing resolution, capped at
|
||||
2.0 seconds.
|
||||
|
||||
`ENTRY_WEARER_BACKCHANNEL` is excluded from headline ENTRY FPR. Predicting
|
||||
wearer activity for a real backchannel is correct under EgoSpeak's dense target;
|
||||
ENTRY-versus-backchannel remains a separate semantic-selectivity diagnostic.
|
||||
|
||||
## YIELD
|
||||
|
||||
An opportunity starts at a non-wearer onset during wearer speech. The score is
|
||||
the maximum `1 - p_wearer` over 0.2--0.6 seconds, censored strictly at that
|
||||
non-wearer's offset. Reviewed `INT_YIELD` items are positive; other
|
||||
backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.
|
||||
|
||||
Current INT reviews do not select one competing onset. The loader therefore
|
||||
uses the latest supplied other interval that crosses the wearer offset within
|
||||
the annotation tolerance. It records how many intervals were eligible so
|
||||
multi-competitor cases can be audited before the protocol is frozen.
|
||||
|
||||
## Provisional EgoSpeak run
|
||||
|
||||
Threshold-free AP on the official test sessions can be run without fitting a
|
||||
test threshold:
|
||||
|
||||
```bash
|
||||
python3 evaluate_egospeak_opportunity_events.py \
|
||||
--reviews reports/easycom_entry_int_sessions1_4_v16/easycom_eot_int_provisional_llm_reviews.jsonl \
|
||||
--run egospeak=results/egospeak_miniroada_floor_eval \
|
||||
--sessions Session_1 Session_2 Session_3 \
|
||||
--allow-provisional-llm-reviews \
|
||||
--print-event-examples 10 \
|
||||
--output results/easycom_egospeak_opportunity_events/sessions1_3_provisional_summary.json
|
||||
```
|
||||
|
||||
The output JSON records label coverage and every exclusion reason. The metrics
|
||||
CSV contains threshold-free ENTRY/YIELD AP. The event CSV contains anchors,
|
||||
causal source cutoffs, censor times, horizons, and wearer probabilities for
|
||||
timeline auditing.
|
||||
|
||||
Do not request matched-recall operating points on Sessions 1--3 without
|
||||
disjoint Sessions 4--12 calibration reviews and out-of-fold predictions.
|
||||
Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD
|
||||
cap, positive anchor-selection policy, and multiple-competitor handling using
|
||||
training-side data only.
|
||||
Reference in New Issue
Block a user