Fix opportunity competing-risk evaluation

This commit is contained in:
Xin Wang
2026-09-16 04:31:18 +08:00
parent 02e83c0c11
commit d4c900ab13
10 changed files with 310 additions and 88 deletions

View File

@@ -1,5 +1,11 @@
# EasyCom EgoSpeak three-level evaluation
> **Protocol erratum.** The dense and raw-boundary results in this record remain
> valid. The opportunity-conditioned ENTRY/INT values were produced by the
> superseded v1 loader, which could cross an intervening ENTRY competitor and
> selected the latest onset in ambiguous multi-competitor INT cases. Do not cite
> those opportunity values; rerun protocol v2 and create a new result record.
This record freezes the first Sessions 1--3 evaluation of the released
EgoSpeak MiniROADA audio-visual checkpoint under three deliberately separate
questions. The values below were produced on the remote experiment host; the
@@ -15,7 +21,7 @@ prediction and per-event artifacts remain on that host and are not committed.
- Raw boundary anticipation: threshold-free AP for wearer `OFF->ON` and
`ON->OFF` changes. These are timing-derived diagnostics, not semantic EOT or
INT claims.
- Opportunity-conditioned behavior: one causally censored activity-derived
- Opportunity-conditioned behavior: historical v1 activity-derived
score per reviewed ENTRY or INT/YIELD opportunity. Labels are provisional
two-pass LLM-assisted reviews and have not been human reviewed.
- No test-set threshold is fitted for any AP reported here. Session 4 is needed
@@ -34,15 +40,16 @@ mean AP, respectively. These lower values do not contradict the dense result:
dense AP rewards correct future state over sustained regions, whereas a raw
boundary event is counted once and has many harder non-boundary comparisons.
On the provisional opportunity labels, ENTRY AP is **52.487%** for 163
The superseded v1 run reported ENTRY AP of **52.487%** for 163
positives and 335 hard negatives, versus 32.731% positive prevalence
(+19.756 points; 1.60x prevalence). INT/YIELD AP is **20.135%** for 28 positives
and 109 hard negatives, essentially equal to its 20.438% prevalence. This is
consistent with the output contract: wearer probability can rank many entry
opportunities, but `1-p(wearer)` does not encode whether an interruption should
cause a yield. The ENTRY and INT loaders additionally exclude 81 and 5
invalid/post-outcome cases; these exclusions require an audit before any final
paper claim.
and 109 hard negatives, essentially equal to its 20.438% prevalence. These
values are retained only for provenance and must be replaced by a v2 rerun. The
original pattern was consistent with the output contract: wearer probability
can rank many entry opportunities, but `1-p(wearer)` does not encode whether an
interruption should cause a yield. The ENTRY and INT loaders additionally
exclude 81 and 5 invalid/post-outcome cases; these exclusions require an audit
before any final paper claim.
## Relation to EgoSpeak Figure 7
@@ -73,4 +80,3 @@ Machine-readable curves and counts are in [`summary.json`](summary.json) and
the compact comparison is in [`metrics.csv`](metrics.csv). The opportunity
protocol and reproduction command are documented in
[`docs/easycom_opportunity_event_evaluation.md`](../../docs/easycom_opportunity_event_evaluation.md).

View File

@@ -1,6 +1,8 @@
{
"recorded_at": "2026-09-16",
"source": "user-provided remote experiment output",
"opportunity_protocol_status": "superseded_v1_rerun_required",
"opportunity_protocol_erratum": "Dense and raw-boundary metrics remain valid; opportunity metrics must be rerun with egospeak-opportunity-conditioned-floor-actions-v2.",
"dataset": "EasyCom",
"sessions": ["Session_1", "Session_2", "Session_3"],
"model": "EgoSpeak MiniROADA audio-visual released checkpoint",
@@ -57,4 +59,3 @@
"threshold_free": true,
"session4_required_for_reported_metrics": false
}