Fix opportunity competing-risk evaluation

This commit is contained in:
Xin Wang
2026-09-16 04:31:18 +08:00
parent 02e83c0c11
commit d4c900ab13
10 changed files with 310 additions and 88 deletions

View File

@@ -10,6 +10,12 @@ head.
A positive must have a reviewer-selected preceding non-wearer release. Its
opportunity starts at that release and resolves at the reviewed wearer entry.
If a new non-wearer onset occurs before or exactly at that entry, the release
has already been resolved by a competing event and the positive is excluded.
The same applies when wearer activity already occurred between the selected
release and the reviewed onset: the selected release is then a stale,
post-outcome anchor. Both cases are counted separately in
`review_diagnostics.skipped`.
Core hard negatives start at the same type of non-wearer release and resolve
when the same speaker resumes (`ENTRY_OTHER_MIDTURN_PAUSE`) or a different
non-wearer takes the next floor (`ENTRY_OTHER_TO_OTHER`). The score is the
@@ -27,10 +33,10 @@ the maximum `1 - p_wearer` over 0.2--0.6 seconds, censored strictly at that
non-wearer's offset. Reviewed `INT_YIELD` items are positive; other
backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.
Current INT reviews do not select one competing onset. The loader therefore
uses the latest supplied other interval that crosses the wearer offset within
the annotation tolerance. It records how many intervals were eligible so
multi-competitor cases can be audited before the protocol is frozen.
Current INT reviews do not select one competing onset. A unique supplied other
interval that crosses the wearer offset within the annotation tolerance is
used. Cases with multiple eligible intervals are excluded and counted rather
than being assigned a deterministic latest-onset anchor.
## Provisional EgoSpeak run
@@ -58,21 +64,24 @@ Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD
cap, positive anchor-selection policy, and multiple-competitor handling using
training-side data only.
## Recorded Sessions 1--3 baseline
## Superseded v1 Sessions 1--3 baseline
The first threshold-free run gives ENTRY AP 52.487% (163 positive, 335 hard
negative; 32.731% prevalence) and INT/YIELD AP 20.135% (28 positive, 109 hard
negative; 20.438% prevalence). It excludes 81 ENTRY and 5 INT cases as invalid
or post-outcome. These are provisional LLM-assisted labels, not human-reviewed
confirmatory results. Dense, raw-boundary, and opportunity metrics and their
Figure 7 interpretation are frozen together in the
The first threshold-free run used the superseded v1 loader and reported ENTRY
AP 52.487% (163 positive, 335 hard negative; 32.731% prevalence) and INT/YIELD
AP 20.135% (28 positive, 109 hard negative; 20.438% prevalence). It excluded
81 ENTRY and 5 INT cases as invalid or post-outcome. These opportunity values
must not be reported: v1 allowed a
selected ENTRY release to cross an intervening other onset and selected the
latest onset in ambiguous multi-competitor INT cases. Dense and raw-boundary
metrics are unaffected. Rerun the opportunity evaluation with protocol v2
before updating the manuscript. The historical output is retained in the
[`2026-09-16 result record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md).
The expanded Omni screen finds no opportunity-ranking gain from the current
The expanded v1 Omni screen found no opportunity-ranking gain from the current
F1 Frame-set Attention model: its three-seed ENTRY/INT AP is
41.561%/22.991%, below the historical pooled-V seed-0 values
46.022%/24.266%. Because F1 and that pooled run use different train/dev model-
selection protocols, this is a stable negative screen rather than a matched
confirmatory comparison. Dense, boundary, per-seed, and provenance details are
selection protocols and used the superseded opportunity loader, only its dense
and raw-boundary findings remain current. Per-seed and provenance details are
in the
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md).

View File

@@ -168,12 +168,13 @@ forecasting 的定性解释,不能代替 boundary 或 semantic opportunity 评
曲线和限制见
[`three-level baseline record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md)。
当前 F1 current-frame person-set screen 的三种子结果同样没有支持 semantic floor
selectivity:ENTRY/INT opportunity AP 为 `41.561 +/- 0.581` / `22.991 +/- 0.202`,
低于历史 pooled-V seed-0 的 `46.022/24.266`;dense mean mAP 也为
`64.951 +/- 0.095`,没有超过历史 pooled-V 的 `65.400`。由于两者 train/dev
选择协议不同,这只能判为稳定的负向 screening,正式归因前必须补同协议三种子 pooled
control。详见 [`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md)。
当前 F1 current-frame person-set screen 的 dense mean mAP 为
`64.951 +/- 0.095`,没有超过历史 pooled-V 的 `65.400`。原 v1 evaluator 给出的
ENTRY/INT opportunity AP(`41.561 +/- 0.581` / `22.991 +/- 0.202`)已失效:v1
可能跨过 competing ENTRY onset,且会在多个 INT competitor 中任意选择 latest onset。
必须用 v2 重跑后才能判断 semantic floor selectivity;正式归因还必须补同协议三种子
pooled control。历史数值与勘误见
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md)。
### P3:第一优先方法——匿名逐人时序旁路