Fix opportunity competing-risk evaluation
This commit is contained in:
@@ -10,6 +10,12 @@ head.
|
||||
|
||||
A positive must have a reviewer-selected preceding non-wearer release. Its
|
||||
opportunity starts at that release and resolves at the reviewed wearer entry.
|
||||
If a new non-wearer onset occurs before or exactly at that entry, the release
|
||||
has already been resolved by a competing event and the positive is excluded.
|
||||
The same applies when wearer activity already occurred between the selected
|
||||
release and the reviewed onset: the selected release is then a stale,
|
||||
post-outcome anchor. Both cases are counted separately in
|
||||
`review_diagnostics.skipped`.
|
||||
Core hard negatives start at the same type of non-wearer release and resolve
|
||||
when the same speaker resumes (`ENTRY_OTHER_MIDTURN_PAUSE`) or a different
|
||||
non-wearer takes the next floor (`ENTRY_OTHER_TO_OTHER`). The score is the
|
||||
@@ -27,10 +33,10 @@ the maximum `1 - p_wearer` over 0.2--0.6 seconds, censored strictly at that
|
||||
non-wearer's offset. Reviewed `INT_YIELD` items are positive; other
|
||||
backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.
|
||||
|
||||
Current INT reviews do not select one competing onset. The loader therefore
|
||||
uses the latest supplied other interval that crosses the wearer offset within
|
||||
the annotation tolerance. It records how many intervals were eligible so
|
||||
multi-competitor cases can be audited before the protocol is frozen.
|
||||
Current INT reviews do not select one competing onset. A unique supplied other
|
||||
interval that crosses the wearer offset within the annotation tolerance is
|
||||
used. Cases with multiple eligible intervals are excluded and counted rather
|
||||
than being assigned a deterministic latest-onset anchor.
|
||||
|
||||
## Provisional EgoSpeak run
|
||||
|
||||
@@ -58,21 +64,24 @@ Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD
|
||||
cap, positive anchor-selection policy, and multiple-competitor handling using
|
||||
training-side data only.
|
||||
|
||||
## Recorded Sessions 1--3 baseline
|
||||
## Superseded v1 Sessions 1--3 baseline
|
||||
|
||||
The first threshold-free run gives ENTRY AP 52.487% (163 positive, 335 hard
|
||||
negative; 32.731% prevalence) and INT/YIELD AP 20.135% (28 positive, 109 hard
|
||||
negative; 20.438% prevalence). It excludes 81 ENTRY and 5 INT cases as invalid
|
||||
or post-outcome. These are provisional LLM-assisted labels, not human-reviewed
|
||||
confirmatory results. Dense, raw-boundary, and opportunity metrics and their
|
||||
Figure 7 interpretation are frozen together in the
|
||||
The first threshold-free run used the superseded v1 loader and reported ENTRY
|
||||
AP 52.487% (163 positive, 335 hard negative; 32.731% prevalence) and INT/YIELD
|
||||
AP 20.135% (28 positive, 109 hard negative; 20.438% prevalence). It excluded
|
||||
81 ENTRY and 5 INT cases as invalid or post-outcome. These opportunity values
|
||||
must not be reported: v1 allowed a
|
||||
selected ENTRY release to cross an intervening other onset and selected the
|
||||
latest onset in ambiguous multi-competitor INT cases. Dense and raw-boundary
|
||||
metrics are unaffected. Rerun the opportunity evaluation with protocol v2
|
||||
before updating the manuscript. The historical output is retained in the
|
||||
[`2026-09-16 result record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md).
|
||||
|
||||
The expanded Omni screen finds no opportunity-ranking gain from the current
|
||||
The expanded v1 Omni screen found no opportunity-ranking gain from the current
|
||||
F1 Frame-set Attention model: its three-seed ENTRY/INT AP is
|
||||
41.561%/22.991%, below the historical pooled-V seed-0 values
|
||||
46.022%/24.266%. Because F1 and that pooled run use different train/dev model-
|
||||
selection protocols, this is a stable negative screen rather than a matched
|
||||
confirmatory comparison. Dense, boundary, per-seed, and provenance details are
|
||||
selection protocols and used the superseded opportunity loader, only its dense
|
||||
and raw-boundary findings remain current. Per-seed and provenance details are
|
||||
in the
|
||||
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md).
|
||||
|
||||
@@ -168,12 +168,13 @@ forecasting 的定性解释,不能代替 boundary 或 semantic opportunity 评
|
||||
曲线和限制见
|
||||
[`three-level baseline record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md)。
|
||||
|
||||
当前 F1 current-frame person-set screen 的三种子结果同样没有支持 semantic floor
|
||||
selectivity:ENTRY/INT opportunity AP 为 `41.561 +/- 0.581` / `22.991 +/- 0.202`,
|
||||
低于历史 pooled-V seed-0 的 `46.022/24.266`;dense mean mAP 也为
|
||||
`64.951 +/- 0.095`,没有超过历史 pooled-V 的 `65.400`。由于两者 train/dev
|
||||
选择协议不同,这只能判为稳定的负向 screening,正式归因前必须补同协议三种子 pooled
|
||||
control。详见 [`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md)。
|
||||
当前 F1 current-frame person-set screen 的 dense mean mAP 为
|
||||
`64.951 +/- 0.095`,没有超过历史 pooled-V 的 `65.400`。原 v1 evaluator 给出的
|
||||
ENTRY/INT opportunity AP(`41.561 +/- 0.581` / `22.991 +/- 0.202`)已失效:v1
|
||||
可能跨过 competing ENTRY onset,且会在多个 INT competitor 中任意选择 latest onset。
|
||||
必须用 v2 重跑后才能判断 semantic floor selectivity;正式归因还必须补同协议三种子
|
||||
pooled control。历史数值与勘误见
|
||||
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md)。
|
||||
|
||||
### P3:第一优先方法——匿名逐人时序旁路
|
||||
|
||||
|
||||
Reference in New Issue
Block a user