Add competing-risk evaluation and align ICASSP story

This commit is contained in:
Xin Wang
2026-09-10 11:12:02 +08:00
parent 0abefd6f77
commit 4b7d2c0c20
30 changed files with 3420 additions and 724 deletions

View File

@@ -31,6 +31,7 @@ tests.
| [`easycom_highrate_person_av.md`](easycom_highrate_person_av.md) | Frozen 25 Hz causal ASD person evidence fused with 5 Hz global Omni context |
| [`easycom_muvap_stage3_baseline.md`](easycom_muvap_stage3_baseline.md) | MuVAP-only frozen VAP+ASD Stage-3 paper reconstruction on EgoSpeak/EasyCom targets |
| [`easycom_oracle_information.md`](easycom_oracle_information.md) | Causal oracle per-person speaking-history upper bound and Session-12 screening commands |
| [`easycom_competing_risk_floor_evaluation.md`](easycom_competing_risk_floor_evaluation.md) | Running and auditing the competing-risk entry/yield evaluator with training-side threshold calibration |
## Design history and implementation options

View File

@@ -0,0 +1,109 @@
# EasyCom competing-risk floor-event evaluation
This runbook evaluates whether a candidate model reduces **false wearer-entry
triggers** and **false wearer-yield triggers** relative to a baseline. It reads
the saved `pred/` and `valid/` arrays; no GPU is required.
The protocol is implemented in `evaluate_floor_state_changes.py`. It is a
timing-derived behavior evaluation, not semantic EOT/INT, addressee, or
individual-speaker ground truth: EasyCom's `others` target is aggregate
activity.
## Event attribution
At each horizon from 0.2 to 2.0 seconds, every eligible aggregate-other
boundary is counted once.
- **Response candidate:** `others` changes from active to inactive while the
wearer is inactive. A wearer onset is a response only if it occurs before
the next `others` onset.
- **Yield candidate:** `others` changes from inactive to active while the
wearer is active. A wearer offset is a yield only if it occurs before the
next `others` offset.
- The competing boundary itself is excluded. A simultaneous wearer transition
is therefore not assigned to the earlier candidate.
- An observed wearer outcome before the recording ends is retained. An
unresolved candidate without its full horizon is right-censored and omitted.
- Invalid future-score paths are omitted and counted separately.
For each event, predictions made from the same source time are max-pooled only
over the observable event window. This is a ranking score, not a calibrated
event probability.
## Confirmatory test command
Select each model's operating threshold on training-side predictions, then
apply that frozen threshold to official test Sessions 1--3. The run names in
`--calibration-run` must exactly match the names in `--run`.
```bash
python3 evaluate_floor_state_changes.py \
--target-root /path/to/easycom_target_frame \
--run baseline=/path/to/baseline/test_result \
--run candidate=/path/to/candidate/test_result \
--calibration-run baseline=/path/to/baseline/calibration_result \
--calibration-run candidate=/path/to/candidate/calibration_result \
--calibration-sessions Session_12 \
--sessions Session_3 Session_2 Session_1 \
--matched-recalls 0.7 0.8 0.9 \
--midturn-pause-max-seconds 2.0 \
--print-event-timelines 3 \
--output reports/easycom_floor_competing_risk/test_summary.json
```
If the calibration and test predictions already coexist under each `--run`
directory, omit `--calibration-run`. Official-test evaluation rejects missing
calibration and rejects calibration on Sessions 1--3. The
`--allow-in-sample-test-thresholds` option exists only for explicitly marked
exploratory audits and must not be used for a paper result.
Session 12 is shown as the current development convention. If the final model
has been trained on Session 12, prefer out-of-fold training-side predictions
or a pre-registered cross-fitting scheme; record exactly how calibration
predictions were produced.
## How to decide whether errors decreased
The command writes:
- `test_summary.json`: full protocol metadata, thresholds, counts, and metrics;
- `test_summary_events.csv`: one row per run and horizon for the primary event
evaluation;
- `test_summary_framewise.csv`: the older endpoint diagnostic, retained only
for secondary analysis.
At the headline recall (normally 0.8), compare the following event-CSV fields:
| Question | Primary field | Supporting checks |
|---|---|---|
| Did false entry decrease? | `no_response_trigger_rate_at_recall_0.8` | count, `response_recall_0.8`, response/no-response denominators |
| Did false entry during a short other-speaker restart decrease? | `short_resume_trigger_rate_at_recall_0.8` | trigger count and subgroup count |
| Did false yield/stop decrease? | `continue_false_stop_rate_at_recall_0.8` | count, `yield_recall_0.8`, yield/continue denominators |
Lower is better for all three error rates. A lower error rate is not evidence
of improvement if the candidate also obtains materially lower test response or
yield recall. The requested recall selects the threshold on calibration data;
`response_recall_0.8` and `yield_recall_0.8` report the recall actually achieved
on test and may differ from 0.8.
Before comparing models, confirm that their event denominators and invalid
window counts match. Then report both absolute percentage-point change and
relative error reduction, for example
`(baseline_rate - candidate_rate) / baseline_rate`. Event AP is a useful
threshold-free secondary result, but it does not replace the error-rate
comparison at a pre-registered operating point.
## Required manual audit before protocol freeze
Use `--print-event-timelines 3` first, then inspect a larger random sample with
an independent script or annotation view. For every printed timeline, verify:
1. the anchor row is the intended aggregate-other boundary;
2. no outcome at or after `C/excluded` is attributed to the earlier event;
3. the fixed-source horizon score and `valid` mask line up with physical time;
4. session/recording boundaries do not create transitions;
5. right-censored and invalid counts are plausible.
The current evaluator does not yet provide `third_party`, `addressed`, or
`group_or_unknown` strata, nor false entries per minute. Those require a
separately frozen addressee/scenario annotation contract and remain P0 TODOs.

View File

@@ -206,6 +206,9 @@ CUDA_VISIBLE_DEVICES=4 PERSON_EVIDENCE=none bash run_easycom_muvap_stage3.sh tes
CUDA_VISIBLE_DEVICES=5 PERSON_EVIDENCE=none bash run_easycom_muvap_stage3.sh test 2
```
Dense AP/mAP is currently the primary screening output. Floor-event results
remain timing-derived proxies until competing-risk censoring is frozen in
`evaluate_floor_state_changes.py`.
Dense AP/mAP is currently the primary screening output. For floor-event
comparison, run the competing-risk v2 evaluator with training-side threshold
calibration as documented in
[`easycom_competing_risk_floor_evaluation.md`](easycom_competing_risk_floor_evaluation.md).
The resulting events remain timing-derived behavior proxies; a real-data audit
and scenario freeze are still required before using them in the paper.

View File

@@ -81,8 +81,10 @@ checkpoint selection, and evaluates Sessions 1--3 once:
CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh final 0 1 2
```
The `final` stage enforces `EPOCHS=4` and exactly Sessions 4--12, clears the
development-session list internally, and saves the last epoch. Its artifacts
The `final` stage overrides any inherited `EPOCHS` value with `4`, requires
exactly Sessions 4--12, clears the development-session list internally, and
saves the last epoch. It validates this configuration before preparing oracle
sidecars. Its artifacts
use separate `_train4_12_fixed4` checkpoint/result roots by default, so they
cannot overwrite the historical Session-12 development screen. Override those
roots only with `FINAL_CHECKPOINT_ROOT` and `FINAL_RESULT_ROOT`.

View File

@@ -193,3 +193,33 @@ Oracle audit、LoRA 和事件协议分别是动机、baseline 和验证工具,
这个结论只有在 learned causal method、错时/容量控制和冻结后的事件指标共同通过后才能写入
摘要。
## 10. 2026-09-10 manuscript 对齐决策
对 `manuscript/main/main.tex` 的逐段审查发现并记录以下冲突;稿件骨架已据此收缩:
1. 标题只写 `floor-entry`,正文却同时主张 entry/yield,统一改为更宽的 floor
forecasting;在 same-rate 对照通过前不在标题使用 `dual-rate`。
2. 单种子 oracle 不再充当摘要 headline;最终摘要必须以 learned method 和行为级结果为
主体。
3. gap 从“Omni 尚未用于这一任务组合”改为 global semantic representation 与 sub-second
wearer-relative floor evidence 之间的 temporal--role mismatch。
4. 四项并列贡献压缩为 adaptation mismatch、一个 local adapter、causal behavioral
evidence 三项;LoRA/oracle/event protocol 分别降为 baseline、动机和验证。
5. 主方法不再预设 track-GRU/identity continuity,改为匿名、no-ID 的短时 visible-person
set;track 只有超过 no-ID/shuffled/capacity controls 后才恢复。
6. wearer current state 默认只用于 entry/yield event interpretation。若作为模型输入,必须
先定义严格因果且部署可得的来源,并加入 no-state 对照。
7. Qwen2.5-Omni-3B 为主 backbone,7B 仅作参数规模控制;MiniCPM-o 和跨家族 sweep 从四页
主线删除。
8. Introduction 不得把尚未实现的 competing-risk censoring 写成已完成;P0 evaluator、测试
和人工时间轴是正式事件结论的硬门槛。
9. 已查看的 Sessions 1--3 单种子 exploratory pooled-vision 表从投稿骨架删除,不用于模型
选择或 confirmatory claim。
10. 最终只保留一个主结果表和一个机制消融表;oracle table、conditional track table 和
候选方法树不得并列占据正文。
11. Conclusion 必须报告最终 learned method、matched-recall false-entry、onset/yield 和
causal controls;若这些证据未通过,改写为 information audit 或停止方法型投稿。
当前 `manuscript/main/main.tex` 是带显式 TODO 的内容骨架。TODO 表示未实现、未冻结或未测量,
不得被引用为项目当前能力或实验结论。

View File

@@ -86,8 +86,9 @@ speaker grounding 或 identity-aware reasoning。
2. 修正事件归因的 competing risk:
- other offset 后的 wearer response 只在下一次 other onset 之前归因;
- other onset 后的 wearer yield 只在下一次 other offset 之前归因。
现有 `evaluate_floor_state_changes.py` 使用固定 horizon 内任意 transition,暂时只能作为
proxy,可能把后来无关的动作归给前一个边界。
`evaluate_floor_state_changes.py` 已实现并通过合成单元测试;投稿前仍需在真实数据上打印
时间轴、复核 censoring/invalid 计数并冻结协议。运行方式见
[`easycom_competing_risk_floor_evaluation.md`](easycom_competing_risk_floor_evaluation.md)。
3. 固定 EasyCom train/test(4--12 / 1--3);阈值、窗口和规则不能在 test 上调。
4. 从训练 session 内留出 calibration/dev 子集,用来选择阈值和 early stopping;最终表格
仍在官方 test 上一次性报告。