Add causal person track GRU residual experiment

2026-09-07 00:30:23 +08:00
parent 9fc4577aea
commit 4aed4697a5
15 changed files with 1185 additions and 43 deletions
--- a/docs/README.md
+++ b/docs/README.md
@@ -23,6 +23,7 @@ tests.
 | [`easycom_encoder_only.md`](easycom_encoder_only.md) | Comparing EasyCom Qwen encoder-only A/V/AV representations |
 | [`ego4d_encoder_only.md`](ego4d_encoder_only.md) | Preparing and running the Ego4D encoder-only comparison |
 | [`ego4d_local_baselines.md`](ego4d_local_baselines.md) | Previewing Ego4D boxes, extracting local features and running 8-GPU jobs |
+| [`easycom_track_gru_residual.md`](easycom_track_gru_residual.md) | Matched pooled-to-track GRU experiment, causal identity contract and L20 commands |

 ## Design history and implementation options

--- a/docs/easycom_track_gru_residual.md
+++ b/docs/easycom_track_gru_residual.md
@@ -0,0 +1,158 @@
+# EasyCom anonymous track-GRU residual
+
+## Decision
+
+The first track-temporal experiment keeps the strongest pooled local baseline
+as an exact, optionally frozen base and learns only a causal person-track
+correction:
+
+```text
+pooled base:  G_t + A_t + pooled-V_t -> concat-MLP -> base logits
+
+per visible person i:
+  past crop observations on the same temporary track
+      -> shared projection -> shared one-layer GRU -> H_t,i
+
+G_t + A_t -> masked attention over {NULL, H_t,1, ..., H_t,N}
+          -> zero-gated residual logits
+
+final logits = base logits + tanh(alpha) * person_valid * residual logits
+alpha starts at exactly zero
+```
+
+This is the smallest experiment that asks whether anonymous visual continuity
+adds information beyond `G+A+pooled-V`. It is not yet person-conditioned audio
+grounding and does not modify the Omni Thinker.
+
+## What was taken from EgoSpeak
+
+The official EgoSpeak repository is cloned for inspection at
+`third_party/EgoSpeak` and excluded from Git. The inspected revision is
+`b5bee35e796d8f2d6aa7da2bf1b07836270f8bfb`.
+
+Its `MiniROADA` model projects concatenated RGB/audio features, applies a
+single-layer GRU, expands the recurrent state into ten anticipation states,
+and uses a shared classifier. The EasyCom configuration uses a 128-step
+window, stride 4, 2048D embedding, 1024D GRU and ten future steps.
+
+We reuse the defensible ideas—causal recurrence, shared weights and a ten-step
+anticipation output—but not the implementation. Our GRU is shared across
+anonymous person tracks, reads only crop observations at or before the current
+cutoff, and corrects the already measured pooled Omni baseline. Consequently,
+this method must be described as an EgoSpeak-inspired track-temporal adapter,
+not an EgoSpeak GRU reproduction.
+
+## Causal and identity contract
+
+- A 5 Hz row with cutoff `(t+1)/5` reads the latest local video frame strictly
+  before that cutoff.
+- Track history contains at most `--person-history-frames` real 2 FPS crop
+  observations and is ordered oldest to newest.
+- Temporary IDs only link observations. They are never embedded or passed to
+  the model, so global ID renaming cannot change a prediction.
+- A source-clip boundary always resets a track. A time gap larger than
+  `--person-track-max-gap-seconds` also resets it.
+- Missing, `-1`, `None`, `null` and `unknown` IDs are isolated observations.
+- Duplicate known IDs in one frame fail instead of being silently merged.
+- Empty person sets produce exactly zero visual correction.
+
+The existing EasyCom local sidecars already contain crop features, frame times,
+source-clip indices and person IDs; this experiment does not require feature
+re-extraction.
+
+## Fair initialization
+
+`track-gru-residual` requires `--initialize-pooled-checkpoint`. The loader
+checks dataset, train/dev split, loss, layer fusion, dimensions and local
+feature contract before copying the pooled path. By default those copied
+parameters are frozen. The residual gate starts at zero, so before optimization
+the track model must reproduce the pooled logits exactly, element for element.
+
+The matched screening protocol is:
+
+- EasyCom Session 4--11: training;
+- Session 12: development/model selection;
+- Session 1--3: final test, run only after history/gap and the method are frozen;
+- 4 epochs, three seeds, identical optimizer/loss for pooled and track runs;
+- 16 observations at 2 FPS (up to 8 seconds), 1-second maximum link gap.
+
+This protocol intentionally re-runs the pooled base. Historical no-dev results
+cannot be used as the matched control.
+
+## Run
+
+After pulling on the L20 machine, run the focused CPU regression tests first:
+
+```bash
+python -m unittest \
+  test_local_turn_features.LocalTests.test_track_loader_builds_causal_histories_and_resets_tracks \
+  test_local_turn_features.LocalTests.test_track_gru_residual_starts_from_exact_pooled_logits \
+  test_local_turn_features.RoundTripTests.test_train_infer_easycom_and_ego4d
+```
+
+One-seed runtime smoke on an L20:
+
+```bash
+EPOCHS=1 \
+SESSIONS="4 5" \
+DEV_SESSIONS=5 \
+SCREEN_SESSIONS=5 \
+CHECKPOINT_ROOT=checkpoints/easycom_track_gru_smoke \
+RESULT_ROOT=results/easycom_track_gru_smoke \
+CUDA_VISIBLE_DEVICES=0 \
+bash run_easycom_track_gru_residual.sh all 0
+```
+
+Then use three GPUs in three foreground terminals. These runs report Session
+12 development scores, not the official test set:
+
+```bash
+SUMMARIZE=0 CUDA_VISIBLE_DEVICES=0 bash run_easycom_track_gru_residual.sh all 0
+SUMMARIZE=0 CUDA_VISIBLE_DEVICES=1 bash run_easycom_track_gru_residual.sh all 1
+SUMMARIZE=0 CUDA_VISIBLE_DEVICES=2 bash run_easycom_track_gru_residual.sh all 2
+```
+
+After all development jobs finish:
+
+```bash
+python summarize_local_baselines.py \
+  --root results/easycom_track_gru_residual \
+  --dataset easycom \
+  --protocol causal_valid_only \
+  --paired-deltas
+```
+
+Only after choosing and freezing the method, evaluate the saved checkpoints
+once on Sessions 1--3:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 bash run_easycom_track_gru_residual.sh test 0 1 2
+```
+
+This writes and summarizes `results/easycom_track_gru_residual_test` without
+retraining. A single GPU is sufficient because head inference is inexpensive.
+
+Paths and hyperparameters can be overridden with the environment variables at
+the top of `run_easycom_track_gru_residual.sh`. The launcher runs in the
+foreground and refuses to overwrite an existing checkpoint or result.
+
+## Interpretation and stopping rule
+
+The initial screen answers only whether track-temporal visual structure can
+improve the dense EgoSpeak-compatible metric. Advance this branch only if
+same-seed `track_gru - pooled` deltas are directionally consistent on Session
+12 without sacrificing short-horizon wearer AP. The official test is a final
+confirmation, not another model-selection stage.
+Because prior F1 variation was roughly a few tenths of a percentage point,
+tiny single-seed changes are not evidence.
+
+If the track branch wins, next run correct-track versus broken-track,
+past-shifted and no-ID temporal controls, followed by person-conditioned audio
+grounding. If it does not win, first compare 8/16/32 observations and optionally
+unfreeze the pooled base using Session 12 only; do not add deeper cross-attention
+merely to rescue a null result.
+
+The paper-level claim still requires the frozen competing-risk event evaluator
+and matched-recall false-entry analysis. Dense AP/mAP improvement alone is a
+representation result, not proof that the model understands multi-person floor
+behavior.
--- a/docs/progressive_feature_fusion_route_2026-09-02.md
+++ b/docs/progressive_feature_fusion_route_2026-09-02.md
@@ -21,11 +21,14 @@
 融合实验从 F1 Frame-set Attention 开始。后续模块只有在前一级通过预先定义的 gate 后
 才实施。

-实现状态（2026-09-02）：F1 的 loader、head、train/infer 路径和回归测试已经加入代码，
-但尚未在远程 PyTorch/L20 环境完成 runtime smoke，不能写成已验证结果。另加入两个仅用
+实现状态（更新于 2026-09-07）：F1 已完成三 seed、四 epoch 的远程筛选，得到
+`81.77745 +/- 0.15638` 的 0.2 秒 mAP 和 `64.95099 +/- 0.09451` 的十 horizon mean
+mAP。它表现稳定，但与旧 pooled 结果的 split/dev 协议并不匹配，而且尚无 correct/wrong
+视觉控制，因此不能写成注意力优于 pooling。另加入两个仅用
 当前 sidecar 的 F5/F6 **screening** 模式：`audio-person-frame-set` 和
 `gated-person-state-adapter`。前者没有 F4 人物历史，后者位于 Thinker readout 之后；二者
-都不能写成完整 F5/F6。F2--F4、真正的 F5/F6 和 D1 仍然只是计划。
+都不能写成完整 F5/F6。第一版 F3 `track-gru-residual` 已实现但尚待远程运行；F2、F4、
+真正的 F5/F6 和 D1 仍然只是计划。

 ## 2. 不可改变的实验条件

@@ -193,6 +196,14 @@ floor-action 结果。
 对照包括 frame-set、无 ID temporal pool、正确 track、打断连续性的 track 以及 past-shift。
 正确 track 必须优于无 ID 和打断 track，才能说明身份连续性有价值。

+第一版实现为 `track-gru-residual`。它没有直接替换 pooled 头，而是从同 seed、同 split、
+同 local contract 的 `G+A+pooled-V` checkpoint 精确复制并默认冻结 base path，再增加共享
+track GRU、G+A 对 track states 的 masked attention 以及零初始化 scalar gate。这样训练
+开始前 logits 与 pooled baseline 逐元素相同，空人物时 correction 恒为零，实验首先回答
+“track continuity 是否提供增量信息”，而不会把随机重训 base 的波动误当作 GRU 收益。
+实现、契约和命令见
+[`easycom_track_gru_residual.md`](easycom_track_gru_residual.md)。
+
 ## 8. F4：Person-conditioned audio/Omni history

 F4 让每个匿名人物分别读取已有的 5 Hz 因果历史：
@@ -268,11 +279,11 @@ README 必须写明 split、seed、checkpoint、输入特征版本、评测协

 ## 12. 最近执行顺序

-1. 完成并测试 P0 competing-risk censoring，人工打印至少一条 response 和 yield 时间轴。
-2. 扩展 local sidecar loader，使其在不破坏现有 pooled 加载路径的情况下返回逐帧人物
-   feature、offset 和 mask；测试成功、失败、空人物和越界路径。
-3. 实现 F1 最小 Frame-set Attention，不加入 bbox 或 track temporal。
-4. 单 session/短片 smoke：检查 shape、cutoff、chosen frame time、mask、空人物和排列
-   不变性。
-5. 在固定 train-internal dev 上运行 B1、B3、F1-a、F1-c、F1-d、F1-e 和 F1-g。
-6. F1 通过后才进入 F2；若未通过，优先审计人物 crop 和 AV 对齐，不继续堆模型。
+1. 用户当前选择先做 dense-AP model screen；运行 matched pooled 与
+   `track-gru-residual`，固定 Session 12 dev，不用测试集选历史长度。
+2. 若 track GRU 有稳定增益，补 broken-track、past-shift、no-ID 和 remove-person 控制；若
+   无增益，只在 dev 上比较 8/16/32 observations 后停止该分支。
+3. correct track 通过控制后，再实现 F4 person-conditioned audio/Omni history；否则回到
+   对齐审计，不直接堆更深 interaction。
+4. 在形成论文事件结论前，仍须完成 P0 competing-risk censoring，并人工打印 response 与
+   yield 时间轴。当前事件结果只能称为 timing-derived proxy。