diff --git a/PROJECT_CONTEXT.md b/PROJECT_CONTEXT.md index 1b939a9..c3f6de6 100644 --- a/PROJECT_CONTEXT.md +++ b/PROJECT_CONTEXT.md @@ -52,7 +52,7 @@ result. matched improvement over the historical no-dev pooled result; correct/wrong visual controls remain missing. -### Implemented, awaiting remote runtime verification +### Measured local-fusion screens - Two current-sidecar screening controls: `audio-person-frame-set` adds a zero-gated audio-to-person read before the F1 query, and @@ -62,10 +62,15 @@ result. - `track-gru-residual` builds strictly causal histories from anonymous provided track IDs, applies one shared GRU per track, lets G+A query current track states, and adds a zero-gated correction to an exactly initialized pooled - `G+A+V` checkpoint. Its three focused loader/head/CLI round-trip tests passed - on the remote PyTorch/L20 environment on 2026-09-07; the one-session GPU - smoke and measured experiment are still pending. See - [`docs/easycom_track_gru_residual.md`](docs/easycom_track_gru_residual.md). + `G+A+V` checkpoint. The matched Session-12 three-seed screen was null + (`+0.01306 +/- 0.02368` percentage points in ten-horizon mean mAP). A seed-0 + Sessions-1--3 audit reproduced pooled predictions to within `-0.00030` + points. Saved residual gates were only `-0.0015` to `+0.0108`; because a zero + scalar gate also blocks the initial branch gradient, this is recorded as an + optimization failure rather than evidence that track continuity is useless. + See the immutable + [`result record`](results/easycom_track_gru_residual_4ep_2026-09-07/README.md) + and [`implementation note`](docs/easycom_track_gru_residual.md). ### Current blocker @@ -89,9 +94,10 @@ diagnostics until competing-risk censoring is fixed. 2. **P1:** probe aggregate activity and per-person active-speaker grounding; compare correct, wrong-person and time-shifted visual inputs. 3. **P2:** re-evaluate `G`, `G+A`, `G+A+pooled-V` under the frozen event protocol. -4. **P3:** run the matched pooled/anonymous-track-GRU residual screen, then add - broken-ID, wrong-time and remove-person controls before claiming that track - continuity matters. +4. **P3:** repair the matched anonymous-track-GRU residual optimization by + replacing the scalar zero gate with a zero-initialized residual output + layer, and verify nonzero branch gradients on Session 12. Add broken-ID, + wrong-time and remove-person controls only if that corrected screen improves. 5. **P4:** attempt person-memory adapter/Thinker LoRA only if P1/P3 pass. The detailed architecture, controls and stopping gates for P3 are frozen in @@ -228,6 +234,7 @@ Do not reorganize them physically without compatibility wrappers. | `derive_easycom_interaction_events.py` | Deterministic transcript/timing proxies | | `adjudicate_easycom_interaction_events.py` | Optional ambiguous-event adjudication | | `audit_easycom_floor_events.py` | Count EOT/takeover/backchannel candidates | +| `prepare_easycom_oracle_features.py` | Build explicit past-only oracle person-activity sidecars for information-value screening | ### Local features @@ -245,6 +252,7 @@ Do not reorganize them physically without compatibility wrappers. | `evaluate_floor_state_changes.py` | CPU floor-state/event proxy evaluation | | `summarize_local_baselines.py` | Aggregate local baseline `metrics.json` files | | `run_easycom_track_gru_residual.sh` | Matched pooled and anonymous track-GRU training/inference launcher | +| `run_easycom_oracle_information.sh` | Prepare and screen aggregate/per-person/shuffled/delayed oracle conditions on Session 12 | | `train_omni_turn_lora.py` | Thinker LoRA training from cached AV | | `eval_omni_turn_lora.py` | Thinker LoRA evaluation | diff --git a/docs/README.md b/docs/README.md index 462df5c..cef2cc2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -24,6 +24,7 @@ tests. | [`ego4d_encoder_only.md`](ego4d_encoder_only.md) | Preparing and running the Ego4D encoder-only comparison | | [`ego4d_local_baselines.md`](ego4d_local_baselines.md) | Previewing Ego4D boxes, extracting local features and running 8-GPU jobs | | [`easycom_track_gru_residual.md`](easycom_track_gru_residual.md) | Matched pooled-to-track GRU experiment, causal identity contract and L20 commands | +| [`easycom_oracle_information.md`](easycom_oracle_information.md) | Causal oracle per-person speaking-history upper bound and Session-12 screening commands | ## Design history and implementation options diff --git a/docs/easycom_oracle_information.md b/docs/easycom_oracle_information.md new file mode 100644 index 0000000..56aa072 --- /dev/null +++ b/docs/easycom_oracle_information.md @@ -0,0 +1,100 @@ +# EasyCom causal oracle-person information screen + +## Question + +Before training a heavier visual grounding model, test whether perfect +participant-resolved **past** speaking activity contains useful information for +future wearer activity beyond the frozen Omni state and recent mixed audio. +This is an information upper-bound experiment, not a deployable input setting. + +## Causal contract + +- The task grid remains 5 Hz. State row `t` has cutoff `(t+1)/5` seconds. +- Oracle bin `t` summarizes transcript frames in + `[t/5,(t+1)/5)` and is therefore completely observed at that cutoff. +- Each person vector stores the last 2 seconds of binary activity followed by + a visibility/availability mask. Activity is exposed only in bins where that + participant was visible according to the provided boxes. No future bin, + future wearer state, transcript text or `Target_of_Speech` value is input. +- Only non-wearer participant IDs present in the latest provided-box frame + strictly before the cutoff receive slots. IDs route annotations during + preparation but are not exposed as embeddings to the head. +- Transcript-derived wearer/others activity must match `target_perframe` by + default. Any mismatch fails closed instead of being silently tolerated. +- Oracle sidecars use `easycom-oracle-person-history-v1` and are accepted only + by the explicit `oracle-frame-set-attention` mode. Normal local-feature + loading rejects them because they use speech labels. + +## Conditions + +| Condition | Information exposed | Purpose | +|---|---|---| +| `audio` | Omni `G` plus the existing recent local audio `A` | deployable baseline | +| `visibility-only` | current visible-person set, but all activity bits are zero | parameter/person-count control | +| `aggregate-visible` | one OR-pooled history over currently visible people | aggregate floor-information upper bound | +| `per-person` | one correct activity history per currently visible participant | main oracle information test | +| `shuffled-id` | preserves the number active at every lag but independently reassigns activity between people | destroys identity continuity | +| `delayed` | correct per-person histories delayed by 1 second | destroys current temporal alignment without future leakage | + +The decisive comparison is `per-person` against `aggregate-visible`, +`shuffled-id` and `delayed`, not only against `audio`. A gain over `audio` alone +could come from perfect aggregate VAD rather than person continuity. + +## Remote preparation and one-seed Session-12 screen + +The source local sidecars provide only mixed audio and the visibility/box +timeline. Their learned visual vectors are not copied into the oracle input. + +```bash +cd /data1/yfzx/wangx/Code/icassp2027-experiment + +export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom +export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack +export TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame +export LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576 +export ORACLE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/oracle_person_history_2s +export CHECKPOINT_ROOT=checkpoints/easycom_oracle_information_4ep +export RESULT_ROOT=results/easycom_oracle_information_4ep + +CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh all 0 +``` + +`all` prepares the five oracle conditions, trains on Sessions 4--11, selects +the checkpoint on Session 12, runs Session-12 inference and prints the standard +dense AP/mAP screen. It deliberately does not run Sessions 1--3. + +If preparation reports any transcript/target mismatch, stop and inspect it. +Do not raise `--max-label-mismatch-rows` merely to make the run continue. + +## Follow-up + +If `per-person` is consistently better on Session 12, add seeds 1 and 2 using +the already prepared sidecars: + +```bash +CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh screen 1 2 +``` + +Only after freezing the method and event evaluator should selected variants be +run once on Sessions 1--3: + +```bash +VARIANTS="audio aggregate-visible per-person shuffled-id delayed" \ +CUDA_VISIBLE_DEVICES=0 \ +bash run_easycom_oracle_information.sh test 0 1 2 +``` + +## Interpretation + +- `per-person > aggregate-visible` and `per-person > shuffled-id`: speaker + continuity has information value; proceed to learned causal ASD/tracks. +- `aggregate-visible > audio`, but no per-person advantage: perfect aggregate + activity helps, but the current target does not justify an identity claim. +- oracle conditions do not improve: do not spend the main GPU budget on a more + complex person encoder until the task/head/evaluator is reconsidered. +- oracle improves but learned Qwen face features do not: this supports a + speaker-grounding bottleneck and motivates EgoASD-like AV evidence. + +Dense 10-horizon mAP is only a screening signal. The publication claim still +requires the frozen competing-risk-censored entry/yield evaluator and matched +wearer-onset-recall false-entry analysis. diff --git a/docs/easycom_track_gru_residual.md b/docs/easycom_track_gru_residual.md index 483cb30..21f3a2d 100644 --- a/docs/easycom_track_gru_residual.md +++ b/docs/easycom_track_gru_residual.md @@ -62,8 +62,14 @@ re-extraction. Runtime status (2026-09-07): the focused causal-history, exact-zero-gate and train/infer round-trip tests passed in 49.699 seconds on the remote PyTorch/L20 -environment. This verifies the code path, not model quality. The one-session -GPU training smoke and full experiment remain unmeasured. +environment. The subsequent matched four-epoch screen produced no meaningful +change: Session-12 three-seed mean-mAP delta was `+0.01306 +/- 0.02368` +percentage points, and the seed-0 Sessions-1--3 delta was `-0.00030` points. +The selected residual gates remained between `-0.0015` and `+0.0108`. Because +the zero scalar gate blocks the track branch's gradient at initialization, this +run is classified as an optimization failure, not a clean negative result for +track continuity. Exact values and caveats are archived in +[`../results/easycom_track_gru_residual_4ep_2026-09-07/README.md`](../results/easycom_track_gru_residual_4ep_2026-09-07/README.md). ## Fair initialization @@ -73,6 +79,14 @@ feature contract before copying the pooled path. By default those copied parameters are frozen. The residual gate starts at zero, so before optimization the track model must reproduce the pooled logits exactly, element for element. +The measured screen exposed a defect in that parameterization. At a zero gate, +the first-step gradient of every residual-branch parameter is also zero; only +the scalar gate can move, and later branch gradients remain multiplied by its +small value. A corrected follow-up should instead zero-initialize the final +residual output layer and remove the scalar multiplier. That preserves exact +pooled logits at initialization while allowing the output layer to receive a +gradient immediately. + The matched screening protocol is: - EasyCom Session 4--11: training; diff --git a/docs/progressive_feature_fusion_route_2026-09-02.md b/docs/progressive_feature_fusion_route_2026-09-02.md index 04b85df..552211b 100644 --- a/docs/progressive_feature_fusion_route_2026-09-02.md +++ b/docs/progressive_feature_fusion_route_2026-09-02.md @@ -204,6 +204,13 @@ track GRU、G+A 对 track states 的 masked attention 以及零初始化 scalar 实现、契约和命令见 [`easycom_track_gru_residual.md`](easycom_track_gru_residual.md)。 +2026-09-07 的实测暴露了这个第一版参数化的优化缺陷:Session 12 三个 seed 的 +ten-horizon mean mAP 配对增量仅为 `+0.01306 +/- 0.02368` 个百分点,保存 checkpoint 的 +`tanh(gate)` 只有 `-0.0015` 到 `+0.0108`。零 gate 在第一步也会把 residual branch 的参数 +梯度乘为零,因此该结果不能用于否定 track continuity。下一版应删除 scalar gate、将 +residual 最后输出层零初始化,并在 dev 上记录 delta-logit、概率变化和非零梯度;详见 +[`结果记录`](../results/easycom_track_gru_residual_4ep_2026-09-07/README.md)。 + ## 8. F4:Person-conditioned audio/Omni history F4 让每个匿名人物分别读取已有的 5 Hz 因果历史: @@ -279,11 +286,14 @@ README 必须写明 split、seed、checkpoint、输入特征版本、评测协 ## 12. 最近执行顺序 -1. 用户当前选择先做 dense-AP model screen;运行 matched pooled 与 - `track-gru-residual`,固定 Session 12 dev,不用测试集选历史长度。 -2. 若 track GRU 有稳定增益,补 broken-track、past-shift、no-ID 和 remove-person 控制;若 - 无增益,只在 dev 上比较 8/16/32 observations 后停止该分支。 -3. correct track 通过控制后,再实现 F4 person-conditioned audio/Omni history;否则回到 +1. 第一版 matched pooled/`track-gru-residual` dense-AP screen 已完成,但因 zero-gate + gradient blockage 判为优化失败,不是 track continuity 的有效阴性结果。 +2. 先用零初始化 residual 输出层修正梯度路径,只在 Session 12 做 seed-0 诊断;确认分支 + 实际改变 logits 后再跑三 seed。若修正后仍无增益,再决定是否比较 8/16/32 + observations;不要根据已经查看过的 Sessions 1--3 seed-0 结果调参。 +3. 若修正后的 track GRU 有稳定增益,补 broken-track、past-shift、no-ID 和 remove-person + 控制;若仍无增益则停止该分支。 +4. correct track 通过控制后,再实现 F4 person-conditioned audio/Omni history;否则回到 对齐审计,不直接堆更深 interaction。 -4. 在形成论文事件结论前,仍须完成 P0 competing-risk censoring,并人工打印 response 与 +5. 在形成论文事件结论前,仍须完成 P0 competing-risk censoring,并人工打印 response 与 yield 时间轴。当前事件结果只能称为 timing-derived proxy。 diff --git a/easycom_head.py b/easycom_head.py index 32f0a88..3b04441 100644 --- a/easycom_head.py +++ b/easycom_head.py @@ -63,6 +63,7 @@ LAYER_FUSION_MODES = ( ) CURRENT_FRAME_PERSON_SET_FUSION_MODES = ( "frame-set-attention", + "oracle-frame-set-attention", "audio-person-frame-set", "gated-person-state-adapter", ) @@ -75,6 +76,7 @@ LOCAL_FUSION_MODES = ("pooled", *PERSON_SET_FUSION_MODES) LOCAL_FUSION_SCOPES = { "pooled": "pooled-baseline", "frame-set-attention": "f1-current-frame-person-set", + "oracle-frame-set-attention": "causal-oracle-visible-person-speaking-history", "audio-person-frame-set": "f5-screening-current-frame-no-person-history", "gated-person-state-adapter": "f6-screening-post-thinker-not-internal", "track-gru-residual": "causal-anonymous-track-gru-pooled-baseline-residual", @@ -1303,6 +1305,7 @@ def load_session_data( sample_id=name, dataset=dataset, expected_contract=local_contract, + oracle_analysis=local_fusion == "oracle-frame-set-attention", ) else: from local_turn_features import load_local_features diff --git a/literature_notes/54_egoasd.md b/literature_notes/54_egoasd.md new file mode 100644 index 0000000..b887872 --- /dev/null +++ b/literature_notes/54_egoasd.md @@ -0,0 +1,26 @@ +# Advancing Active Speaker Detection for Egocentric Videos(EgoASD) + +- 作者:Jaesung Huh、Juan Azcarreta Ortiz、Anurag Kumar、Ashutosh Pandey、Ali Aroudi、Daniel D. E. Wong、Francesco Nesta、Buye Xu、Jacob Donley +- 发表:ICASSP 2025 +- 阅读深度:摘要、方法贡献和 EasyCom 结果筛选;未运行 +- 记录日期:2026-09-07 +- 来源:[IEEE Xplore](https://ieeexplore.ieee.org/document/10888166/)、[DOI](https://doi.org/10.1109/ICASSP49660.2025.10888166) + +## 摘要与项目关系 + +该工作针对第一视角视频中快速头动、运动模糊和视觉质量不稳定的问题,提出固定嘴部区域与 +motion-blur augmentation,并将它们用于一个轻量、因果的音视频 active speaker detector。 +论文报告 EgoASD 在 EasyCom 上比此前结果提高 1.7 个 mAP 点,同时模型规模约小 2.5 倍, +并进一步用于第一视角音视频 diarization。 + +它直接说明:EasyCom 的逐人视觉分支不能只依赖通用图像语义或稀疏静态人脸 embedding;嘴部 +局部运动、短时音唇同步、运动模糊鲁棒性和因果 temporal context 都是更合理的 active-speaker +证据。若本项目最终声称人物视觉 grounding 有效,应把 EgoASD 或同等级 causal ASD feature +作为专用视觉前端基线/候选,而不能只和普通 Qwen face-crop pooling 比较。 + +## 适用边界 + +EgoASD 预测画面中候选人物当前是否发声,不预测不可见 wearer 未来是否进入或让出 floor。 +因此它可以提供逐人 active-speaker evidence 或 oracle 的可学习替代,但不能直接替代本项目的 +wearer floor-action head 和事件级评测。论文报告的 ASD mAP 也不能与本项目 0.2--2.0 秒未来 +活动 mAP 直接比较。 diff --git a/literature_notes/55_speaker_embedding_egocentric_asd.md b/literature_notes/55_speaker_embedding_egocentric_asd.md new file mode 100644 index 0000000..63c1441 --- /dev/null +++ b/literature_notes/55_speaker_embedding_egocentric_asd.md @@ -0,0 +1,28 @@ +# Speaker Embedding Informed Audiovisual Active Speaker Detection for Egocentric Recordings + +- 作者:Jason Clarke、Yoshihiko Gotoh、Stefan Goetze +- 发表:ICASSP 2025 +- 阅读深度:摘要、方法和实验设置重点阅读;未运行 +- 记录日期:2026-09-07 +- 来源:[IEEE Xplore](https://ieeexplore.ieee.org/document/10890414/)、[arXiv](https://arxiv.org/abs/2502.06012)、[作者公开论文](https://eprints.whiterose.ac.uk/id/eprint/223485/1/Jason_ICASSP_2025.pdf) + +## 摘要与项目关系 + +该工作研究第一视角场景中候选人物的音视频 active speaker detection。作者提出 Speaker +Comparison Auxiliary Network(SCAN):除候选人的参考语音 embedding 外,还从当前混合 +音频的滑动窗口提取 speaker embedding,再通过 cross-attention 比较当前声音和已登记身份。 +论文同时提出自监督的 video face recognition 调优方法,用于构建更稳健的 face--speaker +身份库。SCAN 接入 TalkNet 和 Light-ASD 后,在 Ego4D-AVD 上分别报告 14.5% 和 10.3% 的 +相对 mAP 提升。 + +它说明显式 speaker reference/identity memory 在视觉遮挡或嘴部线索不可用时已有明确先例。 +因此本项目不应把“用 track identity 帮助音视频说话者判断”本身作为 novelty。更合适的限定是: +人物轨迹保持匿名,不依赖预登记语音或长期身份库,只使用截止当前时刻可获得的因果轨迹, +目标是预测不可见 wearer 的未来 entry/yield 行为。 + +## 适用边界 + +SCAN 需要候选人的预分离参考语音并建立 face--speaker library;这与本项目计划的在线、匿名、 +无 enrollment 场景不同。它仍然做当前逐人物 ASD,不评估 wearer 的未来错误开口、响应延迟 +或 yield。若以后引入 speaker embedding,必须单列 enrollment 依赖,并确保参考语音只来自 +预测截止时间之前,不能从整段视频或测试片段未来部分建立身份库。 diff --git a/literature_notes/README.md b/literature_notes/README.md index d0743ec..5bea929 100644 --- a/literature_notes/README.md +++ b/literature_notes/README.md @@ -2,7 +2,13 @@ 本目录持久化记录我们在本项目讨论中实际阅读、分析或筛选过的论文。这里的“摘要”均为面向当前课题的中文转述,不是原文摘要的复制。 -最后更新:2026-08-30 +最后更新:2026-09-07 + +最新增补:两篇 ICASSP 2025 第一视角 ASD 工作进一步约束当前视觉方案。 +[EgoASD](54_egoasd.md)表明 EasyCom 局部视觉需要认真处理嘴部运动、运动模糊和因果时序, +而不只是静态人脸语义;[speaker-embedding-informed ASD](55_speaker_embedding_egocentric_asd.md) +说明显式身份/参考语音 memory 已有先例。当前更可辩护的方向是匿名、无需 enrollment 的逐人 +因果证据,并验证它是否改变不可见 wearer 的未来 entry/yield 行为。 最新专题:[多方第一视角 AV wearer floor-state prediction 文献 gap 审计](../docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md)。结论是宽泛组件均有先例;仍可信的交集是单 wearer 严格因果流、不可见 wearer 的 entry/yield、逐人 grounding 和事件级评测。MuVAP、EgoSpeak、3M-Transformer、AV-CONV 与在线多方 EOT 必须正面区分。 @@ -75,6 +81,8 @@ | 51 | [Online Multimodal End-of-Turn Prediction](51_online_multiparty_eot.md) | 2024 | 任务、采集、方法和局限重点阅读 | ICMI;在线三方 EOT,但依赖实验室 gaze/mocap/独立麦克风 | | 52 | [TurnBench](52_turnbench.md) | 2026 | 任务、因果提交和 leaderboard 重点阅读 | 双人音频事件评测;recall/FPR/latency 范式 | | 53 | [AMusE](53_amuse.md) | 2026 | 摘要及官方论文页筛选 | CVPR;MLLM 多说话人身份、角色和 AV grounding 短板 | +| 54 | [Advancing Active Speaker Detection for Egocentric Videos(EgoASD)](54_egoasd.md) | 2025 | 摘要、方法贡献和 EasyCom 结果筛选 | ICASSP;因果 egocentric ASD、固定嘴部区域及运动模糊增强 | +| 55 | [Speaker Embedding Informed Audiovisual Active Speaker Detection](55_speaker_embedding_egocentric_asd.md) | 2025 | 摘要、方法和实验设置重点阅读 | ICASSP;参考语音身份、SCAN 与第一视角 ASD,约束 identity-memory novelty | ## 当前综合判断 diff --git a/local_turn_features.py b/local_turn_features.py index b020592..d357ad1 100644 --- a/local_turn_features.py +++ b/local_turn_features.py @@ -14,6 +14,7 @@ from typing import Any import torch LOCAL_FORMAT = "qwen-local-pooled-v1" +ORACLE_FORMAT = "easycom-oracle-person-history-v1" MODALITIES = {"visual": ("visual",), "audio": ("audio",), "av": ("visual", "audio")} @@ -277,12 +278,14 @@ def _load_validated_payload( dataset: str, modalities: str, expected_contract: dict[str, Any] | None, + oracle_analysis: bool = False, ) -> tuple[torch.Tensor, torch.Tensor, dict[str, Any], dict[str, Any], dict[str, Any]]: if modalities not in MODALITIES: raise ValueError(f"Unknown local modalities: {modalities}") times, indices = reference_alignment(reference) payload = torch.load(path, map_location="cpu", weights_only=False) - if not isinstance(payload, dict) or payload.get("format") != LOCAL_FORMAT: + expected_format = ORACLE_FORMAT if oracle_analysis else LOCAL_FORMAT + if not isinstance(payload, dict) or payload.get("format") != expected_format: raise ValueError(f"Invalid local feature format: {path}") if payload.get("sample_id") != sample_id or payload.get("dataset") != dataset: raise ValueError(f"Local sample/dataset mismatch: {path}") @@ -295,9 +298,22 @@ def _load_validated_payload( if local_times.shape != times.shape or not torch.allclose(local_times, times, atol=1e-7, rtol=0): raise ValueError(f"Local times mismatch: {path}") config = payload.get("config", {}) - if config.get("causal_policy") != "half-open-trailing-window" or config.get("uses_speech_labels") is not False: + if oracle_analysis: + if ( + dataset != "easycom" + or config.get("causal_policy") != "fully-observed-half-open-5hz-bins" + or config.get("uses_speech_labels") is not True + or config.get("oracle_analysis") is not True + ): + raise ValueError( + f"Oracle features lack the explicit causal/oracle contract: {path}" + ) + elif ( + config.get("causal_policy") != "half-open-trailing-window" + or config.get("uses_speech_labels") is not False + ): raise ValueError(f"Local features lack the causal/no-speech-label contract: {path}") - contract = {"format": LOCAL_FORMAT, "modalities": modalities, "config": config} + contract = {"format": expected_format, "modalities": modalities, "config": config} if expected_contract is not None and contract != expected_contract: raise ValueError(f"Local encoder/pooling contract differs across sessions/checkpoint: {path}") return times, indices, payload, config, contract @@ -348,6 +364,7 @@ def load_frame_set_features( dataset: str, expected_contract: dict[str, Any] | None = None, track_max_gap_seconds: float | None = None, + oracle_analysis: bool = False, ) -> tuple[torch.Tensor, FramePersonFeatures, dict[str, Any]]: """Load direct audio and the latest causal per-frame person set per 5 Hz row.""" @@ -358,6 +375,7 @@ def load_frame_set_features( dataset=dataset, modalities="av", expected_contract=expected_contract, + oracle_analysis=oracle_analysis, ) audio = payload.get("audio") audio_valid = payload.get("audio_valid") diff --git a/prepare_easycom_oracle_features.py b/prepare_easycom_oracle_features.py new file mode 100644 index 0000000..6e59099 --- /dev/null +++ b/prepare_easycom_oracle_features.py @@ -0,0 +1,551 @@ +#!/usr/bin/env python3 +"""Build explicit, causal EasyCom oracle-person sidecars for information tests. + +This is not a deployable feature extractor. It combines the already extracted +recent mixed-audio vector with ground-truth, participant-resolved speech +activity from EasyCom transcripts. Only fully observed 200 ms bins ending no +later than each state cutoff are exposed. The output format is deliberately +rejected by ordinary local-feature loaders. +""" + +from __future__ import annotations + +import argparse +import os +import tempfile +from pathlib import Path +from typing import Any, Sequence + +import numpy as np +import torch + +from derive_easycom_interaction_events import ( + CAMERA_WEARER_ID, + FRAMES_PER_TASK_STEP, + Rules, + build_clips, + load_utterances, +) +from easycom_head import ( + NUM_CLASSES, + STEP_SECONDS, + normalize_sample_name, + resolve_feature_path, +) +from easycom_manifest import ALL_SESSIONS +from local_turn_features import ( + ORACLE_FORMAT, + load_frame_set_features, + reference_alignment, +) + + +ORACLE_CONDITIONS = ( + "visibility-only", + "aggregate-visible", + "per-person", + "shuffled-id", + "delayed", +) +UNKNOWN_PERSON_IDS = {"", "-1", "none", "null", "unknown"} + + +def exact_steps(seconds: float, name: str) -> int: + if not np.isfinite(seconds) or seconds <= 0: + raise ValueError(f"{name} must be finite and positive") + steps = round(seconds / STEP_SECONDS) + if not np.isclose(seconds, steps * STEP_SECONDS, atol=1e-9, rtol=0): + raise ValueError(f"{name} must be a multiple of {STEP_SECONDS:g}s") + return int(steps) + + +def participant_activity( + utterances: Sequence[Any], + *, + num_steps: int, + wearer_id: int, +) -> tuple[dict[int, torch.BoolTensor], torch.BoolTensor, dict[str, int]]: + """Rasterize inclusive 20 Hz transcript spans into observed 5 Hz bins.""" + + activity: dict[int, torch.BoolTensor] = {} + wearer = torch.zeros(num_steps, dtype=torch.bool) + discarded_terminal_frames = 0 + grid_frames = num_steps * FRAMES_PER_TASK_STEP + for row in utterances: + dense_start = int(row.dense_start_frame) + dense_end = int(row.dense_end_frame_exclusive) + if dense_end > grid_frames: + overrun = dense_end - grid_frames + if overrun > FRAMES_PER_TASK_STEP - 1: + raise ValueError( + f"Transcript exceeds the discarded terminal sub-step by " + f"{overrun} frames: {row.uid}" + ) + discarded_terminal_frames += max( + 0, dense_end - max(dense_start, grid_frames) + ) + dense_end = grid_frames + if dense_start >= dense_end: + continue + first = dense_start // FRAMES_PER_TASK_STEP + last = (dense_end - 1) // FRAMES_PER_TASK_STEP + if first < 0 or last < first or last >= num_steps: + raise ValueError( + f"Transcript interval leaves target grid: {row.uid}, " + f"bins=[{first},{last}], target_steps={num_steps}" + ) + destination = ( + wearer + if int(row.participant_id) == wearer_id + else activity.setdefault( + int(row.participant_id), + torch.zeros(num_steps, dtype=torch.bool), + ) + ) + destination[first : last + 1] = True + return activity, wearer, { + "discarded_terminal_transcript_frames": discarded_terminal_frames + } + + +def causal_history( + sequence: torch.BoolTensor, + *, + row: int, + history_steps: int, + delay_steps: int = 0, + observed_sequence: torch.BoolTensor | None = None, +) -> torch.Tensor: + """Return [activity, observed-mask] ending at row-delay, never after row.""" + + if sequence.ndim != 1 or sequence.dtype != torch.bool: + raise ValueError("Activity sequence must be bool [T]") + if observed_sequence is not None and ( + observed_sequence.dtype != torch.bool + or observed_sequence.shape != sequence.shape + ): + raise ValueError("Observed sequence must be bool [T] aligned to activity") + if not 0 <= row < len(sequence): + raise IndexError(row) + if history_steps <= 0 or delay_steps < 0: + raise ValueError("Invalid history/delay") + end = row - delay_steps + values = torch.zeros(history_steps, dtype=torch.float32) + observed = torch.zeros(history_steps, dtype=torch.float32) + if end >= 0: + start = max(0, end - history_steps + 1) + count = end - start + 1 + available = ( + torch.ones(count, dtype=torch.bool) + if observed_sequence is None + else observed_sequence[start : end + 1] + ) + values[-count:] = (sequence[start : end + 1] & available).float() + observed[-count:] = available.float() + return torch.cat((values, observed), dim=0) + + +def visible_people_by_row( + source_payload: dict[str, Any], + cutoffs: torch.Tensor, + *, + wearer_id: int, +) -> tuple[list[list[int]], list[list[dict[str, Any]]]]: + """Use the latest provided-box frame strictly before every state cutoff.""" + + persons = source_payload["persons"] + frame_times = torch.as_tensor(persons["frame_times"], dtype=torch.float64) + offsets = torch.as_tensor(persons["frame_offsets"], dtype=torch.long) + records = persons["records"] + row_to_frame = torch.searchsorted(frame_times, cutoffs, right=False) - 1 + ids_by_row: list[list[int]] = [] + records_by_row: list[list[dict[str, Any]]] = [] + for frame_index in row_to_frame.tolist(): + if frame_index < 0: + ids_by_row.append([]) + records_by_row.append([]) + continue + start = int(offsets[frame_index]) + stop = int(offsets[frame_index + 1]) + seen: set[int] = set() + row_ids: list[int] = [] + row_records: list[dict[str, Any]] = [] + for source_record in records[start:stop]: + raw_id = source_record.get("id") + text_id = "" if raw_id is None else str(raw_id).strip() + if text_id.lower() in UNKNOWN_PERSON_IDS: + continue + try: + participant_id = int(text_id) + except ValueError as error: + raise ValueError(f"Non-numeric EasyCom participant ID {text_id!r}") from error + if participant_id == wearer_id or participant_id in seen: + continue + seen.add(participant_id) + row_ids.append(participant_id) + row_records.append(source_record) + ids_by_row.append(row_ids) + records_by_row.append(row_records) + return ids_by_row, records_by_row + + +def oracle_person_rows( + activity: dict[int, torch.BoolTensor], + visibility: dict[int, torch.BoolTensor], + visible_ids: list[list[int]], + source_records: list[list[dict[str, Any]]], + *, + condition: str, + history_steps: int, + delay_steps: int, + seed: int, +) -> tuple[torch.Tensor, torch.LongTensor, list[dict[str, Any]], dict[str, int]]: + """Create a 5 Hz ragged set of causal oracle histories.""" + + if condition not in ORACLE_CONDITIONS: + raise ValueError(f"Unknown oracle condition: {condition}") + if len(visible_ids) != len(source_records): + raise ValueError("Visible ID/record row counts differ") + num_steps = len(visible_ids) + zero_sequence = torch.zeros(num_steps, dtype=torch.bool) + rng = np.random.default_rng(seed) + feature_rows: list[torch.Tensor] = [] + records: list[dict[str, Any]] = [] + offsets = [0] + shuffled_values_changed = 0 + shuffled_values_total = 0 + + for row, (person_ids, row_records) in enumerate(zip(visible_ids, source_records)): + if len(person_ids) != len(row_records): + raise ValueError("Visible person IDs/records differ within a row") + base = torch.stack( + [ + causal_history( + activity.get(person_id, zero_sequence), + row=row, + history_steps=history_steps, + delay_steps=delay_steps if condition == "delayed" else 0, + observed_sequence=visibility.get(person_id, zero_sequence), + ) + for person_id in person_ids + ], + dim=0, + ) if person_ids else torch.empty((0, 2 * history_steps)) + + if condition == "visibility-only": + output = base.clone() + output[:, :history_steps] = 0 + elif condition == "aggregate-visible": + if len(base): + output = torch.cat( + ( + base[:, :history_steps].amax(dim=0, keepdim=True), + base[:, history_steps:].amax(dim=0, keepdim=True), + ), + dim=-1, + ) + else: + output = base + elif condition == "shuffled-id": + output = base.clone() + # Independently reassign each historical time bin across the + # currently visible slots. Per-bin activity counts remain exact, + # while cross-time person continuity is destroyed. + for column in range(history_steps): + if len(base) > 1: + permutation = torch.from_numpy(rng.permutation(len(base))) + output[:, column] = base[permutation, column] + output[:, history_steps + column] = base[ + permutation, history_steps + column + ] + shuffled_values_changed += int( + (output[:, :history_steps] != base[:, :history_steps]).sum() + ) + shuffled_values_total += int(base[:, :history_steps].numel()) + else: + output = base + + feature_rows.extend(output.unbind(dim=0)) + if condition == "aggregate-visible" and person_ids: + records.append( + { + "id": "aggregate-visible", + "source_person_ids": person_ids, + "source_visual_times": [ + float(item.get("source_time", float("nan"))) + for item in row_records + ], + } + ) + else: + for person_id, source_record in zip(person_ids, row_records): + records.append( + { + "id": str(person_id), + "source_clip_index": int( + source_record.get("source_clip_index", 0) + ), + "source_visual_time": float( + source_record.get("source_time", float("nan")) + ), + } + ) + offsets.append(offsets[-1] + len(output)) + + feature_dim = 2 * history_steps + features = ( + torch.stack(feature_rows) + if feature_rows + else torch.empty((0, feature_dim), dtype=torch.float32) + ) + return ( + features, + torch.tensor(offsets, dtype=torch.long), + records, + { + "shuffled_values_changed": shuffled_values_changed, + "shuffled_values_total": shuffled_values_total, + }, + ) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--dataset-root", type=Path, required=True) + parser.add_argument("--feature-root", type=Path, required=True) + parser.add_argument("--target-root", type=Path, required=True) + parser.add_argument("--local-feature-root", type=Path, required=True) + parser.add_argument("--output-root", type=Path, required=True) + parser.add_argument("--sessions", nargs="+", default=list(ALL_SESSIONS)) + parser.add_argument("--conditions", nargs="+", choices=ORACLE_CONDITIONS, + default=list(ORACLE_CONDITIONS)) + parser.add_argument("--history-seconds", type=float, default=2.0) + parser.add_argument("--delay-seconds", type=float, default=1.0) + parser.add_argument("--camera-wearer-id", type=int, default=CAMERA_WEARER_ID) + parser.add_argument("--shuffle-seed", type=int, default=2027) + parser.add_argument("--max-label-mismatch-rows", type=int, default=0) + parser.add_argument("--expected-mode", default="open-stream-replay") + parser.add_argument("--overwrite", action="store_true") + parser.add_argument("--dry-run", action="store_true") + return parser.parse_args() + + +def main() -> None: + args = parse_args() + history_steps = exact_steps(args.history_seconds, "--history-seconds") + delay_steps = exact_steps(args.delay_seconds, "--delay-seconds") + if args.max_label_mismatch_rows < 0: + raise ValueError("--max-label-mismatch-rows cannot be negative") + sessions = [normalize_sample_name(value, "easycom") for value in args.sessions] + unknown = sorted(set(sessions) - set(ALL_SESSIONS)) + if unknown: + raise ValueError(f"Unknown EasyCom sessions: {unknown}") + if len(sessions) != len(set(sessions)): + raise ValueError("Duplicate sessions") + conditions = list(dict.fromkeys(args.conditions)) + rules = Rules() + + print( + f"Oracle analysis only: conditions={','.join(conditions)}, " + f"history={history_steps}x{STEP_SECONDS:g}s, " + f"delay={delay_steps}x{STEP_SECONDS:g}s" + ) + for session_position, session in enumerate(sessions, 1): + reference_path = resolve_feature_path(args.feature_root, session) + target_path = args.target_root / f"{session}.npy" + source_local_path = resolve_feature_path(args.local_feature_root, session) + reference = torch.load(reference_path, map_location="cpu", weights_only=False) + times, label_indices = reference_alignment(reference) + if args.expected_mode != "any" and reference.get("mode") != args.expected_mode: + raise ValueError( + f"{reference_path} mode is {reference.get('mode')!r}, " + f"expected {args.expected_mode!r}" + ) + target = np.load(target_path, allow_pickle=False) + if target.shape != (len(times), NUM_CLASSES): + raise ValueError( + f"Target/reference mismatch: {target_path}, " + f"{target.shape} vs {(len(times), NUM_CLASSES)}" + ) + + direct_audio, _source_people, source_contract = load_frame_set_features( + source_local_path, + reference, + sample_id=session, + dataset="easycom", + ) + source_payload = torch.load( + source_local_path, map_location="cpu", weights_only=False + ) + audio_dim = int(source_contract["config"]["audio_dim"]) + audio = direct_audio[:, :audio_dim] + audio_valid = direct_audio[:, audio_dim].bool() + + clips = build_clips(args.dataset_root.resolve(), session, rules) + utterances = load_utterances(session, clips) + activity, wearer_activity, transcript_audit = participant_activity( + utterances, + num_steps=len(times), + wearer_id=args.camera_wearer_id, + ) + all_other_activity = torch.zeros(len(times), dtype=torch.bool) + for sequence in activity.values(): + all_other_activity |= sequence + target_tensor = torch.from_numpy(np.asarray(target)) + wearer_mismatch = int( + (wearer_activity != target_tensor[:, 1].bool()).sum() + ) + other_mismatch = int( + (all_other_activity != target_tensor[:, 2].bool()).sum() + ) + mismatch_union = int( + ( + (wearer_activity != target_tensor[:, 1].bool()) + | (all_other_activity != target_tensor[:, 2].bool()) + ).sum() + ) + if mismatch_union > args.max_label_mismatch_rows: + raise ValueError( + f"{session}: transcript/target activity differs on " + f"{mismatch_union} rows (wearer={wearer_mismatch}, " + f"others={other_mismatch}); inspect alignment or explicitly " + "raise --max-label-mismatch-rows" + ) + + visible_ids, source_records = visible_people_by_row( + source_payload, + times, + wearer_id=args.camera_wearer_id, + ) + visibility: dict[int, torch.BoolTensor] = { + participant_id: torch.zeros(len(times), dtype=torch.bool) + for participant_id in activity + } + for row, participant_ids in enumerate(visible_ids): + for participant_id in participant_ids: + visibility.setdefault( + participant_id, torch.zeros(len(times), dtype=torch.bool) + )[row] = True + visible_counts = [len(row) for row in visible_ids] + print( + f"[{session_position}/{len(sessions)}] {session}: " + f"states={len(times)}, participants={len(activity)}, " + f"visible_rows={sum(count > 0 for count in visible_counts)}, " + f"max_visible={max(visible_counts, default=0)}, " + f"target_mismatch={mismatch_union}" + ) + + for condition in conditions: + destination = args.output_root / condition / f"{session}.pt" + if destination.exists() and not args.overwrite: + raise FileExistsError( + f"Refusing to overwrite {destination}; pass --overwrite" + ) + condition_seed = ( + args.shuffle_seed + + 1009 * session_position + + ORACLE_CONDITIONS.index(condition) + ) + features, frame_offsets, records, corruption = oracle_person_rows( + activity, + visibility, + visible_ids, + source_records, + condition=condition, + history_steps=history_steps, + delay_steps=delay_steps, + seed=condition_seed, + ) + visual = torch.zeros((len(times), 2 * history_steps)) + visual_valid = torch.zeros(len(times), dtype=torch.bool) + for row in range(len(times)): + start = int(frame_offsets[row]) + stop = int(frame_offsets[row + 1]) + if stop > start: + visual[row] = features[start:stop].mean(dim=0) + visual_valid[row] = True + config = { + "oracle_analysis": True, + "uses_speech_labels": True, + "causal_policy": "fully-observed-half-open-5hz-bins", + "oracle_condition": condition, + "oracle_scope": "currently-visible-nonwearer-participants", + "history_steps": history_steps, + "history_seconds": history_steps * STEP_SECONDS, + "delay_steps": delay_steps if condition == "delayed" else 0, + "delay_seconds": ( + delay_steps * STEP_SECONDS if condition == "delayed" else 0.0 + ), + "shuffle_seed": args.shuffle_seed if condition == "shuffled-id" else None, + "speaker_activity_source": "EasyCom Speech_Transcriptions inclusive 20Hz intervals", + "visibility_source": "provided face boxes in source local sidecar", + "camera_wearer_id": args.camera_wearer_id, + "visual_dim": 2 * history_steps, + "audio_dim": audio_dim, + "source_local_contract": source_contract, + } + payload = { + "format": ORACLE_FORMAT, + "dataset": "easycom", + "sample_id": session, + "times": times, + "label_indices": label_indices, + "partial": False, + "visual": visual, + "audio": audio, + "visual_valid": visual_valid, + "audio_valid": audio_valid, + "persons": { + "features": features, + # Bin k covers [k/5,(k+1)/5); its feature is available at + # the matching state cutoff (k+1)/5. + "frame_times": torch.arange(len(times), dtype=torch.float64) + * STEP_SECONDS, + "frame_offsets": frame_offsets, + "records": records, + }, + "config": config, + "reference_features": str(reference_path.resolve()), + "source_local_features": str(source_local_path.resolve()), + "audit": { + "strictly_past_or_current_bins_only": True, + "transcript_target_mismatch_rows": mismatch_union, + "wearer_mismatch_rows": wearer_mismatch, + "others_mismatch_rows": other_mismatch, + "rows_with_visible_people": sum( + count > 0 for count in visible_counts + ), + "max_visible_people": max(visible_counts, default=0), + "person_observations": len(features), + "generation_seed": condition_seed, + **transcript_audit, + **corruption, + }, + } + if args.dry_run: + print( + f" {condition}: would write {destination}; " + f"observations={len(features)}" + ) + continue + destination.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + dir=destination.parent, + prefix=f".{session}.", + suffix=".tmp", + delete=False, + ) as handle: + temporary = Path(handle.name) + try: + torch.save(payload, temporary) + os.replace(temporary, destination) + finally: + temporary.unlink(missing_ok=True) + print( + f" {condition}: saved {destination}; " + f"observations={len(features)}" + ) + + +if __name__ == "__main__": + main() diff --git a/results/README.md b/results/README.md index 1e7c06d..2af7524 100644 --- a/results/README.md +++ b/results/README.md @@ -14,6 +14,7 @@ readers to the right comparison. | [`easycom_loss_structure_4ep_2026-08-26/`](easycom_loss_structure_4ep_2026-08-26/README.md) | LSTR-MCE versus independent BCE versus four-state CE | | [`easycom_transition_auxiliary_4ep_2026-08-27/`](easycom_transition_auxiliary_4ep_2026-08-27/README.md) | Supervised transition head and expected-reward objectives | | [`easycom_encoder_only_1s_4ep_2026-08-28/`](easycom_encoder_only_1s_4ep_2026-08-28/README.md) | Encoder-only A/V/AV versus global-Thinker local-feature baselines | +| [`easycom_track_gru_residual_4ep_2026-09-07/`](easycom_track_gru_residual_4ep_2026-09-07/README.md) | Matched pooled/anonymous-track-GRU screen and near-zero residual-gate diagnosis | ## Interpretation rules diff --git a/results/easycom_track_gru_residual_4ep_2026-09-07/README.md b/results/easycom_track_gru_residual_4ep_2026-09-07/README.md new file mode 100644 index 0000000..4dc8f30 --- /dev/null +++ b/results/easycom_track_gru_residual_4ep_2026-09-07/README.md @@ -0,0 +1,116 @@ +# EasyCom anonymous track-GRU residual: matched four-epoch screen + +## Question + +Does causal continuity within anonymous face tracks improve future floor-state +prediction beyond the same-seed `G+A+pooled-V` head? + +This experiment compares the matched pooled checkpoint with a residual branch +containing a shared per-track GRU and a `G+A` query over current track states. +The pooled path is copied exactly and frozen. The residual is multiplied by a +learned scalar initialized at zero. + +## Protocol and provenance + +- Global features: `open_stream_2fps_8s_last4_stack`. +- Local sidecars: `local_qwen_1s_encoded_tf4576`. +- Training: EasyCom Sessions 4--11. +- Development/model selection: Session 12. +- Official test: Sessions 1--3. +- Development seeds: 0, 1 and 2; four epochs. +- Test seeds observed: seed 0 only. +- Loss: `lstr-mce`; layer fusion: `concat-mlp`. +- Person history: 16 observations at 2 FPS, with a 1-second maximum track gap. +- Reported table protocol: `causal_valid_only`; all values are percentage + points. +- Checkpoints: `checkpoints/easycom_track_gru_residual/` on the remote L20 + machine. +- Predictions: `results/easycom_track_gru_residual/` for development and + `results/easycom_track_gru_residual_test_check/` for the one-seed test audit. + +The test audit was run before the residual optimization defect described below +was diagnosed. It is recorded for transparency and must not be used to tune a +corrected architecture. + +## Session 12 development result + +| Run | 0.2 s mAP | BG AP | wearer AP | others AP | 10-horizon mean mAP | +|---|---:|---:|---:|---:|---:| +| pooled seed 0 | 72.91519 | 58.39896 | 68.44729 | 91.89933 | 59.42976 | +| track-GRU seed 0 | 72.91538 | 58.39965 | 68.44750 | 91.89900 | 59.42956 | +| pooled seed 1 | 73.02189 | 57.73258 | 69.35309 | 91.97999 | 59.50838 | +| track-GRU seed 1 | 73.02358 | 57.74019 | 69.35432 | 91.97624 | 59.50736 | +| pooled seed 2 | 72.45010 | 57.26123 | 68.35503 | 91.73403 | 59.53730 | +| track-GRU seed 2 | 72.47576 | 57.15675 | 68.59149 | 91.67904 | 59.57771 | + +Three-seed paired `track-GRU - pooled` deltas, mean +/- sample standard +deviation: + +| Metric | Delta (percentage points) | +|---|---:| +| 0.2 s mAP | +0.00918 +/- 0.01429 | +| BG AP | -0.03206 +/- 0.06281 | +| wearer AP | +0.07930 +/- 0.13610 | +| others AP | -0.01969 +/- 0.03062 | +| 10-horizon mean mAP | +0.01306 +/- 0.02368 | + +The development difference is effectively zero and is much smaller than the +between-seed variation. + +## Sessions 1--3 test audit + +Only seed 0 was evaluated after the matched checkpoints had already been +selected on Session 12. + +| Run | 0.2 s mAP | BG AP | wearer AP | others AP | 10-horizon mean mAP | +|---|---:|---:|---:|---:|---:| +| pooled seed 0 | 81.74161 | 68.57326 | 84.71790 | 91.93366 | 64.88810 | +| track-GRU seed 0 | 81.74160 | 68.57367 | 84.71783 | 91.93330 | 64.88780 | +| track-GRU - pooled | -0.00001 | +0.00041 | -0.00007 | -0.00037 | -0.00030 | + +Under the EgoSpeak full-timeline printout, the corresponding ten-horizon means +were 64.87493 for pooled and 64.87463 for track-GRU. The small difference from +the table is due to the evaluation mask; protocols must not be mixed. + +## Gate diagnostic + +The selected track checkpoints retained an almost closed residual gate: + +| Seed | Selected epoch | saved `tanh(gate)` | selection-time dev mean mAP | +|---|---:|---:|---:| +| 0 | 1 | -0.00152618 | 59.45070 | +| 1 | 1 | +0.00409810 | 59.52841 | +| 2 | 2 | +0.01081575 | 59.59214 | + +The checkpoint JSON contains history only through the epoch at which that best +checkpoint was written, so it is not a complete log of later non-improving +epochs. The values above nevertheless describe the models used for inference. + +With logits + +```text +base_logits + tanh(gate) * delta(track_parameters) +``` + +the zero-initialized gate makes the first-step gradient of all track/delta +parameters exactly zero. Only the scalar gate can move initially; subsequent +branch gradients are scaled by its still-small value. The saved gates and the +near-identical test predictions are therefore consistent with an optimization +failure in the residual parameterization. + +## Interpretation and decision + +- The implemented zero-gated track-GRU produces no meaningful dense-AP gain. +- This run does **not** establish that track continuity is uninformative, + because the residual branch was almost closed throughout the selected + models. +- It also does not support event-level or visual-grounding claims: no corrected + competing-risk event evaluation or wrong-track/time-shift control was run. +- The appropriate next diagnostic is to remove the scalar gate, zero-initialize + the last residual output layer, keep the pooled base frozen, and verify that + residual-output and GRU gradients become nonzero during optimization. +- Tune that corrected model on Session 12 only. Do not inspect Sessions 1--3 + again until its architecture and stopping rule are frozen. + +Machine-readable values are in [`summary.json`](summary.json) and per-run rows +are in [`metrics.csv`](metrics.csv). diff --git a/results/easycom_track_gru_residual_4ep_2026-09-07/metrics.csv b/results/easycom_track_gru_residual_4ep_2026-09-07/metrics.csv new file mode 100644 index 0000000..ae035b0 --- /dev/null +++ b/results/easycom_track_gru_residual_4ep_2026-09-07/metrics.csv @@ -0,0 +1,9 @@ +split,protocol,variant,seed,map_0p2,bg_ap_0p2,wearer_ap_0p2,others_ap_0p2,mean_map,best_epoch,saved_track_gate +dev_session12,causal_valid_only,pooled,0,72.91519,58.39896,68.44729,91.89933,59.42976,, +dev_session12,causal_valid_only,track_gru,0,72.91538,58.39965,68.44750,91.89900,59.42956,1,-0.0015261819353327155 +dev_session12,causal_valid_only,pooled,1,73.02189,57.73258,69.35309,91.97999,59.50838,, +dev_session12,causal_valid_only,track_gru,1,73.02358,57.74019,69.35432,91.97624,59.50736,1,0.004098099190741777 +dev_session12,causal_valid_only,pooled,2,72.45010,57.26123,68.35503,91.73403,59.53730,, +dev_session12,causal_valid_only,track_gru,2,72.47576,57.15675,68.59149,91.67904,59.57771,2,0.010815753601491451 +test_sessions1_3,causal_valid_only,pooled,0,81.74161,68.57326,84.71790,91.93366,64.88810,, +test_sessions1_3,causal_valid_only,track_gru,0,81.74160,68.57367,84.71783,91.93330,64.88780,1,-0.0015261819353327155 diff --git a/results/easycom_track_gru_residual_4ep_2026-09-07/summary.json b/results/easycom_track_gru_residual_4ep_2026-09-07/summary.json new file mode 100644 index 0000000..dc11c7d --- /dev/null +++ b/results/easycom_track_gru_residual_4ep_2026-09-07/summary.json @@ -0,0 +1,61 @@ +{ + "record": "easycom_track_gru_residual_4ep_2026-09-07", + "dataset": "easycom", + "global_features": "open_stream_2fps_8s_last4_stack", + "local_features": "local_qwen_1s_encoded_tf4576", + "train_sessions": [4, 5, 6, 7, 8, 9, 10, 11], + "dev_sessions": [12], + "test_sessions": [1, 2, 3], + "epochs": 4, + "loss": "lstr-mce", + "layer_fusion": "concat-mlp", + "person_history_frames": 16, + "person_history_fps": 2.0, + "person_track_max_gap_seconds": 1.0, + "protocol": "causal_valid_only", + "units": "percentage_points", + "dev": { + "seeds": [0, 1, 2], + "paired_delta_track_minus_pooled": { + "map_0p2": {"mean": 0.00918, "sample_std": 0.01429}, + "bg_ap_0p2": {"mean": -0.03206, "sample_std": 0.06281}, + "wearer_ap_0p2": {"mean": 0.07930, "sample_std": 0.13610}, + "others_ap_0p2": {"mean": -0.01969, "sample_std": 0.03062}, + "mean_map": {"mean": 0.01306, "sample_std": 0.02368} + } + }, + "test_audit": { + "seeds": [0], + "pooled": { + "map_0p2": 81.74161, + "bg_ap_0p2": 68.57326, + "wearer_ap_0p2": 84.71790, + "others_ap_0p2": 91.93366, + "mean_map": 64.88810, + "full_timeline_mean_map": 64.87493 + }, + "track_gru": { + "map_0p2": 81.74160, + "bg_ap_0p2": 68.57367, + "wearer_ap_0p2": 84.71783, + "others_ap_0p2": 91.93330, + "mean_map": 64.88780, + "full_timeline_mean_map": 64.87463 + }, + "paired_delta_track_minus_pooled": { + "map_0p2": -0.00001, + "bg_ap_0p2": 0.00041, + "wearer_ap_0p2": -0.00007, + "others_ap_0p2": -0.00037, + "mean_map": -0.00030 + }, + "warning": "Observed before diagnosing residual optimization; do not use to tune the corrected architecture." + }, + "selected_track_checkpoints": [ + {"seed": 0, "best_epoch": 1, "saved_tanh_gate": -0.0015261819353327155, "selection_dev_mean_map": 0.5945069810061214}, + {"seed": 1, "best_epoch": 1, "saved_tanh_gate": 0.004098099190741777, "selection_dev_mean_map": 0.5952841235072595}, + {"seed": 2, "best_epoch": 2, "saved_tanh_gate": 0.010815753601491451, "selection_dev_mean_map": 0.5959213772894789} + ], + "decision": "No meaningful gain. Treat as a zero-gate optimization failure, not evidence against track continuity.", + "next_step": "Remove scalar gate, zero-initialize the final residual layer, verify nonzero branch gradients, and screen only on Session 12." +} diff --git a/run_easycom_oracle_information.sh b/run_easycom_oracle_information.sh new file mode 100755 index 0000000..8746177 --- /dev/null +++ b/run_easycom_oracle_information.sh @@ -0,0 +1,191 @@ +#!/usr/bin/env bash +set -euo pipefail + +stage="${1:-screen}" +shift || true +if (( $# )); then + seeds=("$@") +else + seeds=(0) +fi + +: "${DATASET_ROOT:?Set DATASET_ROOT to the extracted EasyCom root}" +: "${FEATURE_ROOT:?Set FEATURE_ROOT to the verified global Omni features}" +: "${TARGET_ROOT:?Set TARGET_ROOT to EasyCom target_perframe files}" +: "${LOCAL_ROOT:?Set LOCAL_ROOT to the complete Qwen local AV sidecars}" + +ORACLE_ROOT="${ORACLE_ROOT:-features/easycom_oracle_information_2s}" +CHECKPOINT_ROOT="${CHECKPOINT_ROOT:-checkpoints/easycom_oracle_information_4ep}" +RESULT_ROOT="${RESULT_ROOT:-results/easycom_oracle_information_4ep}" +TRAIN_SESSIONS="${TRAIN_SESSIONS:-4 5 6 7 8 9 10 11 12}" +DEV_SESSIONS="${DEV_SESSIONS:-12}" +SCREEN_SESSIONS="${SCREEN_SESSIONS:-12}" +TEST_SESSIONS="${TEST_SESSIONS:-1 2 3}" +EPOCHS="${EPOCHS:-4}" +WARMUP_EPOCHS="${WARMUP_EPOCHS:-1}" +LAYER_FUSION="${LAYER_FUSION:-concat-mlp}" +EXPECTED_MODE="${EXPECTED_MODE:-open-stream-replay}" +VARIANTS="${VARIANTS:-audio visibility-only aggregate-visible per-person shuffled-id delayed}" +ORACLE_CONDITIONS="${ORACLE_CONDITIONS:-visibility-only aggregate-visible per-person shuffled-id delayed}" +PYTHON_BIN="${PYTHON_BIN:-python}" + +read -r -a train_sessions <<<"${TRAIN_SESSIONS}" +read -r -a dev_sessions <<<"${DEV_SESSIONS}" +read -r -a screen_sessions <<<"${SCREEN_SESSIONS}" +read -r -a test_sessions <<<"${TEST_SESSIONS}" +read -r -a variants <<<"${VARIANTS}" +read -r -a oracle_conditions <<<"${ORACLE_CONDITIONS}" + +layer_args=() +if [[ "${LAYER_FUSION}" != "single" ]]; then + layer_args=(--layer-fusion "${LAYER_FUSION}") +fi + +prepare_oracles() { + "${PYTHON_BIN}" prepare_easycom_oracle_features.py \ + --dataset-root "${DATASET_ROOT}" \ + --feature-root "${FEATURE_ROOT}" \ + --target-root "${TARGET_ROOT}" \ + --local-feature-root "${LOCAL_ROOT}" \ + --output-root "${ORACLE_ROOT}" \ + --conditions "${oracle_conditions[@]}" \ + --history-seconds 2.0 \ + --delay-seconds 1.0 \ + --expected-mode "${EXPECTED_MODE}" +} + +variant_local_args() { + local variant="$1" + if [[ "${variant}" == "audio" ]]; then + printf '%s\0' \ + --local-feature-root "${LOCAL_ROOT}" \ + --local-modalities audio \ + --local-fusion pooled + else + printf '%s\0' \ + --local-feature-root "${ORACLE_ROOT}/${variant}" \ + --local-modalities av \ + --local-fusion oracle-frame-set-attention \ + --person-attention-dim 64 \ + --person-attention-heads 4 + fi +} + +train_and_screen() { + mkdir -p "${CHECKPOINT_ROOT}" "${RESULT_ROOT}" + for seed in "${seeds[@]}"; do + for variant in "${variants[@]}"; do + checkpoint="${CHECKPOINT_ROOT}/${variant}_seed${seed}.pth" + result="${RESULT_ROOT}/${variant}_seed${seed}" + local_feature_root="${ORACLE_ROOT}/${variant}" + if [[ "${variant}" == "audio" ]]; then + local_feature_root="${LOCAL_ROOT}" + fi + if [[ -e "${checkpoint}" || -e "${result}" ]]; then + echo "Refusing to overwrite ${checkpoint} or ${result}" >&2 + exit 1 + fi + local_args=() + while IFS= read -r -d '' item; do + local_args+=("${item}") + done < <(variant_local_args "${variant}") + echo "$(date): training ${variant}, seed=${seed}" + "${PYTHON_BIN}" train_turn_head.py \ + --feature-root "${FEATURE_ROOT}" \ + --target-root "${TARGET_ROOT}" \ + --output "${checkpoint}" \ + --sessions "${train_sessions[@]}" \ + --dev-sessions "${dev_sessions[@]}" \ + --epochs "${EPOCHS}" \ + --batch-size 512 \ + --learning-rate 7e-5 \ + --weight-decay 5e-5 \ + --warmup-epochs "${WARMUP_EPOCHS}" \ + --warmup-factor 0.3 \ + --dropout 0.2 \ + --loss lstr-mce \ + "${layer_args[@]}" \ + "${local_args[@]}" \ + --local-projection-dim 256 \ + --expected-mode "${EXPECTED_MODE}" \ + --seed "${seed}" \ + --device cuda:0 + + echo "$(date): screening ${variant}, seed=${seed} on ${screen_sessions[*]}" + "${PYTHON_BIN}" infer_turn_head.py \ + --feature-root "${FEATURE_ROOT}" \ + --target-root "${TARGET_ROOT}" \ + --checkpoint "${checkpoint}" \ + --output-dir "${result}" \ + --sessions "${screen_sessions[@]}" \ + --local-feature-root "${local_feature_root}" \ + --expected-mode "${EXPECTED_MODE}" \ + --device cuda:0 + done + done + "${PYTHON_BIN}" summarize_local_baselines.py \ + --root "${RESULT_ROOT}" \ + --dataset easycom \ + --protocol causal_valid_only +} + +test_selected() { + mkdir -p "${RESULT_ROOT}_test" + for seed in "${seeds[@]}"; do + for variant in "${variants[@]}"; do + checkpoint="${CHECKPOINT_ROOT}/${variant}_seed${seed}.pth" + result="${RESULT_ROOT}_test/${variant}_seed${seed}" + local_feature_root="${ORACLE_ROOT}/${variant}" + if [[ "${variant}" == "audio" ]]; then + local_feature_root="${LOCAL_ROOT}" + fi + if [[ ! -f "${checkpoint}" ]]; then + echo "Missing checkpoint: ${checkpoint}" >&2 + exit 1 + fi + if [[ -e "${result}" ]]; then + echo "Refusing to overwrite ${result}" >&2 + exit 1 + fi + "${PYTHON_BIN}" infer_turn_head.py \ + --feature-root "${FEATURE_ROOT}" \ + --target-root "${TARGET_ROOT}" \ + --checkpoint "${checkpoint}" \ + --output-dir "${result}" \ + --sessions "${test_sessions[@]}" \ + --local-feature-root "${local_feature_root}" \ + --expected-mode "${EXPECTED_MODE}" \ + --device cuda:0 + done + done + "${PYTHON_BIN}" summarize_local_baselines.py \ + --root "${RESULT_ROOT}_test" \ + --dataset easycom \ + --protocol causal_valid_only +} + +case "${stage}" in + prepare) + prepare_oracles + ;; + screen) + train_and_screen + ;; + all) + prepare_oracles + train_and_screen + ;; + test) + test_selected + ;; + summarize) + "${PYTHON_BIN}" summarize_local_baselines.py \ + --root "${RESULT_ROOT}" \ + --dataset easycom \ + --protocol causal_valid_only + ;; + *) + echo "Usage: $0 {prepare|screen|all|test|summarize} [seed ...]" >&2 + exit 2 + ;; +esac diff --git a/test_easycom_oracle_features.py b/test_easycom_oracle_features.py new file mode 100644 index 0000000..dec1d9a --- /dev/null +++ b/test_easycom_oracle_features.py @@ -0,0 +1,199 @@ +"""Focused tests for the causal EasyCom oracle-information experiment.""" + +from __future__ import annotations + +from pathlib import Path +from types import SimpleNamespace +import tempfile +import unittest + +import numpy as np +import torch + +from easycom_head import TurnPredictionHead +from local_turn_features import ORACLE_FORMAT, load_frame_set_features, load_local_features +from prepare_easycom_oracle_features import ( + causal_history, + oracle_person_rows, + participant_activity, +) + + +class OracleFeatureTests(unittest.TestCase): + def setUp(self) -> None: + torch.set_num_threads(1) + + def test_transcript_intervals_are_rasterized_into_half_open_bins(self) -> None: + rows = [ + SimpleNamespace( + uid="a", + participant_id=3, + dense_start_frame=3, + dense_end_frame_exclusive=5, + ), + SimpleNamespace( + uid="b", + participant_id=2, + dense_start_frame=8, + dense_end_frame_exclusive=12, + ), + ] + activity, wearer, audit = participant_activity( + rows, num_steps=4, wearer_id=2 + ) + self.assertEqual(activity[3].tolist(), [True, True, False, False]) + self.assertEqual(wearer.tolist(), [False, False, True, False]) + self.assertEqual(audit["discarded_terminal_transcript_frames"], 0) + + def test_history_at_row_does_not_read_later_activity(self) -> None: + first = torch.tensor([0, 1, 0, 0, 0], dtype=torch.bool) + second = first.clone() + second[3:] = True + observed = causal_history(first, row=2, history_steps=4) + changed_future = causal_history(second, row=2, history_steps=4) + self.assertTrue(torch.equal(observed, changed_future)) + self.assertEqual( + observed.tolist(), + [0, 0, 1, 0, 0, 1, 1, 1], + ) + delayed = causal_history(second, row=2, history_steps=4, delay_steps=2) + self.assertEqual( + delayed.tolist(), + [0, 0, 0, 0, 0, 0, 0, 1], + ) + visibility = torch.tensor([1, 0, 1, 1, 1], dtype=torch.bool) + visible_only = causal_history( + second, + row=2, + history_steps=4, + observed_sequence=visibility, + ) + self.assertEqual( + visible_only.tolist(), + [0, 0, 0, 0, 0, 1, 0, 1], + ) + + def test_shuffled_id_preserves_every_lag_count_but_breaks_histories(self) -> None: + activity = { + 3: torch.tensor([1, 1, 0, 0, 1, 1], dtype=torch.bool), + 4: torch.tensor([0, 0, 1, 1, 0, 0], dtype=torch.bool), + 5: torch.tensor([0, 1, 0, 1, 0, 1], dtype=torch.bool), + } + visible = [[3, 4, 5] for _ in range(6)] + visibility = { + person_id: torch.ones(6, dtype=torch.bool) + for person_id in activity + } + records = [ + [ + {"id": str(person_id), "source_clip_index": 0, "source_time": row / 5} + for person_id in person_ids + ] + for row, person_ids in enumerate(visible) + ] + original, offsets, _, _ = oracle_person_rows( + activity, + visibility, + visible, + records, + condition="per-person", + history_steps=4, + delay_steps=5, + seed=9, + ) + shuffled, shuffled_offsets, _, audit = oracle_person_rows( + activity, + visibility, + visible, + records, + condition="shuffled-id", + history_steps=4, + delay_steps=5, + seed=9, + ) + self.assertTrue(torch.equal(offsets, shuffled_offsets)) + for row in range(6): + start, stop = int(offsets[row]), int(offsets[row + 1]) + self.assertTrue( + torch.equal( + original[start:stop, :4].sum(dim=0), + shuffled[start:stop, :4].sum(dim=0), + ) + ) + self.assertGreater(audit["shuffled_values_changed"], 0) + + def test_oracle_format_requires_explicit_loader_and_head_mode(self) -> None: + n = 4 + times = (torch.arange(n, dtype=torch.float64) + 1) / 5 + reference = { + "states": torch.randn(n, 8), + "label_indices": torch.arange(n), + "times": times, + "mode": "open-stream-replay", + "start": 0, + } + sidecar = { + "format": ORACLE_FORMAT, + "dataset": "easycom", + "sample_id": "Session_4", + "times": times.clone(), + "label_indices": torch.arange(n), + "partial": False, + "visual": torch.randn(n, 6), + "audio": torch.randn(n, 5), + "visual_valid": torch.ones(n, dtype=torch.bool), + "audio_valid": torch.ones(n, dtype=torch.bool), + "persons": { + "features": torch.randn(n, 6), + "frame_times": torch.arange(n, dtype=torch.float64) / 5, + "frame_offsets": torch.arange(n + 1, dtype=torch.long), + "records": [{"id": "3"} for _ in range(n)], + }, + "config": { + "oracle_analysis": True, + "uses_speech_labels": True, + "causal_policy": "fully-observed-half-open-5hz-bins", + "oracle_condition": "per-person", + "visual_dim": 6, + "audio_dim": 5, + }, + } + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "Session_4.pt" + torch.save(sidecar, path) + with self.assertRaisesRegex(ValueError, "format"): + load_local_features( + path, + reference, + sample_id="Session_4", + dataset="easycom", + modalities="av", + ) + audio, people, contract = load_frame_set_features( + path, + reference, + sample_id="Session_4", + dataset="easycom", + oracle_analysis=True, + ) + self.assertEqual(audio.shape, (n, 6)) + self.assertEqual(people.dimension, 6) + self.assertEqual(contract["format"], ORACLE_FORMAT) + head = TurnPredictionHead( + hidden_size=8, + num_horizons=2, + num_classes=3, + local_visual_dim=6, + local_audio_dim=5, + local_projection_dim=4, + local_fusion="oracle-frame-set-attention", + person_attention_dim=4, + person_attention_heads=1, + ) + padded, mask = people.padded() + logits = head(reference["states"], audio, padded, mask) + self.assertEqual(logits.shape, (n, 2, 3)) + + +if __name__ == "__main__": + unittest.main()