wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add causal oracle person information screen

Browse Source
This commit is contained in:
Xin Wang
2026-09-07 08:41:34 +08:00
parent fb22108c57
commit ecf00185f1
17 changed files with 1364 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
PROJECT_CONTEXT.md
View File

@@ -52,7 +52,7 @@ result.
matched improvement over the historical no-dev pooled result; correct/wrong
visual controls remain missing.
### Implemented, awaiting remote runtime verification
### Measured local-fusion screens
- Two current-sidecar screening controls: `audio-person-frame-set` adds a
zero-gated audio-to-person read before the F1 query, and
@@ -62,10 +62,15 @@ result.
- `track-gru-residual` builds strictly causal histories from anonymous provided
track IDs, applies one shared GRU per track, lets G+A query current track
states, and adds a zero-gated correction to an exactly initialized pooled
`G+A+V` checkpoint. Its three focused loader/head/CLI round-trip tests passed
on the remote PyTorch/L20 environment on 2026-09-07; the one-session GPU
smoke and measured experiment are still pending. See
[`docs/easycom_track_gru_residual.md`](docs/easycom_track_gru_residual.md).
`G+A+V` checkpoint. The matched Session-12 three-seed screen was null
(`+0.01306 +/- 0.02368` percentage points in ten-horizon mean mAP). A seed-0
Sessions-1--3 audit reproduced pooled predictions to within `-0.00030`
points. Saved residual gates were only `-0.0015` to `+0.0108`; because a zero
scalar gate also blocks the initial branch gradient, this is recorded as an
optimization failure rather than evidence that track continuity is useless.
See the immutable
[`result record`](results/easycom_track_gru_residual_4ep_2026-09-07/README.md)
and [`implementation note`](docs/easycom_track_gru_residual.md).
### Current blocker
@@ -89,9 +94,10 @@ diagnostics until competing-risk censoring is fixed.
2. **P1:** probe aggregate activity and per-person active-speaker grounding;
compare correct, wrong-person and time-shifted visual inputs.
3. **P2:** re-evaluate `G`, `G+A`, `G+A+pooled-V` under the frozen event protocol.
4. **P3:** run the matched pooled/anonymous-track-GRU residual screen, then add
broken-ID, wrong-time and remove-person controls before claiming that track
continuity matters.
4. **P3:** repair the matched anonymous-track-GRU residual optimization by
replacing the scalar zero gate with a zero-initialized residual output
layer, and verify nonzero branch gradients on Session 12. Add broken-ID,
wrong-time and remove-person controls only if that corrected screen improves.
5. **P4:** attempt person-memory adapter/Thinker LoRA only if P1/P3 pass.
The detailed architecture, controls and stopping gates for P3 are frozen in
@@ -228,6 +234,7 @@ Do not reorganize them physically without compatibility wrappers.
| `derive_easycom_interaction_events.py` | Deterministic transcript/timing proxies |
| `adjudicate_easycom_interaction_events.py` | Optional ambiguous-event adjudication |
| `audit_easycom_floor_events.py` | Count EOT/takeover/backchannel candidates |
| `prepare_easycom_oracle_features.py` | Build explicit past-only oracle person-activity sidecars for information-value screening |
### Local features
@@ -245,6 +252,7 @@ Do not reorganize them physically without compatibility wrappers.
| `evaluate_floor_state_changes.py` | CPU floor-state/event proxy evaluation |
| `summarize_local_baselines.py` | Aggregate local baseline `metrics.json` files |
| `run_easycom_track_gru_residual.sh` | Matched pooled and anonymous track-GRU training/inference launcher |
| `run_easycom_oracle_information.sh` | Prepare and screen aggregate/per-person/shuffled/delayed oracle conditions on Session 12 |
| `train_omni_turn_lora.py` | Thinker LoRA training from cached AV |
| `eval_omni_turn_lora.py` | Thinker LoRA evaluation |

1
docs/README.md
View File

@@ -24,6 +24,7 @@ tests.
| [`ego4d_encoder_only.md`](ego4d_encoder_only.md) | Preparing and running the Ego4D encoder-only comparison |
| [`ego4d_local_baselines.md`](ego4d_local_baselines.md) | Previewing Ego4D boxes, extracting local features and running 8-GPU jobs |
| [`easycom_track_gru_residual.md`](easycom_track_gru_residual.md) | Matched pooled-to-track GRU experiment, causal identity contract and L20 commands |
| [`easycom_oracle_information.md`](easycom_oracle_information.md) | Causal oracle per-person speaking-history upper bound and Session-12 screening commands |
## Design history and implementation options

100
docs/easycom_oracle_information.md Normal file
View File

@@ -0,0 +1,100 @@
# EasyCom causal oracle-person information screen
## Question
Before training a heavier visual grounding model, test whether perfect
participant-resolved **past** speaking activity contains useful information for
future wearer activity beyond the frozen Omni state and recent mixed audio.
This is an information upper-bound experiment, not a deployable input setting.
## Causal contract
- The task grid remains 5 Hz. State row `t` has cutoff `(t+1)/5` seconds.
- Oracle bin `t` summarizes transcript frames in
`[t/5,(t+1)/5)` and is therefore completely observed at that cutoff.
- Each person vector stores the last 2 seconds of binary activity followed by
a visibility/availability mask. Activity is exposed only in bins where that
participant was visible according to the provided boxes. No future bin,
future wearer state, transcript text or `Target_of_Speech` value is input.
- Only non-wearer participant IDs present in the latest provided-box frame
strictly before the cutoff receive slots. IDs route annotations during
preparation but are not exposed as embeddings to the head.
- Transcript-derived wearer/others activity must match `target_perframe` by
default. Any mismatch fails closed instead of being silently tolerated.
- Oracle sidecars use `easycom-oracle-person-history-v1` and are accepted only
by the explicit `oracle-frame-set-attention` mode. Normal local-feature
loading rejects them because they use speech labels.
## Conditions
| Condition | Information exposed | Purpose |
|---|---|---|
| `audio` | Omni `G` plus the existing recent local audio `A` | deployable baseline |
| `visibility-only` | current visible-person set, but all activity bits are zero | parameter/person-count control |
| `aggregate-visible` | one OR-pooled history over currently visible people | aggregate floor-information upper bound |
| `per-person` | one correct activity history per currently visible participant | main oracle information test |
| `shuffled-id` | preserves the number active at every lag but independently reassigns activity between people | destroys identity continuity |
| `delayed` | correct per-person histories delayed by 1 second | destroys current temporal alignment without future leakage |
The decisive comparison is `per-person` against `aggregate-visible`,
`shuffled-id` and `delayed`, not only against `audio`. A gain over `audio` alone
could come from perfect aggregate VAD rather than person continuity.
## Remote preparation and one-seed Session-12 screen
The source local sidecars provide only mixed audio and the visibility/box
timeline. Their learned visual vectors are not copied into the oracle input.
```bash
cd /data1/yfzx/wangx/Code/icassp2027-experiment
export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
export TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
export LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
export ORACLE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/oracle_person_history_2s
export CHECKPOINT_ROOT=checkpoints/easycom_oracle_information_4ep
export RESULT_ROOT=results/easycom_oracle_information_4ep
CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh all 0
```
`all` prepares the five oracle conditions, trains on Sessions 4--11, selects
the checkpoint on Session 12, runs Session-12 inference and prints the standard
dense AP/mAP screen. It deliberately does not run Sessions 1--3.
If preparation reports any transcript/target mismatch, stop and inspect it.
Do not raise `--max-label-mismatch-rows` merely to make the run continue.
## Follow-up
If `per-person` is consistently better on Session 12, add seeds 1 and 2 using
the already prepared sidecars:
```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh screen 1 2
```
Only after freezing the method and event evaluator should selected variants be
run once on Sessions 1--3:
```bash
VARIANTS="audio aggregate-visible per-person shuffled-id delayed" \
CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_oracle_information.sh test 0 1 2
```
## Interpretation
- `per-person > aggregate-visible` and `per-person > shuffled-id`: speaker
continuity has information value; proceed to learned causal ASD/tracks.
- `aggregate-visible > audio`, but no per-person advantage: perfect aggregate
activity helps, but the current target does not justify an identity claim.
- oracle conditions do not improve: do not spend the main GPU budget on a more
complex person encoder until the task/head/evaluator is reconsidered.
- oracle improves but learned Qwen face features do not: this supports a
speaker-grounding bottleneck and motivates EgoASD-like AV evidence.
Dense 10-horizon mAP is only a screening signal. The publication claim still
requires the frozen competing-risk-censored entry/yield evaluator and matched
wearer-onset-recall false-entry analysis.

18
docs/easycom_track_gru_residual.md
View File

@@ -62,8 +62,14 @@ re-extraction.
Runtime status (2026-09-07): the focused causal-history, exact-zero-gate and
train/infer round-trip tests passed in 49.699 seconds on the remote PyTorch/L20
environment. This verifies the code path, not model quality. The one-session
GPU training smoke and full experiment remain unmeasured.
environment. The subsequent matched four-epoch screen produced no meaningful
change: Session-12 three-seed mean-mAP delta was `+0.01306 +/- 0.02368`
percentage points, and the seed-0 Sessions-1--3 delta was `-0.00030` points.
The selected residual gates remained between `-0.0015` and `+0.0108`. Because
the zero scalar gate blocks the track branch's gradient at initialization, this
run is classified as an optimization failure, not a clean negative result for
track continuity. Exact values and caveats are archived in
[`../results/easycom_track_gru_residual_4ep_2026-09-07/README.md`](../results/easycom_track_gru_residual_4ep_2026-09-07/README.md).
## Fair initialization
@@ -73,6 +79,14 @@ feature contract before copying the pooled path. By default those copied
parameters are frozen. The residual gate starts at zero, so before optimization
the track model must reproduce the pooled logits exactly, element for element.
The measured screen exposed a defect in that parameterization. At a zero gate,
the first-step gradient of every residual-branch parameter is also zero; only
the scalar gate can move, and later branch gradients remain multiplied by its
small value. A corrected follow-up should instead zero-initialize the final
residual output layer and remove the scalar multiplier. That preserves exact
pooled logits at initialization while allowing the output layer to receive a
gradient immediately.
The matched screening protocol is:
- EasyCom Session 4--11: training;

22
docs/progressive_feature_fusion_route_2026-09-02.md
View File

@@ -204,6 +204,13 @@ track GRU、G+A 对 track states 的 masked attention 以及零初始化 scalar
实现、契约和命令见
[`easycom_track_gru_residual.md`](easycom_track_gru_residual.md)。
2026-09-07 的实测暴露了这个第一版参数化的优化缺陷：Session 12 三个 seed 的
ten-horizon mean mAP 配对增量仅为 `+0.01306 +/- 0.02368` 个百分点，保存 checkpoint 的
`tanh(gate)` 只有 `-0.0015` 到 `+0.0108`。零 gate 在第一步也会把 residual branch 的参数
梯度乘为零，因此该结果不能用于否定 track continuity。下一版应删除 scalar gate、将
residual 最后输出层零初始化，并在 dev 上记录 delta-logit、概率变化和非零梯度；详见
[`结果记录`](../results/easycom_track_gru_residual_4ep_2026-09-07/README.md)。
## 8. F4：Person-conditioned audio/Omni history
F4 让每个匿名人物分别读取已有的 5 Hz 因果历史：
@@ -279,11 +286,14 @@ README 必须写明 split、seed、checkpoint、输入特征版本、评测协
## 12. 最近执行顺序
1. 用户当前选择先做 dense-AP model screen；运行 matched pooled 与
`track-gru-residual`，固定 Session 12 dev，不用测试集选历史长度。
2. 若 track GRU 有稳定增益，补 broken-track、past-shift、no-ID 和 remove-person 控制；若
无增益，只在 dev 上比较 8/16/32 observations 后停止该分支。
3. correct track 通过控制后，再实现 F4 person-conditioned audio/Omni history；否则回到
1. 第一版 matched pooled/`track-gru-residual` dense-AP screen 已完成，但因 zero-gate
gradient blockage 判为优化失败，不是 track continuity 的有效阴性结果。
2. 先用零初始化 residual 输出层修正梯度路径，只在 Session 12 做 seed-0 诊断；确认分支
实际改变 logits 后再跑三 seed。若修正后仍无增益，再决定是否比较 8/16/32
observations；不要根据已经查看过的 Sessions 1--3 seed-0 结果调参。
3. 若修正后的 track GRU 有稳定增益，补 broken-track、past-shift、no-ID 和 remove-person
控制；若仍无增益则停止该分支。
4. correct track 通过控制后，再实现 F4 person-conditioned audio/Omni history；否则回到
对齐审计，不直接堆更深 interaction。
4. 在形成论文事件结论前，仍须完成 P0 competing-risk censoring，并人工打印 response 与
5. 在形成论文事件结论前，仍须完成 P0 competing-risk censoring，并人工打印 response 与
yield 时间轴。当前事件结果只能称为 timing-derived proxy。

3
easycom_head.py
View File

@@ -63,6 +63,7 @@ LAYER_FUSION_MODES = (
)
CURRENT_FRAME_PERSON_SET_FUSION_MODES = (
"frame-set-attention",
"oracle-frame-set-attention",
"audio-person-frame-set",
"gated-person-state-adapter",
)
@@ -75,6 +76,7 @@ LOCAL_FUSION_MODES = ("pooled", *PERSON_SET_FUSION_MODES)
LOCAL_FUSION_SCOPES = {
"pooled": "pooled-baseline",
"frame-set-attention": "f1-current-frame-person-set",
"oracle-frame-set-attention": "causal-oracle-visible-person-speaking-history",
"audio-person-frame-set": "f5-screening-current-frame-no-person-history",
"gated-person-state-adapter": "f6-screening-post-thinker-not-internal",
"track-gru-residual": "causal-anonymous-track-gru-pooled-baseline-residual",
@@ -1303,6 +1305,7 @@ def load_session_data(
sample_id=name,
dataset=dataset,
expected_contract=local_contract,
oracle_analysis=local_fusion == "oracle-frame-set-attention",
)
else:
from local_turn_features import load_local_features

26
literature_notes/54_egoasd.md Normal file
View File

@@ -0,0 +1,26 @@
# Advancing Active Speaker Detection for Egocentric Videos（EgoASD）
- 作者：Jaesung Huh、Juan Azcarreta Ortiz、Anurag Kumar、Ashutosh Pandey、Ali Aroudi、Daniel D. E. Wong、Francesco Nesta、Buye Xu、Jacob Donley
- 发表：ICASSP 2025
- 阅读深度：摘要、方法贡献和 EasyCom 结果筛选；未运行
- 记录日期：2026-09-07
- 来源：[IEEE Xplore](https://ieeexplore.ieee.org/document/10888166/)、[DOI](https://doi.org/10.1109/ICASSP49660.2025.10888166)
## 摘要与项目关系
该工作针对第一视角视频中快速头动、运动模糊和视觉质量不稳定的问题，提出固定嘴部区域与
motion-blur augmentation，并将它们用于一个轻量、因果的音视频 active speaker detector。
论文报告 EgoASD 在 EasyCom 上比此前结果提高 1.7 个 mAP 点，同时模型规模约小 2.5 倍，
并进一步用于第一视角音视频 diarization。
它直接说明：EasyCom 的逐人视觉分支不能只依赖通用图像语义或稀疏静态人脸 embedding；嘴部
局部运动、短时音唇同步、运动模糊鲁棒性和因果 temporal context 都是更合理的 active-speaker
证据。若本项目最终声称人物视觉 grounding 有效，应把 EgoASD 或同等级 causal ASD feature
作为专用视觉前端基线/候选，而不能只和普通 Qwen face-crop pooling 比较。
## 适用边界
EgoASD 预测画面中候选人物当前是否发声，不预测不可见 wearer 未来是否进入或让出 floor。
因此它可以提供逐人 active-speaker evidence 或 oracle 的可学习替代，但不能直接替代本项目的
wearer floor-action head 和事件级评测。论文报告的 ASD mAP 也不能与本项目 0.2--2.0 秒未来
活动 mAP 直接比较。

28
literature_notes/55_speaker_embedding_egocentric_asd.md Normal file
View File

@@ -0,0 +1,28 @@
# Speaker Embedding Informed Audiovisual Active Speaker Detection for Egocentric Recordings
- 作者：Jason Clarke、Yoshihiko Gotoh、Stefan Goetze
- 发表：ICASSP 2025
- 阅读深度：摘要、方法和实验设置重点阅读；未运行
- 记录日期：2026-09-07
- 来源：[IEEE Xplore](https://ieeexplore.ieee.org/document/10890414/)、[arXiv](https://arxiv.org/abs/2502.06012)、[作者公开论文](https://eprints.whiterose.ac.uk/id/eprint/223485/1/Jason_ICASSP_2025.pdf)
## 摘要与项目关系
该工作研究第一视角场景中候选人物的音视频 active speaker detection。作者提出 Speaker
Comparison Auxiliary Network（SCAN）：除候选人的参考语音 embedding 外，还从当前混合
音频的滑动窗口提取 speaker embedding，再通过 cross-attention 比较当前声音和已登记身份。
论文同时提出自监督的 video face recognition 调优方法，用于构建更稳健的 face--speaker
身份库。SCAN 接入 TalkNet 和 Light-ASD 后，在 Ego4D-AVD 上分别报告 14.5% 和 10.3% 的
相对 mAP 提升。
它说明显式 speaker reference/identity memory 在视觉遮挡或嘴部线索不可用时已有明确先例。
因此本项目不应把“用 track identity 帮助音视频说话者判断”本身作为 novelty。更合适的限定是：
人物轨迹保持匿名，不依赖预登记语音或长期身份库，只使用截止当前时刻可获得的因果轨迹，
目标是预测不可见 wearer 的未来 entry/yield 行为。
## 适用边界
SCAN 需要候选人的预分离参考语音并建立 face--speaker library；这与本项目计划的在线、匿名、
无 enrollment 场景不同。它仍然做当前逐人物 ASD，不评估 wearer 的未来错误开口、响应延迟
或 yield。若以后引入 speaker embedding，必须单列 enrollment 依赖，并确保参考语音只来自
预测截止时间之前，不能从整段视频或测试片段未来部分建立身份库。

10
literature_notes/README.md
View File

@@ -2,7 +2,13 @@
本目录持久化记录我们在本项目讨论中实际阅读、分析或筛选过的论文。这里的“摘要”均为面向当前课题的中文转述，不是原文摘要的复制。
最后更新：2026-08-30
最后更新：2026-09-07
最新增补：两篇 ICASSP 2025 第一视角 ASD 工作进一步约束当前视觉方案。
[EgoASD](54_egoasd.md)表明 EasyCom 局部视觉需要认真处理嘴部运动、运动模糊和因果时序，
而不只是静态人脸语义；[speaker-embedding-informed ASD](55_speaker_embedding_egocentric_asd.md)
说明显式身份/参考语音 memory 已有先例。当前更可辩护的方向是匿名、无需 enrollment 的逐人
因果证据，并验证它是否改变不可见 wearer 的未来 entry/yield 行为。
最新专题：[多方第一视角 AV wearer floor-state prediction 文献 gap 审计](../docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md)。结论是宽泛组件均有先例；仍可信的交集是单 wearer 严格因果流、不可见 wearer 的 entry/yield、逐人 grounding 和事件级评测。MuVAP、EgoSpeak、3M-Transformer、AV-CONV 与在线多方 EOT 必须正面区分。
@@ -75,6 +81,8 @@
| 51 | [Online Multimodal End-of-Turn Prediction](51_online_multiparty_eot.md) | 2024 | 任务、采集、方法和局限重点阅读 | ICMI；在线三方 EOT，但依赖实验室 gaze/mocap/独立麦克风 |
| 52 | [TurnBench](52_turnbench.md) | 2026 | 任务、因果提交和 leaderboard 重点阅读 | 双人音频事件评测；recall/FPR/latency 范式 |
| 53 | [AMusE](53_amuse.md) | 2026 | 摘要及官方论文页筛选 | CVPR；MLLM 多说话人身份、角色和 AV grounding 短板 |
| 54 | [Advancing Active Speaker Detection for Egocentric Videos（EgoASD）](54_egoasd.md) | 2025 | 摘要、方法贡献和 EasyCom 结果筛选 | ICASSP；因果 egocentric ASD、固定嘴部区域及运动模糊增强 |
| 55 | [Speaker Embedding Informed Audiovisual Active Speaker Detection](55_speaker_embedding_egocentric_asd.md) | 2025 | 摘要、方法和实验设置重点阅读 | ICASSP；参考语音身份、SCAN 与第一视角 ASD，约束 identity-memory novelty |
## 当前综合判断

24
local_turn_features.py
View File

@@ -14,6 +14,7 @@ from typing import Any
import torch
LOCAL_FORMAT = "qwen-local-pooled-v1"
ORACLE_FORMAT = "easycom-oracle-person-history-v1"
MODALITIES = {"visual": ("visual",), "audio": ("audio",), "av": ("visual", "audio")}
@@ -277,12 +278,14 @@ def _load_validated_payload(
dataset: str,
modalities: str,
expected_contract: dict[str, Any] | None,
oracle_analysis: bool = False,
) -> tuple[torch.Tensor, torch.Tensor, dict[str, Any], dict[str, Any], dict[str, Any]]:
if modalities not in MODALITIES:
raise ValueError(f"Unknown local modalities: {modalities}")
times, indices = reference_alignment(reference)
payload = torch.load(path, map_location="cpu", weights_only=False)
if not isinstance(payload, dict) or payload.get("format") != LOCAL_FORMAT:
expected_format = ORACLE_FORMAT if oracle_analysis else LOCAL_FORMAT
if not isinstance(payload, dict) or payload.get("format") != expected_format:
raise ValueError(f"Invalid local feature format: {path}")
if payload.get("sample_id") != sample_id or payload.get("dataset") != dataset:
raise ValueError(f"Local sample/dataset mismatch: {path}")
@@ -295,9 +298,22 @@ def _load_validated_payload(
if local_times.shape != times.shape or not torch.allclose(local_times, times, atol=1e-7, rtol=0):
raise ValueError(f"Local times mismatch: {path}")
config = payload.get("config", {})
if config.get("causal_policy") != "half-open-trailing-window" or config.get("uses_speech_labels") is not False:
if oracle_analysis:
if (
dataset != "easycom"
or config.get("causal_policy") != "fully-observed-half-open-5hz-bins"
or config.get("uses_speech_labels") is not True
or config.get("oracle_analysis") is not True
):
raise ValueError(
f"Oracle features lack the explicit causal/oracle contract: {path}"
)
elif (
config.get("causal_policy") != "half-open-trailing-window"
or config.get("uses_speech_labels") is not False
):
raise ValueError(f"Local features lack the causal/no-speech-label contract: {path}")
contract = {"format": LOCAL_FORMAT, "modalities": modalities, "config": config}
contract = {"format": expected_format, "modalities": modalities, "config": config}
if expected_contract is not None and contract != expected_contract:
raise ValueError(f"Local encoder/pooling contract differs across sessions/checkpoint: {path}")
return times, indices, payload, config, contract
@@ -348,6 +364,7 @@ def load_frame_set_features(
dataset: str,
expected_contract: dict[str, Any] | None = None,
track_max_gap_seconds: float | None = None,
oracle_analysis: bool = False,
) -> tuple[torch.Tensor, FramePersonFeatures, dict[str, Any]]:
"""Load direct audio and the latest causal per-frame person set per 5 Hz row."""
@@ -358,6 +375,7 @@ def load_frame_set_features(
dataset=dataset,
modalities="av",
expected_contract=expected_contract,
oracle_analysis=oracle_analysis,
)
audio = payload.get("audio")
audio_valid = payload.get("audio_valid")

551
prepare_easycom_oracle_features.py Normal file
View File

@@ -0,0 +1,551 @@
#!/usr/bin/env python3
"""Build explicit, causal EasyCom oracle-person sidecars for information tests.
This is not a deployable feature extractor. It combines the already extracted
recent mixed-audio vector with ground-truth, participant-resolved speech
activity from EasyCom transcripts. Only fully observed 200 ms bins ending no
later than each state cutoff are exposed. The output format is deliberately
rejected by ordinary local-feature loaders.
"""
from __future__ import annotations
import argparse
import os
import tempfile
from pathlib import Path
from typing import Any, Sequence
import numpy as np
import torch
from derive_easycom_interaction_events import (
CAMERA_WEARER_ID,
FRAMES_PER_TASK_STEP,
Rules,
build_clips,
load_utterances,
)
from easycom_head import (
NUM_CLASSES,
STEP_SECONDS,
normalize_sample_name,
resolve_feature_path,
)
from easycom_manifest import ALL_SESSIONS
from local_turn_features import (
ORACLE_FORMAT,
load_frame_set_features,
reference_alignment,
)
ORACLE_CONDITIONS = (
"visibility-only",
"aggregate-visible",
"per-person",
"shuffled-id",
"delayed",
)
UNKNOWN_PERSON_IDS = {"", "-1", "none", "null", "unknown"}
def exact_steps(seconds: float, name: str) -> int:
if not np.isfinite(seconds) or seconds <= 0:
raise ValueError(f"{name} must be finite and positive")
steps = round(seconds / STEP_SECONDS)
if not np.isclose(seconds, steps * STEP_SECONDS, atol=1e-9, rtol=0):
raise ValueError(f"{name} must be a multiple of {STEP_SECONDS:g}s")
return int(steps)
def participant_activity(
utterances: Sequence[Any],
*,
num_steps: int,
wearer_id: int,
) -> tuple[dict[int, torch.BoolTensor], torch.BoolTensor, dict[str, int]]:
"""Rasterize inclusive 20 Hz transcript spans into observed 5 Hz bins."""
activity: dict[int, torch.BoolTensor] = {}
wearer = torch.zeros(num_steps, dtype=torch.bool)
discarded_terminal_frames = 0
grid_frames = num_steps * FRAMES_PER_TASK_STEP
for row in utterances:
dense_start = int(row.dense_start_frame)
dense_end = int(row.dense_end_frame_exclusive)
if dense_end > grid_frames:
overrun = dense_end - grid_frames
if overrun > FRAMES_PER_TASK_STEP - 1:
raise ValueError(
f"Transcript exceeds the discarded terminal sub-step by "
f"{overrun} frames: {row.uid}"
)
discarded_terminal_frames += max(
0, dense_end - max(dense_start, grid_frames)
)
dense_end = grid_frames
if dense_start >= dense_end:
continue
first = dense_start // FRAMES_PER_TASK_STEP
last = (dense_end - 1) // FRAMES_PER_TASK_STEP
if first < 0 or last < first or last >= num_steps:
raise ValueError(
f"Transcript interval leaves target grid: {row.uid}, "
f"bins=[{first},{last}], target_steps={num_steps}"
)
destination = (
wearer
if int(row.participant_id) == wearer_id
else activity.setdefault(
int(row.participant_id),
torch.zeros(num_steps, dtype=torch.bool),
)
)
destination[first : last + 1] = True
return activity, wearer, {
"discarded_terminal_transcript_frames": discarded_terminal_frames
}
def causal_history(
sequence: torch.BoolTensor,
*,
row: int,
history_steps: int,
delay_steps: int = 0,
observed_sequence: torch.BoolTensor | None = None,
) -> torch.Tensor:
"""Return [activity, observed-mask] ending at row-delay, never after row."""
if sequence.ndim != 1 or sequence.dtype != torch.bool:
raise ValueError("Activity sequence must be bool [T]")
if observed_sequence is not None and (
observed_sequence.dtype != torch.bool
or observed_sequence.shape != sequence.shape
):
raise ValueError("Observed sequence must be bool [T] aligned to activity")
if not 0 <= row < len(sequence):
raise IndexError(row)
if history_steps <= 0 or delay_steps < 0:
raise ValueError("Invalid history/delay")
end = row - delay_steps
values = torch.zeros(history_steps, dtype=torch.float32)
observed = torch.zeros(history_steps, dtype=torch.float32)
if end >= 0:
start = max(0, end - history_steps + 1)
count = end - start + 1
available = (
torch.ones(count, dtype=torch.bool)
if observed_sequence is None
else observed_sequence[start : end + 1]
)
values[-count:] = (sequence[start : end + 1] & available).float()
observed[-count:] = available.float()
return torch.cat((values, observed), dim=0)
def visible_people_by_row(
source_payload: dict[str, Any],
cutoffs: torch.Tensor,
*,
wearer_id: int,
) -> tuple[list[list[int]], list[list[dict[str, Any]]]]:
"""Use the latest provided-box frame strictly before every state cutoff."""
persons = source_payload["persons"]
frame_times = torch.as_tensor(persons["frame_times"], dtype=torch.float64)
offsets = torch.as_tensor(persons["frame_offsets"], dtype=torch.long)
records = persons["records"]
row_to_frame = torch.searchsorted(frame_times, cutoffs, right=False) - 1
ids_by_row: list[list[int]] = []
records_by_row: list[list[dict[str, Any]]] = []
for frame_index in row_to_frame.tolist():
if frame_index < 0:
ids_by_row.append([])
records_by_row.append([])
continue
start = int(offsets[frame_index])
stop = int(offsets[frame_index + 1])
seen: set[int] = set()
row_ids: list[int] = []
row_records: list[dict[str, Any]] = []
for source_record in records[start:stop]:
raw_id = source_record.get("id")
text_id = "" if raw_id is None else str(raw_id).strip()
if text_id.lower() in UNKNOWN_PERSON_IDS:
continue
try:
participant_id = int(text_id)
except ValueError as error:
raise ValueError(f"Non-numeric EasyCom participant ID {text_id!r}") from error
if participant_id == wearer_id or participant_id in seen:
continue
seen.add(participant_id)
row_ids.append(participant_id)
row_records.append(source_record)
ids_by_row.append(row_ids)
records_by_row.append(row_records)
return ids_by_row, records_by_row
def oracle_person_rows(
activity: dict[int, torch.BoolTensor],
visibility: dict[int, torch.BoolTensor],
visible_ids: list[list[int]],
source_records: list[list[dict[str, Any]]],
*,
condition: str,
history_steps: int,
delay_steps: int,
seed: int,
) -> tuple[torch.Tensor, torch.LongTensor, list[dict[str, Any]], dict[str, int]]:
"""Create a 5 Hz ragged set of causal oracle histories."""
if condition not in ORACLE_CONDITIONS:
raise ValueError(f"Unknown oracle condition: {condition}")
if len(visible_ids) != len(source_records):
raise ValueError("Visible ID/record row counts differ")
num_steps = len(visible_ids)
zero_sequence = torch.zeros(num_steps, dtype=torch.bool)
rng = np.random.default_rng(seed)
feature_rows: list[torch.Tensor] = []
records: list[dict[str, Any]] = []
offsets = [0]
shuffled_values_changed = 0
shuffled_values_total = 0
for row, (person_ids, row_records) in enumerate(zip(visible_ids, source_records)):
if len(person_ids) != len(row_records):
raise ValueError("Visible person IDs/records differ within a row")
base = torch.stack(
[
causal_history(
activity.get(person_id, zero_sequence),
row=row,
history_steps=history_steps,
delay_steps=delay_steps if condition == "delayed" else 0,
observed_sequence=visibility.get(person_id, zero_sequence),
)
for person_id in person_ids
],
dim=0,
) if person_ids else torch.empty((0, 2 * history_steps))
if condition == "visibility-only":
output = base.clone()
output[:, :history_steps] = 0
elif condition == "aggregate-visible":
if len(base):
output = torch.cat(
(
base[:, :history_steps].amax(dim=0, keepdim=True),
base[:, history_steps:].amax(dim=0, keepdim=True),
),
dim=-1,
)
else:
output = base
elif condition == "shuffled-id":
output = base.clone()
# Independently reassign each historical time bin across the
# currently visible slots. Per-bin activity counts remain exact,
# while cross-time person continuity is destroyed.
for column in range(history_steps):
if len(base) > 1:
permutation = torch.from_numpy(rng.permutation(len(base)))
output[:, column] = base[permutation, column]
output[:, history_steps + column] = base[
permutation, history_steps + column
]
shuffled_values_changed += int(
(output[:, :history_steps] != base[:, :history_steps]).sum()
)
shuffled_values_total += int(base[:, :history_steps].numel())
else:
output = base
feature_rows.extend(output.unbind(dim=0))
if condition == "aggregate-visible" and person_ids:
records.append(
{
"id": "aggregate-visible",
"source_person_ids": person_ids,
"source_visual_times": [
float(item.get("source_time", float("nan")))
for item in row_records
],
}
)
else:
for person_id, source_record in zip(person_ids, row_records):
records.append(
{
"id": str(person_id),
"source_clip_index": int(
source_record.get("source_clip_index", 0)
),
"source_visual_time": float(
source_record.get("source_time", float("nan"))
),
}
)
offsets.append(offsets[-1] + len(output))
feature_dim = 2 * history_steps
features = (
torch.stack(feature_rows)
if feature_rows
else torch.empty((0, feature_dim), dtype=torch.float32)
)
return (
features,
torch.tensor(offsets, dtype=torch.long),
records,
{
"shuffled_values_changed": shuffled_values_changed,
"shuffled_values_total": shuffled_values_total,
},
)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--dataset-root", type=Path, required=True)
parser.add_argument("--feature-root", type=Path, required=True)
parser.add_argument("--target-root", type=Path, required=True)
parser.add_argument("--local-feature-root", type=Path, required=True)
parser.add_argument("--output-root", type=Path, required=True)
parser.add_argument("--sessions", nargs="+", default=list(ALL_SESSIONS))
parser.add_argument("--conditions", nargs="+", choices=ORACLE_CONDITIONS,
default=list(ORACLE_CONDITIONS))
parser.add_argument("--history-seconds", type=float, default=2.0)
parser.add_argument("--delay-seconds", type=float, default=1.0)
parser.add_argument("--camera-wearer-id", type=int, default=CAMERA_WEARER_ID)
parser.add_argument("--shuffle-seed", type=int, default=2027)
parser.add_argument("--max-label-mismatch-rows", type=int, default=0)
parser.add_argument("--expected-mode", default="open-stream-replay")
parser.add_argument("--overwrite", action="store_true")
parser.add_argument("--dry-run", action="store_true")
return parser.parse_args()
def main() -> None:
args = parse_args()
history_steps = exact_steps(args.history_seconds, "--history-seconds")
delay_steps = exact_steps(args.delay_seconds, "--delay-seconds")
if args.max_label_mismatch_rows < 0:
raise ValueError("--max-label-mismatch-rows cannot be negative")
sessions = [normalize_sample_name(value, "easycom") for value in args.sessions]
unknown = sorted(set(sessions) - set(ALL_SESSIONS))
if unknown:
raise ValueError(f"Unknown EasyCom sessions: {unknown}")
if len(sessions) != len(set(sessions)):
raise ValueError("Duplicate sessions")
conditions = list(dict.fromkeys(args.conditions))
rules = Rules()
print(
f"Oracle analysis only: conditions={','.join(conditions)}, "
f"history={history_steps}x{STEP_SECONDS:g}s, "
f"delay={delay_steps}x{STEP_SECONDS:g}s"
)
for session_position, session in enumerate(sessions, 1):
reference_path = resolve_feature_path(args.feature_root, session)
target_path = args.target_root / f"{session}.npy"
source_local_path = resolve_feature_path(args.local_feature_root, session)
reference = torch.load(reference_path, map_location="cpu", weights_only=False)
times, label_indices = reference_alignment(reference)
if args.expected_mode != "any" and reference.get("mode") != args.expected_mode:
raise ValueError(
f"{reference_path} mode is {reference.get('mode')!r}, "
f"expected {args.expected_mode!r}"
)
target = np.load(target_path, allow_pickle=False)
if target.shape != (len(times), NUM_CLASSES):
raise ValueError(
f"Target/reference mismatch: {target_path}, "
f"{target.shape} vs {(len(times), NUM_CLASSES)}"
)
direct_audio, _source_people, source_contract = load_frame_set_features(
source_local_path,
reference,
sample_id=session,
dataset="easycom",
)
source_payload = torch.load(
source_local_path, map_location="cpu", weights_only=False
)
audio_dim = int(source_contract["config"]["audio_dim"])
audio = direct_audio[:, :audio_dim]
audio_valid = direct_audio[:, audio_dim].bool()
clips = build_clips(args.dataset_root.resolve(), session, rules)
utterances = load_utterances(session, clips)
activity, wearer_activity, transcript_audit = participant_activity(
utterances,
num_steps=len(times),
wearer_id=args.camera_wearer_id,
)
all_other_activity = torch.zeros(len(times), dtype=torch.bool)
for sequence in activity.values():
all_other_activity |= sequence
target_tensor = torch.from_numpy(np.asarray(target))
wearer_mismatch = int(
(wearer_activity != target_tensor[:, 1].bool()).sum()
)
other_mismatch = int(
(all_other_activity != target_tensor[:, 2].bool()).sum()
)
mismatch_union = int(
(
(wearer_activity != target_tensor[:, 1].bool())
| (all_other_activity != target_tensor[:, 2].bool())
).sum()
)
if mismatch_union > args.max_label_mismatch_rows:
raise ValueError(
f"{session}: transcript/target activity differs on "
f"{mismatch_union} rows (wearer={wearer_mismatch}, "
f"others={other_mismatch}); inspect alignment or explicitly "
"raise --max-label-mismatch-rows"
)
visible_ids, source_records = visible_people_by_row(
source_payload,
times,
wearer_id=args.camera_wearer_id,
)
visibility: dict[int, torch.BoolTensor] = {
participant_id: torch.zeros(len(times), dtype=torch.bool)
for participant_id in activity
}
for row, participant_ids in enumerate(visible_ids):
for participant_id in participant_ids:
visibility.setdefault(
participant_id, torch.zeros(len(times), dtype=torch.bool)
)[row] = True
visible_counts = [len(row) for row in visible_ids]
print(
f"[{session_position}/{len(sessions)}] {session}: "
f"states={len(times)}, participants={len(activity)}, "
f"visible_rows={sum(count > 0 for count in visible_counts)}, "
f"max_visible={max(visible_counts, default=0)}, "
f"target_mismatch={mismatch_union}"
)
for condition in conditions:
destination = args.output_root / condition / f"{session}.pt"
if destination.exists() and not args.overwrite:
raise FileExistsError(
f"Refusing to overwrite {destination}; pass --overwrite"
)
condition_seed = (
args.shuffle_seed
+ 1009 * session_position
+ ORACLE_CONDITIONS.index(condition)
)
features, frame_offsets, records, corruption = oracle_person_rows(
activity,
visibility,
visible_ids,
source_records,
condition=condition,
history_steps=history_steps,
delay_steps=delay_steps,
seed=condition_seed,
)
visual = torch.zeros((len(times), 2 * history_steps))
visual_valid = torch.zeros(len(times), dtype=torch.bool)
for row in range(len(times)):
start = int(frame_offsets[row])
stop = int(frame_offsets[row + 1])
if stop > start:
visual[row] = features[start:stop].mean(dim=0)
visual_valid[row] = True
config = {
"oracle_analysis": True,
"uses_speech_labels": True,
"causal_policy": "fully-observed-half-open-5hz-bins",
"oracle_condition": condition,
"oracle_scope": "currently-visible-nonwearer-participants",
"history_steps": history_steps,
"history_seconds": history_steps * STEP_SECONDS,
"delay_steps": delay_steps if condition == "delayed" else 0,
"delay_seconds": (
delay_steps * STEP_SECONDS if condition == "delayed" else 0.0
),
"shuffle_seed": args.shuffle_seed if condition == "shuffled-id" else None,
"speaker_activity_source": "EasyCom Speech_Transcriptions inclusive 20Hz intervals",
"visibility_source": "provided face boxes in source local sidecar",
"camera_wearer_id": args.camera_wearer_id,
"visual_dim": 2 * history_steps,
"audio_dim": audio_dim,
"source_local_contract": source_contract,
}
payload = {
"format": ORACLE_FORMAT,
"dataset": "easycom",
"sample_id": session,
"times": times,
"label_indices": label_indices,
"partial": False,
"visual": visual,
"audio": audio,
"visual_valid": visual_valid,
"audio_valid": audio_valid,
"persons": {
"features": features,
# Bin k covers [k/5,(k+1)/5); its feature is available at
# the matching state cutoff (k+1)/5.
"frame_times": torch.arange(len(times), dtype=torch.float64)
* STEP_SECONDS,
"frame_offsets": frame_offsets,
"records": records,
},
"config": config,
"reference_features": str(reference_path.resolve()),
"source_local_features": str(source_local_path.resolve()),
"audit": {
"strictly_past_or_current_bins_only": True,
"transcript_target_mismatch_rows": mismatch_union,
"wearer_mismatch_rows": wearer_mismatch,
"others_mismatch_rows": other_mismatch,
"rows_with_visible_people": sum(
count > 0 for count in visible_counts
),
"max_visible_people": max(visible_counts, default=0),
"person_observations": len(features),
"generation_seed": condition_seed,
**transcript_audit,
**corruption,
},
}
if args.dry_run:
print(
f" {condition}: would write {destination}; "
f"observations={len(features)}"
)
continue
destination.parent.mkdir(parents=True, exist_ok=True)
with tempfile.NamedTemporaryFile(
dir=destination.parent,
prefix=f".{session}.",
suffix=".tmp",
delete=False,
) as handle:
temporary = Path(handle.name)
try:
torch.save(payload, temporary)
os.replace(temporary, destination)
finally:
temporary.unlink(missing_ok=True)
print(
f" {condition}: saved {destination}; "
f"observations={len(features)}"
)
if __name__ == "__main__":
main()

1
results/README.md
View File

@@ -14,6 +14,7 @@ readers to the right comparison.
| [`easycom_loss_structure_4ep_2026-08-26/`](easycom_loss_structure_4ep_2026-08-26/README.md) | LSTR-MCE versus independent BCE versus four-state CE |
| [`easycom_transition_auxiliary_4ep_2026-08-27/`](easycom_transition_auxiliary_4ep_2026-08-27/README.md) | Supervised transition head and expected-reward objectives |
| [`easycom_encoder_only_1s_4ep_2026-08-28/`](easycom_encoder_only_1s_4ep_2026-08-28/README.md) | Encoder-only A/V/AV versus global-Thinker local-feature baselines |
| [`easycom_track_gru_residual_4ep_2026-09-07/`](easycom_track_gru_residual_4ep_2026-09-07/README.md) | Matched pooled/anonymous-track-GRU screen and near-zero residual-gate diagnosis |
## Interpretation rules

116
results/easycom_track_gru_residual_4ep_2026-09-07/README.md Normal file
View File

@@ -0,0 +1,116 @@
# EasyCom anonymous track-GRU residual: matched four-epoch screen
## Question
Does causal continuity within anonymous face tracks improve future floor-state
prediction beyond the same-seed `G+A+pooled-V` head?
This experiment compares the matched pooled checkpoint with a residual branch
containing a shared per-track GRU and a `G+A` query over current track states.
The pooled path is copied exactly and frozen. The residual is multiplied by a
learned scalar initialized at zero.
## Protocol and provenance
- Global features: `open_stream_2fps_8s_last4_stack`.
- Local sidecars: `local_qwen_1s_encoded_tf4576`.
- Training: EasyCom Sessions 4--11.
- Development/model selection: Session 12.
- Official test: Sessions 1--3.
- Development seeds: 0, 1 and 2; four epochs.
- Test seeds observed: seed 0 only.
- Loss: `lstr-mce`; layer fusion: `concat-mlp`.
- Person history: 16 observations at 2 FPS, with a 1-second maximum track gap.
- Reported table protocol: `causal_valid_only`; all values are percentage
points.
- Checkpoints: `checkpoints/easycom_track_gru_residual/` on the remote L20
machine.
- Predictions: `results/easycom_track_gru_residual/` for development and
`results/easycom_track_gru_residual_test_check/` for the one-seed test audit.
The test audit was run before the residual optimization defect described below
was diagnosed. It is recorded for transparency and must not be used to tune a
corrected architecture.
## Session 12 development result
| Run | 0.2 s mAP | BG AP | wearer AP | others AP | 10-horizon mean mAP |
|---|---:|---:|---:|---:|---:|
| pooled seed 0 | 72.91519 | 58.39896 | 68.44729 | 91.89933 | 59.42976 |
| track-GRU seed 0 | 72.91538 | 58.39965 | 68.44750 | 91.89900 | 59.42956 |
| pooled seed 1 | 73.02189 | 57.73258 | 69.35309 | 91.97999 | 59.50838 |
| track-GRU seed 1 | 73.02358 | 57.74019 | 69.35432 | 91.97624 | 59.50736 |
| pooled seed 2 | 72.45010 | 57.26123 | 68.35503 | 91.73403 | 59.53730 |
| track-GRU seed 2 | 72.47576 | 57.15675 | 68.59149 | 91.67904 | 59.57771 |
Three-seed paired `track-GRU - pooled` deltas, mean +/- sample standard
deviation:
| Metric | Delta (percentage points) |
|---|---:|
| 0.2 s mAP | +0.00918 +/- 0.01429 |
| BG AP | -0.03206 +/- 0.06281 |
| wearer AP | +0.07930 +/- 0.13610 |
| others AP | -0.01969 +/- 0.03062 |
| 10-horizon mean mAP | +0.01306 +/- 0.02368 |
The development difference is effectively zero and is much smaller than the
between-seed variation.
## Sessions 1--3 test audit
Only seed 0 was evaluated after the matched checkpoints had already been
selected on Session 12.
| Run | 0.2 s mAP | BG AP | wearer AP | others AP | 10-horizon mean mAP |
|---|---:|---:|---:|---:|---:|
| pooled seed 0 | 81.74161 | 68.57326 | 84.71790 | 91.93366 | 64.88810 |
| track-GRU seed 0 | 81.74160 | 68.57367 | 84.71783 | 91.93330 | 64.88780 |
| track-GRU - pooled | -0.00001 | +0.00041 | -0.00007 | -0.00037 | -0.00030 |
Under the EgoSpeak full-timeline printout, the corresponding ten-horizon means
were 64.87493 for pooled and 64.87463 for track-GRU. The small difference from
the table is due to the evaluation mask; protocols must not be mixed.
## Gate diagnostic
The selected track checkpoints retained an almost closed residual gate:
| Seed | Selected epoch | saved `tanh(gate)` | selection-time dev mean mAP |
|---|---:|---:|---:|
| 0 | 1 | -0.00152618 | 59.45070 |
| 1 | 1 | +0.00409810 | 59.52841 |
| 2 | 2 | +0.01081575 | 59.59214 |
The checkpoint JSON contains history only through the epoch at which that best
checkpoint was written, so it is not a complete log of later non-improving
epochs. The values above nevertheless describe the models used for inference.
With logits
```text
base_logits + tanh(gate) * delta(track_parameters)
```
the zero-initialized gate makes the first-step gradient of all track/delta
parameters exactly zero. Only the scalar gate can move initially; subsequent
branch gradients are scaled by its still-small value. The saved gates and the
near-identical test predictions are therefore consistent with an optimization
failure in the residual parameterization.
## Interpretation and decision
- The implemented zero-gated track-GRU produces no meaningful dense-AP gain.
- This run does **not** establish that track continuity is uninformative,
because the residual branch was almost closed throughout the selected
models.
- It also does not support event-level or visual-grounding claims: no corrected
competing-risk event evaluation or wrong-track/time-shift control was run.
- The appropriate next diagnostic is to remove the scalar gate, zero-initialize
the last residual output layer, keep the pooled base frozen, and verify that
residual-output and GRU gradients become nonzero during optimization.
- Tune that corrected model on Session 12 only. Do not inspect Sessions 1--3
again until its architecture and stopping rule are frozen.
Machine-readable values are in [`summary.json`](summary.json) and per-run rows
are in [`metrics.csv`](metrics.csv).

9
results/easycom_track_gru_residual_4ep_2026-09-07/metrics.csv Normal file
View File

@@ -0,0 +1,9 @@
split,protocol,variant,seed,map_0p2,bg_ap_0p2,wearer_ap_0p2,others_ap_0p2,mean_map,best_epoch,saved_track_gate
dev_session12,causal_valid_only,pooled,0,72.91519,58.39896,68.44729,91.89933,59.42976,,
dev_session12,causal_valid_only,track_gru,0,72.91538,58.39965,68.44750,91.89900,59.42956,1,-0.0015261819353327155
dev_session12,causal_valid_only,pooled,1,73.02189,57.73258,69.35309,91.97999,59.50838,,
dev_session12,causal_valid_only,track_gru,1,73.02358,57.74019,69.35432,91.97624,59.50736,1,0.004098099190741777
dev_session12,causal_valid_only,pooled,2,72.45010,57.26123,68.35503,91.73403,59.53730,,
dev_session12,causal_valid_only,track_gru,2,72.47576,57.15675,68.59149,91.67904,59.57771,2,0.010815753601491451
test_sessions1_3,causal_valid_only,pooled,0,81.74161,68.57326,84.71790,91.93366,64.88810,,
test_sessions1_3,causal_valid_only,track_gru,0,81.74160,68.57367,84.71783,91.93330,64.88780,1,-0.0015261819353327155
1 split protocol variant seed map_0p2 bg_ap_0p2 wearer_ap_0p2 others_ap_0p2 mean_map best_epoch saved_track_gate
2 dev_session12 causal_valid_only pooled 0 72.91519 58.39896 68.44729 91.89933 59.42976
3 dev_session12 causal_valid_only track_gru 0 72.91538 58.39965 68.44750 91.89900 59.42956 1 -0.0015261819353327155
4 dev_session12 causal_valid_only pooled 1 73.02189 57.73258 69.35309 91.97999 59.50838
5 dev_session12 causal_valid_only track_gru 1 73.02358 57.74019 69.35432 91.97624 59.50736 1 0.004098099190741777
6 dev_session12 causal_valid_only pooled 2 72.45010 57.26123 68.35503 91.73403 59.53730
7 dev_session12 causal_valid_only track_gru 2 72.47576 57.15675 68.59149 91.67904 59.57771 2 0.010815753601491451
8 test_sessions1_3 causal_valid_only pooled 0 81.74161 68.57326 84.71790 91.93366 64.88810
9 test_sessions1_3 causal_valid_only track_gru 0 81.74160 68.57367 84.71783 91.93330 64.88780 1 -0.0015261819353327155

61
results/easycom_track_gru_residual_4ep_2026-09-07/summary.json Normal file
View File

@@ -0,0 +1,61 @@
{
"record": "easycom_track_gru_residual_4ep_2026-09-07",
"dataset": "easycom",
"global_features": "open_stream_2fps_8s_last4_stack",
"local_features": "local_qwen_1s_encoded_tf4576",
"train_sessions": [4, 5, 6, 7, 8, 9, 10, 11],
"dev_sessions": [12],
"test_sessions": [1, 2, 3],
"epochs": 4,
"loss": "lstr-mce",
"layer_fusion": "concat-mlp",
"person_history_frames": 16,
"person_history_fps": 2.0,
"person_track_max_gap_seconds": 1.0,
"protocol": "causal_valid_only",
"units": "percentage_points",
"dev": {
"seeds": [0, 1, 2],
"paired_delta_track_minus_pooled": {
"map_0p2": {"mean": 0.00918, "sample_std": 0.01429},
"bg_ap_0p2": {"mean": -0.03206, "sample_std": 0.06281},
"wearer_ap_0p2": {"mean": 0.07930, "sample_std": 0.13610},
"others_ap_0p2": {"mean": -0.01969, "sample_std": 0.03062},
"mean_map": {"mean": 0.01306, "sample_std": 0.02368}
}
},
"test_audit": {
"seeds": [0],
"pooled": {
"map_0p2": 81.74161,
"bg_ap_0p2": 68.57326,
"wearer_ap_0p2": 84.71790,
"others_ap_0p2": 91.93366,
"mean_map": 64.88810,
"full_timeline_mean_map": 64.87493
},
"track_gru": {
"map_0p2": 81.74160,
"bg_ap_0p2": 68.57367,
"wearer_ap_0p2": 84.71783,
"others_ap_0p2": 91.93330,
"mean_map": 64.88780,
"full_timeline_mean_map": 64.87463
},
"paired_delta_track_minus_pooled": {
"map_0p2": -0.00001,
"bg_ap_0p2": 0.00041,
"wearer_ap_0p2": -0.00007,
"others_ap_0p2": -0.00037,
"mean_map": -0.00030
},
"warning": "Observed before diagnosing residual optimization; do not use to tune the corrected architecture."
},
"selected_track_checkpoints": [
{"seed": 0, "best_epoch": 1, "saved_tanh_gate": -0.0015261819353327155, "selection_dev_mean_map": 0.5945069810061214},
{"seed": 1, "best_epoch": 1, "saved_tanh_gate": 0.004098099190741777, "selection_dev_mean_map": 0.5952841235072595},
{"seed": 2, "best_epoch": 2, "saved_tanh_gate": 0.010815753601491451, "selection_dev_mean_map": 0.5959213772894789}
],
"decision": "No meaningful gain. Treat as a zero-gate optimization failure, not evidence against track continuity.",
"next_step": "Remove scalar gate, zero-initialize the final residual layer, verify nonzero branch gradients, and screen only on Session 12."
}

191
run_easycom_oracle_information.sh Executable file
View File

@@ -0,0 +1,191 @@
#!/usr/bin/env bash
set -euo pipefail
stage="${1:-screen}"
shift || true
if (( $# )); then
seeds=("$@")
else
seeds=(0)
fi
: "${DATASET_ROOT:?Set DATASET_ROOT to the extracted EasyCom root}"
: "${FEATURE_ROOT:?Set FEATURE_ROOT to the verified global Omni features}"
: "${TARGET_ROOT:?Set TARGET_ROOT to EasyCom target_perframe files}"
: "${LOCAL_ROOT:?Set LOCAL_ROOT to the complete Qwen local AV sidecars}"
ORACLE_ROOT="${ORACLE_ROOT:-features/easycom_oracle_information_2s}"
CHECKPOINT_ROOT="${CHECKPOINT_ROOT:-checkpoints/easycom_oracle_information_4ep}"
RESULT_ROOT="${RESULT_ROOT:-results/easycom_oracle_information_4ep}"
TRAIN_SESSIONS="${TRAIN_SESSIONS:-4 5 6 7 8 9 10 11 12}"
DEV_SESSIONS="${DEV_SESSIONS:-12}"
SCREEN_SESSIONS="${SCREEN_SESSIONS:-12}"
TEST_SESSIONS="${TEST_SESSIONS:-1 2 3}"
EPOCHS="${EPOCHS:-4}"
WARMUP_EPOCHS="${WARMUP_EPOCHS:-1}"
LAYER_FUSION="${LAYER_FUSION:-concat-mlp}"
EXPECTED_MODE="${EXPECTED_MODE:-open-stream-replay}"
VARIANTS="${VARIANTS:-audio visibility-only aggregate-visible per-person shuffled-id delayed}"
ORACLE_CONDITIONS="${ORACLE_CONDITIONS:-visibility-only aggregate-visible per-person shuffled-id delayed}"
PYTHON_BIN="${PYTHON_BIN:-python}"
read -r -a train_sessions <<<"${TRAIN_SESSIONS}"
read -r -a dev_sessions <<<"${DEV_SESSIONS}"
read -r -a screen_sessions <<<"${SCREEN_SESSIONS}"
read -r -a test_sessions <<<"${TEST_SESSIONS}"
read -r -a variants <<<"${VARIANTS}"
read -r -a oracle_conditions <<<"${ORACLE_CONDITIONS}"
layer_args=()
if [[ "${LAYER_FUSION}" != "single" ]]; then
layer_args=(--layer-fusion "${LAYER_FUSION}")
fi
prepare_oracles() {
"${PYTHON_BIN}" prepare_easycom_oracle_features.py \
--dataset-root "${DATASET_ROOT}" \
--feature-root "${FEATURE_ROOT}" \
--target-root "${TARGET_ROOT}" \
--local-feature-root "${LOCAL_ROOT}" \
--output-root "${ORACLE_ROOT}" \
--conditions "${oracle_conditions[@]}" \
--history-seconds 2.0 \
--delay-seconds 1.0 \
--expected-mode "${EXPECTED_MODE}"
}
variant_local_args() {
local variant="$1"
if [[ "${variant}" == "audio" ]]; then
printf '%s\0' \
--local-feature-root "${LOCAL_ROOT}" \
--local-modalities audio \
--local-fusion pooled
else
printf '%s\0' \
--local-feature-root "${ORACLE_ROOT}/${variant}" \
--local-modalities av \
--local-fusion oracle-frame-set-attention \
--person-attention-dim 64 \
--person-attention-heads 4
fi
}
train_and_screen() {
mkdir -p "${CHECKPOINT_ROOT}" "${RESULT_ROOT}"
for seed in "${seeds[@]}"; do
for variant in "${variants[@]}"; do
checkpoint="${CHECKPOINT_ROOT}/${variant}_seed${seed}.pth"
result="${RESULT_ROOT}/${variant}_seed${seed}"
local_feature_root="${ORACLE_ROOT}/${variant}"
if [[ "${variant}" == "audio" ]]; then
local_feature_root="${LOCAL_ROOT}"
fi
if [[ -e "${checkpoint}" || -e "${result}" ]]; then
echo "Refusing to overwrite ${checkpoint} or ${result}" >&2
exit 1
fi
local_args=()
while IFS= read -r -d '' item; do
local_args+=("${item}")
done < <(variant_local_args "${variant}")
echo "$(date): training ${variant}, seed=${seed}"
"${PYTHON_BIN}" train_turn_head.py \
--feature-root "${FEATURE_ROOT}" \
--target-root "${TARGET_ROOT}" \
--output "${checkpoint}" \
--sessions "${train_sessions[@]}" \
--dev-sessions "${dev_sessions[@]}" \
--epochs "${EPOCHS}" \
--batch-size 512 \
--learning-rate 7e-5 \
--weight-decay 5e-5 \
--warmup-epochs "${WARMUP_EPOCHS}" \
--warmup-factor 0.3 \
--dropout 0.2 \
--loss lstr-mce \
"${layer_args[@]}" \
"${local_args[@]}" \
--local-projection-dim 256 \
--expected-mode "${EXPECTED_MODE}" \
--seed "${seed}" \
--device cuda:0
echo "$(date): screening ${variant}, seed=${seed} on ${screen_sessions[*]}"
"${PYTHON_BIN}" infer_turn_head.py \
--feature-root "${FEATURE_ROOT}" \
--target-root "${TARGET_ROOT}" \
--checkpoint "${checkpoint}" \
--output-dir "${result}" \
--sessions "${screen_sessions[@]}" \
--local-feature-root "${local_feature_root}" \
--expected-mode "${EXPECTED_MODE}" \
--device cuda:0
done
done
"${PYTHON_BIN}" summarize_local_baselines.py \
--root "${RESULT_ROOT}" \
--dataset easycom \
--protocol causal_valid_only
}
test_selected() {
mkdir -p "${RESULT_ROOT}_test"
for seed in "${seeds[@]}"; do
for variant in "${variants[@]}"; do
checkpoint="${CHECKPOINT_ROOT}/${variant}_seed${seed}.pth"
result="${RESULT_ROOT}_test/${variant}_seed${seed}"
local_feature_root="${ORACLE_ROOT}/${variant}"
if [[ "${variant}" == "audio" ]]; then
local_feature_root="${LOCAL_ROOT}"
fi
if [[ ! -f "${checkpoint}" ]]; then
echo "Missing checkpoint: ${checkpoint}" >&2
exit 1
fi
if [[ -e "${result}" ]]; then
echo "Refusing to overwrite ${result}" >&2
exit 1
fi
"${PYTHON_BIN}" infer_turn_head.py \
--feature-root "${FEATURE_ROOT}" \
--target-root "${TARGET_ROOT}" \
--checkpoint "${checkpoint}" \
--output-dir "${result}" \
--sessions "${test_sessions[@]}" \
--local-feature-root "${local_feature_root}" \
--expected-mode "${EXPECTED_MODE}" \
--device cuda:0
done
done
"${PYTHON_BIN}" summarize_local_baselines.py \
--root "${RESULT_ROOT}_test" \
--dataset easycom \
--protocol causal_valid_only
}
case "${stage}" in
prepare)
prepare_oracles
;;
screen)
train_and_screen
;;
all)
prepare_oracles
train_and_screen
;;
test)
test_selected
;;
summarize)
"${PYTHON_BIN}" summarize_local_baselines.py \
--root "${RESULT_ROOT}" \
--dataset easycom \
--protocol causal_valid_only
;;
*)
echo "Usage: $0 {prepare|screen|all|test|summarize} [seed ...]" >&2
exit 2
;;
esac

199
test_easycom_oracle_features.py Normal file
View File

@@ -0,0 +1,199 @@
"""Focused tests for the causal EasyCom oracle-information experiment."""
from __future__ import annotations
from pathlib import Path
from types import SimpleNamespace
import tempfile
import unittest
import numpy as np
import torch
from easycom_head import TurnPredictionHead
from local_turn_features import ORACLE_FORMAT, load_frame_set_features, load_local_features
from prepare_easycom_oracle_features import (
causal_history,
oracle_person_rows,
participant_activity,
)
class OracleFeatureTests(unittest.TestCase):
def setUp(self) -> None:
torch.set_num_threads(1)
def test_transcript_intervals_are_rasterized_into_half_open_bins(self) -> None:
rows = [
SimpleNamespace(
uid="a",
participant_id=3,
dense_start_frame=3,
dense_end_frame_exclusive=5,
),
SimpleNamespace(
uid="b",
participant_id=2,
dense_start_frame=8,
dense_end_frame_exclusive=12,
),
]
activity, wearer, audit = participant_activity(
rows, num_steps=4, wearer_id=2
)
self.assertEqual(activity[3].tolist(), [True, True, False, False])
self.assertEqual(wearer.tolist(), [False, False, True, False])
self.assertEqual(audit["discarded_terminal_transcript_frames"], 0)
def test_history_at_row_does_not_read_later_activity(self) -> None:
first = torch.tensor([0, 1, 0, 0, 0], dtype=torch.bool)
second = first.clone()
second[3:] = True
observed = causal_history(first, row=2, history_steps=4)
changed_future = causal_history(second, row=2, history_steps=4)
self.assertTrue(torch.equal(observed, changed_future))
self.assertEqual(
observed.tolist(),
[0, 0, 1, 0, 0, 1, 1, 1],
)
delayed = causal_history(second, row=2, history_steps=4, delay_steps=2)
self.assertEqual(
delayed.tolist(),
[0, 0, 0, 0, 0, 0, 0, 1],
)
visibility = torch.tensor([1, 0, 1, 1, 1], dtype=torch.bool)
visible_only = causal_history(
second,
row=2,
history_steps=4,
observed_sequence=visibility,
)
self.assertEqual(
visible_only.tolist(),
[0, 0, 0, 0, 0, 1, 0, 1],
)
def test_shuffled_id_preserves_every_lag_count_but_breaks_histories(self) -> None:
activity = {
3: torch.tensor([1, 1, 0, 0, 1, 1], dtype=torch.bool),
4: torch.tensor([0, 0, 1, 1, 0, 0], dtype=torch.bool),
5: torch.tensor([0, 1, 0, 1, 0, 1], dtype=torch.bool),
}
visible = [[3, 4, 5] for _ in range(6)]
visibility = {
person_id: torch.ones(6, dtype=torch.bool)
for person_id in activity
}
records = [
[
{"id": str(person_id), "source_clip_index": 0, "source_time": row / 5}
for person_id in person_ids
]
for row, person_ids in enumerate(visible)
]
original, offsets, _, _ = oracle_person_rows(
activity,
visibility,
visible,
records,
condition="per-person",
history_steps=4,
delay_steps=5,
seed=9,
)
shuffled, shuffled_offsets, _, audit = oracle_person_rows(
activity,
visibility,
visible,
records,
condition="shuffled-id",
history_steps=4,
delay_steps=5,
seed=9,
)
self.assertTrue(torch.equal(offsets, shuffled_offsets))
for row in range(6):
start, stop = int(offsets[row]), int(offsets[row + 1])
self.assertTrue(
torch.equal(
original[start:stop, :4].sum(dim=0),
shuffled[start:stop, :4].sum(dim=0),
)
)
self.assertGreater(audit["shuffled_values_changed"], 0)
def test_oracle_format_requires_explicit_loader_and_head_mode(self) -> None:
n = 4
times = (torch.arange(n, dtype=torch.float64) + 1) / 5
reference = {
"states": torch.randn(n, 8),
"label_indices": torch.arange(n),
"times": times,
"mode": "open-stream-replay",
"start": 0,
}
sidecar = {
"format": ORACLE_FORMAT,
"dataset": "easycom",
"sample_id": "Session_4",
"times": times.clone(),
"label_indices": torch.arange(n),
"partial": False,
"visual": torch.randn(n, 6),
"audio": torch.randn(n, 5),
"visual_valid": torch.ones(n, dtype=torch.bool),
"audio_valid": torch.ones(n, dtype=torch.bool),
"persons": {
"features": torch.randn(n, 6),
"frame_times": torch.arange(n, dtype=torch.float64) / 5,
"frame_offsets": torch.arange(n + 1, dtype=torch.long),
"records": [{"id": "3"} for _ in range(n)],
},
"config": {
"oracle_analysis": True,
"uses_speech_labels": True,
"causal_policy": "fully-observed-half-open-5hz-bins",
"oracle_condition": "per-person",
"visual_dim": 6,
"audio_dim": 5,
},
}
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "Session_4.pt"
torch.save(sidecar, path)
with self.assertRaisesRegex(ValueError, "format"):
load_local_features(
path,
reference,
sample_id="Session_4",
dataset="easycom",
modalities="av",
)
audio, people, contract = load_frame_set_features(
path,
reference,
sample_id="Session_4",
dataset="easycom",
oracle_analysis=True,
)
self.assertEqual(audio.shape, (n, 6))
self.assertEqual(people.dimension, 6)
self.assertEqual(contract["format"], ORACLE_FORMAT)
head = TurnPredictionHead(
hidden_size=8,
num_horizons=2,
num_classes=3,
local_visual_dim=6,
local_audio_dim=5,
local_projection_dim=4,
local_fusion="oracle-frame-set-attention",
person_attention_dim=4,
person_attention_heads=1,
)
padded, mask = people.padded()
logits = head(reference["states"], audio, padded, mask)
self.assertEqual(logits.shape, (n, 2, 3))
if __name__ == "__main__":
unittest.main()