diff --git a/PROJECT_CONTEXT.md b/PROJECT_CONTEXT.md index c3f6de6..df8b343 100644 --- a/PROJECT_CONTEXT.md +++ b/PROJECT_CONTEXT.md @@ -6,9 +6,10 @@ reference; it is not the experiment scheduler. ## One-sentence objective -Determine whether strictly causal, speaker-grounded audiovisual evidence from -Qwen2.5-Omni can improve an invisible egocentric wearer's **floor-entry** and -**floor-yield** behavior in natural multiparty conversation. +Determine whether strictly causal global Omni context plus local audiovisual +floor evidence organized around visible people can improve an invisible +egocentric wearer's **floor-entry** and **floor-yield** behavior in natural +multiparty conversation. The paper is not simply “replace EgoSpeak's encoders with a newer backbone.” The decisive test is whether correct person-aligned vision reduces false @@ -19,6 +20,7 @@ recall, without hurting addressed responses or yield behavior. | Need | Authoritative file | |---|---| +| Current paper thesis and claim boundary | [docs/global_local_omni_paper_position_2026-09-07.md](docs/global_local_omni_paper_position_2026-09-07.md) | | Progressive feature-fusion implementation route | [docs/progressive_feature_fusion_route_2026-09-02.md](docs/progressive_feature_fusion_route_2026-09-02.md) | | Current hypotheses, experiment gates and stopping rules | [`docs/omni_speaker_floor_experiment_plan_2026-08-30.md`](docs/omni_speaker_floor_experiment_plan_2026-08-30.md) | | Novelty boundary and claim restrictions | [`docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md`](docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md) | @@ -72,6 +74,19 @@ result. [`result record`](results/easycom_track_gru_residual_4ep_2026-09-07/README.md) and [`implementation note`](docs/easycom_track_gru_residual.md). +### Measured oracle-information screen + +- A seed-0, four-epoch Session-12 development screen supplied transcript-derived + current/past speech activity only through the explicit analysis-only oracle + contract. `aggregate-visible` improved over the matched local-audio baseline + by `+6.96649` points at 0.2 seconds and `+2.91047` points in ten-horizon mean + mAP. Visibility-only and 1-second-delayed controls stayed near audio. +- Person identity continuity is not supported: `aggregate-visible` exceeded + `per-person`, while `per-person` and `shuffled-id` were effectively tied. + The result motivates a learned causal local active-speaker/floor-evidence + branch, but does not justify a track-continuity claim. See the immutable + [`result record`](results/easycom_oracle_information_4ep_2026-09-07/README.md). + ### Current blocker `evaluate_floor_state_changes.py` counts a wearer transition anywhere in a @@ -91,10 +106,12 @@ unfinished. This does not relax the paper claim: event numbers remain proxy diagnostics until competing-risk censoring is fixed. 1. **P0:** fix/freeze event attribution, splits and scenario definitions. -2. **P1:** probe aggregate activity and per-person active-speaker grounding; +2. **P1:** confirm the aggregate/per-person oracle screen over seeds 1--2, + then train a causal visible-person active-speaker/floor-evidence estimator; compare correct, wrong-person and time-shifted visual inputs. 3. **P2:** re-evaluate `G`, `G+A`, `G+A+pooled-V` under the frozen event protocol. -4. **P3:** repair the matched anonymous-track-GRU residual optimization by +4. **P3:** only if learned person-organized evidence beats aggregate/no-ID + controls, repair the matched anonymous-track-GRU residual optimization by replacing the scalar zero gate with a zero-initialized residual output layer, and verify nonzero branch gradients on Session 12. Add broken-ID, wrong-time and remove-person controls only if that corrected screen improves. diff --git a/docs/README.md b/docs/README.md index cef2cc2..8f0e3ea 100644 --- a/docs/README.md +++ b/docs/README.md @@ -7,6 +7,7 @@ implemented or validated. | Document | Role | |---|---| +| [global_local_omni_paper_position_2026-09-07.md](global_local_omni_paper_position_2026-09-07.md) | Evidence-backed paper thesis after the causal oracle information screen | | [progressive_feature_fusion_route_2026-09-02.md](progressive_feature_fusion_route_2026-09-02.md) | Gated route from pooled local evidence to Frame-set Attention, temporal person memory and Thinker adapters | | [`omni_speaker_floor_experiment_plan_2026-08-30.md`](omni_speaker_floor_experiment_plan_2026-08-30.md) | Current hypotheses, experiment gates, main table and stopping rules | | [`multiparty_egocentric_floor_gap_audit_2026-08-30.md`](multiparty_egocentric_floor_gap_audit_2026-08-30.md) | Novelty boundary, closest work and restricted claims | diff --git a/docs/omni_speaker_floor_experiment_plan_2026-08-30.md b/docs/omni_speaker_floor_experiment_plan_2026-08-30.md index 4f90bde..232f9f4 100644 --- a/docs/omni_speaker_floor_experiment_plan_2026-08-30.md +++ b/docs/omni_speaker_floor_experiment_plan_2026-08-30.md @@ -7,13 +7,14 @@ GRU/Transformer”作为最终主线。它只是一个 representation baseline 主问题应收敛为： -> 严格因果的 Omni 模型是否真正编码了多人场景中的“谁在何时说话、话语是否面向 -> wearer”这类视听 grounding；把这种逐人证据加入预测后，能否使不可见的 camera -> wearer 更合理地进入或让出话轮？ +> 严格因果的全局 Omni state 是否保留了多人场景中及时的局部视听 floor evidence；用 +> 人物 crop 组织的局部分支补充这种信息后，能否使不可见的 camera wearer 更合理地进入 +> 或让出话轮？进一步保留逐人 track continuity 是否真的提供 aggregate activity 之外的 +> 增量，是实验问题而不是预设。 最重要的可证伪目标是： -> 在相同 wearer 开口事件召回率下，正确对齐的逐人视觉应减少 other↔other 交谈期间 +> 在相同 wearer 开口事件召回率下，正确对齐的局部视听证据应减少 other↔other 交谈期间 > 的错误开口预测；同时，不能降低对真正 addressed-to-wearer 接话事件的召回率，也 > 不能恶化 wearer 正在讲话时的让话预测。 @@ -34,7 +35,7 @@ grounding，但它们尚未完整覆盖以下交集： - 使用逐人/对象指向的视觉证据； - 用事件召回、错误开口率和响应延迟，而不只用逐帧未来活动 mAP。 -更稳妥的论文表述是： +只有逐人控制通过后，才可使用下面的强表述： > To our knowledge, this is the first study of strictly causal, > speaker-grounded audiovisual forecasting of both floor-entry and floor-yield @@ -45,6 +46,10 @@ grounding，但它们尚未完整覆盖以下交集： “首个在线多人 EOT”。完整边界见 [多人 egocentric floor gap audit](multiparty_egocentric_floor_gap_audit_2026-08-30.md)。 +在当前 oracle 结果下，更稳妥的主张是研究 **global Omni context 与 causal local AV +floor evidence 的互补性**。人物框/track 暂时只是组织局部观测的接口，不能写成已验证的 +speaker grounding 或 identity-aware reasoning。 + ### 不足以成为贡献的内容 - 仅把 EgoSpeak 的旧 backbone 换成 Qwen2.5-Omni； @@ -62,13 +67,13 @@ grounding，但它们尚未完整覆盖以下交集： | 假设 | 问题 | 可观察证据 | |---|---|---| | H1 | 全局 Omni state 是否包含 aggregate floor 信息？ | wearer/others 活动与未来活动 probe 优于简单先验和 encoder-only baseline | -| H2 | 全局 Omni state 是否可靠地区分画面中的不同说话人？ | 匿名 person-slot active-speaker probe 明显优于随机/时序错位视觉 | -| H3 | 逐人 grounding 是否改变 wearer 的开口决策？ | matched onset recall 下，other↔other 时段 false entry/min 显著下降 | +| H2 | 真实因果 AV 能否恢复 oracle 所显示的 timely local floor evidence？ | learned local branch 优于 audio、visibility、错人/错时和参数匹配控制，并恢复可报告比例的 oracle gap | +| H3 | 逐人 continuity 是否提供 aggregate evidence 之外的增量？ | correct track 稳定优于 aggregate、no-ID、shuffled/broken-ID 控制 | | H4 | 这种抑制是否只是让模型更保守？ | addressed-to-wearer response recall/latency 和 wearer yield 不退化 | -如果 H2 不成立，论文不能宣称原始 Omni 已经具有可靠 speaker grounding；这时逐人分支 -本身就是所需的适配模块。如果 H2 成立但 H3 不成立，则 representation 有说话者信息， -但它没有转化为 floor-action 收益，也不能强行讲成成功故事。 +如果 H2 不成立，论文不能宣称局部 AV 分支解决了 Omni 的 floor-evidence 缺口；应先审计 +编码和时间对齐。如果 H2 成立但 H3 不成立，则采用简单 aggregate/person-set 分支，并 +明确 track identity 没有增量，不能强行讲成 speaker-continuity 故事。 ## 4. 实验顺序 @@ -119,6 +124,16 @@ P1 的通过条件不是一个固定绝对阈值，而是：正确对齐的人 且三次随机种子置信区间不完全被简单 pooled baseline 覆盖。否则不要进入昂贵的 Thinker adapter 实验。 +2026-09-07 的 seed-0 Session-12 oracle screen 给出了更精确的门控信号：完全对齐的 +`aggregate-visible` 活动相对 local-audio baseline 提升 `+6.96649` 个点的 0.2 秒 mAP 和 +`+2.91047` 个点的十 horizon mean mAP，而 visibility-only 和延迟 1 秒的活动接近 audio。 +这支持“及时的局部 floor evidence 有信息价值”。但是 `aggregate-visible` 高于 +`per-person`，且 `per-person` 与 `shuffled-id` 基本相同，因此当前结果不支持人物 ID +连续性。下一步优先学习可部署的 causal visible-person active-speaker/floor estimator；只有 +正确逐人证据稳定优于 aggregate、no-ID、broken-continuity 和错时控制，才进入 track-aware +主方法。完整数值见 +[`oracle result record`](../results/easycom_oracle_information_4ep_2026-09-07/README.md)。 + ### P2：建立 floor-action 基线，并证明普通池化视觉的局限 已有结果应重新放入同一个事件协议： @@ -250,9 +265,10 @@ train 内 calibration/dev，不读取官方 test 结果做超参数搜索。 ## 8. 最小可投稿版本与强版本 -**最小可投稿版本**：新的 wearer entry/yield 事件协议、Omni speaker-grounding 诊断、 -EgoSpeak/encoder-only/Thinker/local-audio/pooled-video 强基线、EasyCom+Ego4D 复现，并清楚 -展示逐帧 mAP 隐藏的行为差异。 +**最小可投稿版本**：新的 wearer entry/yield 事件协议、global/local information audit、 +EgoSpeak/encoder-only/Thinker/local-audio/pooled-video 强基线、至少一个从真实 AV 恢复 +oracle local-floor gap 的模型，以及 EasyCom+Ego4D 复现；并清楚展示逐帧 mAP 隐藏的行为 +差异。 **强版本**：在上述基础上，匿名逐人 causal fusion 在 matched wearer recall 下显著降低 third-party false entry，保持 addressed response 与 yield，并通过 wrong-person、time-shift、 diff --git a/docs/progressive_feature_fusion_route_2026-09-02.md b/docs/progressive_feature_fusion_route_2026-09-02.md index 552211b..b1d6826 100644 --- a/docs/progressive_feature_fusion_route_2026-09-02.md +++ b/docs/progressive_feature_fusion_route_2026-09-02.md @@ -27,8 +27,11 @@ mAP。它表现稳定，但与旧 pooled 结果的 split/dev 协议并不匹配 视觉控制，因此不能写成注意力优于 pooling。另加入两个仅用 当前 sidecar 的 F5/F6 **screening** 模式：`audio-person-frame-set` 和 `gated-person-state-adapter`。前者没有 F4 人物历史，后者位于 Thinker readout 之后；二者 -都不能写成完整 F5/F6。第一版 F3 `track-gru-residual` 已实现但尚待远程运行；F2、F4、 -真正的 F5/F6 和 D1 仍然只是计划。 +都不能写成完整 F5/F6。第一版 F3 `track-gru-residual` 已完成远程运行，但因 zero-gate +阻塞早期梯度只能记为优化失败，不能用于判断 track continuity。随后完成的 oracle +information screen 表明及时的 aggregate visible activity 有明显价值，而逐人历史与 +shuffled-ID 无差别。因此当前优先级已从“继续加深 track 模型”改为“先从真实 AV 学出及时 +的局部 floor evidence”；F2、F4、真正的 F5/F6 和 D1 仍然只是计划。 ## 2. 不可改变的实验条件 @@ -70,7 +73,24 @@ P0 修正前的 entry/yield 数值只能称为 timing-derived proxy，不得作 | F6 | Thinker person-memory adapter | 旁路证据能否安全接入 Omni 内部 | D1 | | D1 | 在线 detector + causal tracker | 部署管线与 provided-box 上限的差距 | 完整系统 | -F1 是下一项实现。F2 及以后不并行堆叠，避免无法判断收益来自哪一层。 +F1 已完成初筛，但尚未通过视觉错位/错人和事件 gate。当前先执行下面的 oracle 驱动 +学习步骤；F2 及以后仍不并行堆叠，避免无法判断收益来自哪一层。 + +### 3.1 Oracle 信息门对路线的修正 + +2026-09-07 的 Session 12 seed-0、四 epoch screen 使用严格 past/current-only 的 +transcript-derived activity 作为 analysis-only sidecar。`aggregate-visible` 相对相同的 +G+A baseline 提升 `+6.96649` 个点的 0.2 秒 mAP 和 `+2.91047` 个点的十 horizon mean +mAP；visibility-only 和延迟 1 秒的 activity 仍接近 G+A。这说明当前模型缺少的是及时的 +局部 floor activity，而不只是“画面里有人”或额外参数。 + +但是 `aggregate-visible` 高于 `per-person`，且 `per-person` 与 `shuffled-id` 的差异仅为 +`-0.05826`（0.2 秒）和 `-0.07725`（mean mAP）个百分点。当前 dense aggregate-others +目标没有显示人物 ID 连续性的额外信息价值。因此，下一项学习模块首先应是从因果音频与 +人物 crop 估计 aggregate visible-person floor evidence，并报告它恢复了多少 oracle gap。 +只有正确人物/时间的逐人表示进一步稳定优于 aggregate、no-ID、broken-continuity 和错时 +控制，F3/F4 的 track 叙事才恢复为主线。完整结果见 +[`oracle result record`](../results/easycom_oracle_information_4ep_2026-09-07/README.md)。 ## 4. F0：冻结现有 pooling 基线 @@ -286,14 +306,13 @@ README 必须写明 split、seed、checkpoint、输入特征版本、评测协 ## 12. 最近执行顺序 -1. 第一版 matched pooled/`track-gru-residual` dense-AP screen 已完成，但因 zero-gate - gradient blockage 判为优化失败，不是 track continuity 的有效阴性结果。 -2. 先用零初始化 residual 输出层修正梯度路径，只在 Session 12 做 seed-0 诊断；确认分支 - 实际改变 logits 后再跑三 seed。若修正后仍无增益，再决定是否比较 8/16/32 - observations；不要根据已经查看过的 Sessions 1--3 seed-0 结果调参。 -3. 若修正后的 track GRU 有稳定增益，补 broken-track、past-shift、no-ID 和 remove-person - 控制；若仍无增益则停止该分支。 -4. correct track 通过控制后，再实现 F4 person-conditioned audio/Omni history；否则回到 - 对齐审计，不直接堆更深 interaction。 -5. 在形成论文事件结论前，仍须完成 P0 competing-risk censoring，并人工打印 response 与 +1. 用 seeds 1--2 复核六个 oracle 条件；只看 Session 12，不能根据 Sessions 1--3 调整结构。 +2. 从真实因果 AV 学习 aggregate visible-person activity/floor evidence，和 G+A、visibility、 + past-shift、wrong-person/wrong-time、参数匹配分支比较，并报告 oracle-gap recovery。 +3. 若正确视觉没有稳定优于错时/错人控制，停止复杂 fusion，先修复局部编码或 AV 对齐。 +4. 只有 learned person-organized evidence 优于 aggregate/no-ID 时，才用零初始化 residual + 输出层修复 track-GRU 梯度路径，并补 broken-track、past-shift 和 remove-person 控制。 +5. correct track 通过上述控制后，再实现 F4 person-conditioned audio/Omni history；否则 + 论文采用更简单的 global Omni + local aggregate floor-evidence 模型。 +6. 在形成论文事件结论前，仍须完成 P0 competing-risk censoring，并人工打印 response 与 yield 时间轴。当前事件结果只能称为 timing-derived proxy。 diff --git a/manuscript/main/main.tex b/manuscript/main/main.tex index 2be889e..1f91e44 100644 --- a/manuscript/main/main.tex +++ b/manuscript/main/main.tex @@ -10,8 +10,8 @@ \newcommand{\vect}[1]{\boldsymbol{#1}} \setlength{\emergencystretch}{2em} -\title{WHEN SHOULD AN EGOCENTRIC CONVERSATIONAL AGENT SPEAK? PROBING AND -ADAPTING OMNI MODELS FOR MULTIPARTY FLOOR ACTION} +\title{GLOBAL OMNI CONTEXT AND CAUSAL LOCAL AUDIOVISUAL EVIDENCE FOR +EGOCENTRIC MULTIPARTY FLOOR-ACTION FORECASTING} % TODO: Insert the final author list and affiliations after the ICASSP 2027 % review policy and author order are finalized. @@ -24,26 +24,28 @@ ADAPTING OMNI MODELS FOR MULTIPARTY FLOOR ACTION} \begin{abstract} An egocentric conversational agent operating among several people must decide -whether to speak or keep listening before it generates a response. EgoSpeak -operationalizes this problem using recordings from human camera wearers: -causal first-person audio--video forms the observation, and the wearer's future -speaking behavior forms the action target. The wearer is thus a data-collection -surrogate rather than the deployed system. EgoSpeak evaluates -separately extracted audio--visual features with Transformer, GRU, and Mamba -temporal models; whether end-to-end pretrained omni models that jointly process -raw audio and video expose the -fine-grained cues required for a decision every 200~ms remains unclear. We -probe representative omni backbones using frozen readout, -parameter-efficient tuning, and causal side branches that preserve recent -audio and visible-person evidence. The study tests whether a global omni state -underrepresents other-person activity and whether local evidence compensates -without making the agent passive. An EasyCom event protocol measures false -agent entries during multi-other exchanges at matched recall of wearer-derived -onset targets. Exploratory Qwen2.5-Omni-3B results motivate this hypothesis but -do not yet establish a cross-model limitation or determine the final adapter. -\todo{After the train-side study is locked, name the selected backbones and -adapter and insert: frozen/PEFT/local deltas, multi-other false-entry reduction -at 80\% onset recall, onset/yield non-degradation, and 95\% intervals.} +whether to speak or keep listening before generating a response. Following +EgoSpeak, we use causal first-person audio--video as observation and the future +speaking behavior of the human camera wearer as an imitation target for this +agent decision. We ask whether a jointly pretrained Omni model exposes the +fine-grained floor cues required every 200~ms. Our global--local formulation +uses a causal global Omni readout for broad scene and conversational context, +while a causal local branch preserves recent mixed audio and a variable set of +track-organized person observations. Anonymous tracks are an organizational +interface whose incremental value must be established, not an assumption that +the model knows speaker identity. On EasyCom development data, a single-seed +oracle analysis finds that perfectly aligned aggregate activity among visible +non-wearers improves 0.2-s mAP by 6.97 points and ten-horizon mean mAP by 2.91 +points over global Omni plus recent audio. Person-resolved histories do not +outperform an identity-shuffled control. This result motivates timely local +floor evidence, while explicitly withholding a person-continuity claim. The +full study compares frozen, parameter-efficient, and learned local variants +and evaluates false agent entry at matched wearer-onset recall. +\todo{After the train-side study is locked, replace the final two sentences +with the selected learned local variant and insert: its oracle-gap recovery, +multi-other false-entry reduction at 80\% onset recall, onset/yield +non-degradation, and 95\% intervals. Mention track continuity only if its +controls separate.} \end{abstract} \begin{keywords} @@ -79,26 +81,39 @@ wav2vec2 feature extractors with Transformer/LSTR, single-layer GRU, and Mamba temporal models, forecasting the camera wearer and aggregate other-person activity over the next two seconds \cite{kim2025egospeak}. Thus our novelty is not the task definition or the use of first-person data. The unresolved model -question is whether an end-to-end pretrained audio--visual omni backbone, +question is whether a jointly pretrained audio--visual Omni backbone, operating on causal raw media, exposes an adequate state for this dense floor-action controller. To our knowledge, such omni representations have not been systematically evaluated or adapted for the EgoSpeak task. -Our working hypothesis is specific. An omni model may retain broad scene and -speech semantics while its global causal state makes recent prosody, activity -boundaries, and individual visible-speaker motion difficult for a shallow -5-Hz head to recover. This would appear as inaccurate prediction of -other-person activity and poorly timed agent entries, especially in -multiparty exchanges. A causal local side branch may compensate by preserving -recent audio and anonymous person-level evidence before fusion with the global -state. This is a hypothesis to test, not an assumed omni-model deficiency: -frozen, parameter-efficiently tuned, and locally augmented variants must be -compared within each backbone. +Our working hypothesis is a representational decomposition. A global causal +Omni state may retain broad scene, speech, and semantic context while making +recent prosody, activity boundaries, and localized person motion difficult for +a shallow 5-Hz controller to recover. A causal local branch therefore +preserves recent mixed audio and a variable-size set of person observations +before fusing them back into the global state. The local set can be pooled +within a frame or organized into anonymous tracks across time. Whether that +continuity adds useful information beyond aggregate activity is itself a +falsifiable question. Frozen, parameter-efficiently tuned, and locally +augmented variants must consequently be compared within each backbone. + +An initial causal oracle screen sharpens this hypothesis. On held-out EasyCom +Session~12, perfectly timed aggregate activity from currently visible +non-wearers improves a matched global-plus-audio model by 6.97 points at 0.2~s +and 2.91 points in mean mAP. Visibility alone and activity delayed by one +second remain near the audio baseline. However, person-resolved histories are +weaker than aggregate histories and indistinguishable from histories whose +identity continuity is shuffled. Thus the current evidence supports the value +of timely local floor information, not a claim that person identity or track +continuity has already been exploited. Voice Activity Projection (VAP) provides the broader future-activity view of turn taking \cite{ekstedt2022vap}. EgoSpeak's role-relative output preserves compatibility with that view, but its aggregate ``others'' channel discards whether the same or different visible people produced recent activity. +Our oracle result shows that this discarded distinction is not automatically +useful under the dense target: identity continuity must beat aggregate and +shuffled controls before it enters the method claim. Moreover, conventional per-frame average precision can hide the behavioral failure of entering during an exchange between others. We therefore retain the dense EgoSpeak target while adding event evaluation of agent entry and yield; @@ -121,37 +136,40 @@ direct, continuous agent floor-action prediction and whether causal local evidence helps when the visible participant set changes. We call this task \emph{egocentric floor-action projection}. The study asks -four questions: (RQ1) how accurately a frozen global omni state predicts the -wearer-derived agent target and other-person activity; (RQ2) whether standard -parameter-efficient tuning recovers missing performance; (RQ3) whether causal -local audio and person-level visual evidence provide complementary -information; and (RQ4) whether any aggregate gain reduces false agent entries -without making the controller globally passive. EasyCom is suitable because +four questions: (RQ1) how accurately a frozen global Omni state predicts the +wearer-derived agent target and other-person activity; (RQ2) whether timely +causal local audio and visible-person evidence complement that state; (RQ3) +whether aggregate person-set evidence is sufficient or anonymous track +continuity adds measurable value; and (RQ4) whether any dense gain reduces +false agent entries without making the controller globally passive. +Parameter-efficient tuning is a matched adaptation control rather than an +assumed remedy. EasyCom is suitable because it provides synchronized egocentric audio--video, per-person tracks, and voice activity \cite{donley2021easycom}. Target-of-speech annotations, when used, are analysis strata only and are never model input or required output. -The intended contributions are therefore: a controlled cross-backbone study -of pretrained omni representations for EgoSpeak-style floor action; a -lightweight local-evidence adaptation selected from -pre-registered candidates; and a behavior-aligned event evaluation that maps -wearer demonstrations to agent entry and yield decisions. Current -Qwen2.5-Omni observations are motivation, not a claim that all omni models -intrinsically fail. Likewise, track-consistent cross-attention becomes the -final method only if it beats pooled, no-ID, and capacity-matched controls. +The intended contributions are therefore: a controlled study of global Omni +representations for EgoSpeak-style floor action; an oracle information audit +that separates timely aggregate floor evidence from person continuity; a +global--local adaptation selected from progressively structured candidates; +and a behavior-aligned event evaluation that maps wearer demonstrations to +agent entry and yield decisions. Current Qwen2.5-Omni observations are +motivation, not a claim that all Omni models intrinsically fail. Likewise, +track-consistent cross-attention becomes the final method only if it beats +aggregate, shuffled/no-ID, and capacity-matched controls. \begin{figure}[t] \centering \fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering -\todo{After model selection, draw the final system figure. For now it must show -the shared experiment interface: raw causal AV; online face/head detection and -causal tracking; backbone-specific global readout; recent audio and a masked -set of anonymous track states; the gated local adapter; and the common -$10\times3$ head. Mark causal cutoffs, trainable modules, parameter budgets, -NULL/OFFSCREEN handling, and source-boundary resets.}}} -\caption{Planned omni-model study for egocentric conversational floor action. -The local adapter remains a candidate family until the controlled train-side -comparison is complete.} +\todo{After model selection, draw the final system figure. Show the shared +interface: raw causal AV to a global Omni readout; a parallel recent-audio and +visible-person path; aggregate/no-ID/track-organized variants; residual fusion; +and the common $10\times3$ head. Draw the transcript-derived oracle only as a +dashed, analysis-only upper-bound path. Mark causal cutoffs, provided versus +online tracks, trainable modules, NULL/OFFSCREEN handling, and resets.}}} +\caption{Global context and causal local evidence for egocentric floor action. +Track organization is a controlled candidate, not a presupposed source of +speaker identity.} \label{fig:method} \end{figure} @@ -207,13 +225,14 @@ expose a comparable causal state, report it as unsupported rather than inventing a non-equivalent readout. Add future backbones only before the experiment protocol is locked.} -\subsection{Candidate local-evidence adaptation} +\subsection{Global context and causal local evidence} -The current hypothesis is that global omni states preserve broad semantic -context but may make recent prosody, activity boundaries, and individual -visual motion difficult for a shallow floor head to recover. From the most +The global state $\vect g^{(m)}_t$ is responsible for broad scene and +conversational context, but is not assumed to retain every localized cue needed +for floor control. From the most recent $L_a$ seconds, a causal local audio encoder produces $\vect a_t$. At -time $t$, an online detector and tracker produce a variable-size anonymous set +time $t$, provided boxes during method isolation---and an online detector and +tracker in the deployable condition---produce a variable-size anonymous set $\mathcal P_t=\{(\vect v_{t,i},\vect b_{t,i},\vect r_{t,i})\}_{i=1}^{N_t}$, where $\vect v_{t,i}$ is a crop feature, $\vect b_{t,i}$ is box geometry, and $\vect r_{t,i}$ contains only causal reliability variables such as visibility, @@ -231,8 +250,8 @@ current person set by masked cross-attention, W_q[\vect g^{(m)}_t;\vect a_t], [\vect h_{\varnothing};\vect h_{t,1};\ldots;\vect h_{t,N_t}] \right), \\ - \vect z^{(m)}_t &= \vect g^{(m)}_t+\tanh(\alpha)W_o\vect u_t, - \qquad \alpha\vert_{\mathrm{init}}=0, + \vect z^{(m)}_t &= \vect g^{(m)}_t+W_o\vect u_t, + \qquad W_o\vert_{\mathrm{init}}=0, \label{eq:adapter} \end{align} where $\vect h_{\varnothing}$ is a learned NULL/OFFSCREEN token. Padding is @@ -243,14 +262,23 @@ reported. The operator is permutation equivariant \cite{lee2019settransformer}; temporary tracker IDs provide continuity, not identity semantics. +Equation~\eqref{eq:track} is a candidate organization of local evidence, not a +claim that recurrent identity is necessary. Simpler alternatives replace each +$\vect h_{t,i}$ with a current-frame observation or pool the set into one +aggregate vector. This distinction is important because the current oracle +screen favors aggregate activity and does not separate correct from shuffled +identity histories. + This decomposition is informed by MuVAP: its global audio representation queries same-frame causal ASD track representations, followed by temporal modeling and gated global-to-speaker fusion \cite{qi2026muvap}. We retain the -useful separation between global floor dynamics and person-grounded evidence, +useful separation between global floor dynamics and person-organized evidence, but not its role-relative output: our fixed target is the invisible wearer's -entry/yield action, not the next visible speaker. The zero-initialized gate -also follows the conservative adapter principle used to inject visual evidence -into strong audio models \cite{rouditchenko2024whisperflamingo}. Local evidence +entry/yield action, not the next visible speaker. The zero-initialized residual +output follows the conservative adapter principle used to inject visual +evidence without perturbing the base prediction at initialization +\cite{rouditchenko2024whisperflamingo}. Unlike a zero scalar gate, it permits +the residual output layer to receive a gradient immediately. Local evidence is derived from the same causal raw media, not close microphones, transcripts, speaker labels, global identity anchors, or future frames. @@ -258,8 +286,10 @@ We evaluate candidates in increasing structural complexity: pooled recent audio/vision; a capacity-matched temporal branch; a per-frame person set with no cross-time identity; a shared track-wise temporal encoder; and person--audio cross-attention as in Eqs.~\eqref{eq:track}--\eqref{eq:adapter}. Only a -candidate that improves the pre-registered event metric over simpler and -capacity-matched controls becomes the final method. A limited latent +candidate that improves the pre-registered event metric over simpler, +aggregate, identity-shuffled, and capacity-matched controls becomes the final +method. If person continuity does not separate from those controls, the final +model remains a global-plus-local aggregate floor-evidence model. A limited latent bottleneck is tested only if large visible sets make full person attention a runtime bottleneck; it is not assumed to improve accuracy. @@ -339,6 +369,18 @@ clip-local anonymous participant tracks; its per-person voice coverage on the 433-clip EgoSpeak split must be audited before reporting the same multi-other event analysis. +\textbf{Oracle information audit.} +Before selecting a learned local module, we measure what information could in +principle complement the global state. Analysis-only sidecars expose +transcript-derived speech activity no later than the current cutoff under six +conditions: local audio, visibility only, one-second-delayed activity, timely +aggregate visible activity, per-person histories, and histories with their +cross-time identity assignments shuffled. These labels never enter a +deployable model. The audit separates three hypotheses that ordinary feature +ablation conflates: the value of seeing people, the value of knowing current +visible-other activity, and the incremental value of preserving who produced +that activity. + \textbf{Backbone comparison.} The initial study covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o~4.5. For each backbone $m$, we first compare its frozen causal state @@ -383,9 +425,11 @@ confirmatory three-seed run.} For each backbone we compare: frozen global state $G_m$; PEFT $P_m$; $G_m$ plus recent local audio; and $G_m$ plus pooled local audio--visual evidence. This compact matrix tests the representation hypothesis before investing in a -complex fusion module. If local vision survives three seeds and the event -gate, we then compare a parameter-matched temporal branch without person -continuity, a separately trained per-frame/no-ID set branch, and a +complex fusion module. The first learned target after the oracle audit is a +causal estimator of timely aggregate visible-person floor evidence. Only if +person-organized features survive three seeds, visual time/person corruptions, +and the event gate do we compare a parameter-matched temporal branch without +person continuity, a separately trained per-frame/no-ID set branch, and a track-consistent masked cross-attention branch. Provided-track and online-track inputs are reported separately. On a frozen track-aware checkpoint, a clip-consistent global renaming must leave outputs unchanged, whereas \emph{broken @@ -419,6 +463,36 @@ after the architecture is frozen.} \section{Results and Analysis} \label{sec:results} +\begin{table}[t] +\caption{Causal oracle-information screen on EasyCom development Session~12 +(seed 0, four epochs). Oracle speech activity is analysis-only. Values are +percentages under the causal-valid protocol.} +\label{tab:oracle} +\centering +\small +\begin{tabular}{lcc} +\hline +Input condition & 0.2-s mAP & Mean mAP\\ +\hline +Global + local audio & 72.58 & 58.77\\ +Visibility only & 71.95 & 58.86\\ +Activity delayed by 1 s & 72.02 & 58.90\\ +Aggregate visible activity & \textbf{79.55} & \textbf{61.68}\\ +Per-person activity histories & 78.80 & 61.11\\ +Identity-shuffled histories & 78.86 & 61.19\\ +\hline +\end{tabular} +\end{table} + +Table~\ref{tab:oracle} establishes an information opportunity, not a deployable +result. Timely aggregate activity among visible non-wearers provides a large +gain, while visibility alone and stale activity do not. However, preserving +person-resolved history gives no advantage over destroying its cross-time +identity assignment, and both trail the aggregate representation. The learned +method must therefore first recover useful local floor evidence; person-track +continuity is retained only if subsequent no-ID and broken-track controls +separate. + \begin{table}[t] \caption{Exploratory Qwen2.5-Omni-3B local-branch evidence on EasyCom test Sessions 1--3. Values are percentages from one seed, not a cross-backbone or @@ -441,9 +515,9 @@ $\Delta$ visual & +0.30 & +0.02 & +0.29\\ Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient. Pooled vision gives a small gain, while immediate wearer AP is nearly -unchanged. This is compatible with either weak but useful social grounding or -generic visual regularization; only matched-recall event evaluation and -track-continuity controls can distinguish them. +unchanged. This is compatible with either useful generic local context or +regularization; it is not evidence of person grounding. Only matched-recall +event evaluation and visual-alignment controls can distinguish these cases. A separate Qwen-3B dense-cache diagnostic gave 58.16 mean mAP for its frozen readout and 56.20 for one LoRA recipe. Because that diagnostic used a different feature protocol, it is not combined numerically with @@ -522,10 +596,11 @@ rejected PEFT recipes and fusion candidates to supplementary material unless they directly explain the selected method.} \textbf{Qualitative evidence.} -For a multi-other exchange, visualize per-person speech activity and track IDs, -wearer ground truth, the frozen-global and selected-local wearer-entry scores, -and the single frozen threshold on one physical time axis. If the selected -candidate is track-aware, also show its broken-continuity score. A useful +For a multi-other exchange, visualize annotation-derived per-person activity +as an analysis reference, provided track IDs when applicable, wearer ground +truth, the frozen-global and selected-local wearer-entry scores, and the single +frozen threshold on one physical time axis. If the selected candidate is +track-aware, also show its broken-continuity score. A useful example shows the frozen baseline rising at an other-speaker boundary while the selected adapter remains below threshold; a companion positive example must show that a real wearer onset is not suppressed. @@ -541,17 +616,21 @@ event result and cannot establish it alone.} \label{sec:conclusion} We study when an egocentric agent should enter or yield a multiparty floor, not -merely whether someone will speak. The central question is model-agnostic: -whether frozen or parameter-efficiently adapted omni representations expose -the fine-grained floor cues required by a 5-Hz controller, and whether causal -local evidence closes any gap. Dense activity forecasting supplies a common -target, while matched-recall event evaluation tests whether an apparent gain -actually prevents intrusions during multi-other exchanges. Current Qwen-3B -results are exploratory motivation only. \todo{After the controlled study, -state which backbone/adapter was selected, whether the representation pattern -generalized beyond Qwen, the quantitative false-entry and onset/yield results, -and the main deployment limitation. Do not infer success from aggregate mAP or -selected examples alone.} +merely whether someone will speak. The working decomposition is global Omni +context plus causal local audiovisual floor evidence. The current +Qwen2.5-Omni-3B oracle audit supports the second term: timely aggregate +visible-other activity is informative beyond the global state and recent +mixed audio. It does not support the stronger claim that participant identity +or track continuity adds value, because per-person histories do not beat an +identity-shuffled control. Dense activity forecasting supplies a comparable +target, while matched-recall event evaluation must determine whether a learned +local estimator actually prevents intrusions during multi-other exchanges +without suppressing valid wearer entry or yield. \todo{After the controlled +study, state which learned local variant was selected, how much of the oracle +gap it recovered, whether the pattern generalized beyond Qwen, the +quantitative false-entry and onset/yield results, and the main deployment +limitation. If no learned variant passes these tests, conclude with the +information audit rather than claiming a track-based solution.} \bibliographystyle{IEEEbib} \bibliography{references} diff --git a/results/README.md b/results/README.md index 2af7524..7a6ff9d 100644 --- a/results/README.md +++ b/results/README.md @@ -15,6 +15,7 @@ readers to the right comparison. | [`easycom_transition_auxiliary_4ep_2026-08-27/`](easycom_transition_auxiliary_4ep_2026-08-27/README.md) | Supervised transition head and expected-reward objectives | | [`easycom_encoder_only_1s_4ep_2026-08-28/`](easycom_encoder_only_1s_4ep_2026-08-28/README.md) | Encoder-only A/V/AV versus global-Thinker local-feature baselines | | [`easycom_track_gru_residual_4ep_2026-09-07/`](easycom_track_gru_residual_4ep_2026-09-07/README.md) | Matched pooled/anonymous-track-GRU screen and near-zero residual-gate diagnosis | +| [`easycom_oracle_information_4ep_2026-09-07/`](easycom_oracle_information_4ep_2026-09-07/README.md) | Causal aggregate/per-person oracle information value and identity-continuity control | ## Interpretation rules