diff --git a/PROJECT_CONTEXT.md b/PROJECT_CONTEXT.md index 459c8e1..5593fd4 100644 --- a/PROJECT_CONTEXT.md +++ b/PROJECT_CONTEXT.md @@ -20,6 +20,7 @@ recall, without hurting addressed responses or yield behavior. | Need | Authoritative file | |---|---| +| Current four-page manuscript structure and narrowed claim scope (2026-09-17) | [docs/manuscript_local_evidence_structure_2026-09-17.md](docs/manuscript_local_evidence_structure_2026-09-17.md) | | ICASSP submission story, pruning and acceptance evidence bar | [docs/icassp2027_acceptance_storyline_2026-09-10.md](docs/icassp2027_acceptance_storyline_2026-09-10.md) | | Current paper thesis and claim boundary | [docs/global_local_omni_paper_position_2026-09-07.md](docs/global_local_omni_paper_position_2026-09-07.md) | | Progressive feature-fusion implementation route | [docs/progressive_feature_fusion_route_2026-09-02.md](docs/progressive_feature_fusion_route_2026-09-02.md) | @@ -34,6 +35,24 @@ When files disagree, prefer the newest scoped document and verify behavior in code/tests. A planned experiment is not an implemented feature or a measured result. +The 2026-09-17 manuscript restructure uses causal local evidence adaptation +as its umbrella: model architecture (actual G + recent 1-s Qwen audio, +optional pooled vision) and streaming inference architecture (8-s maximum +presentation budget, completed blocks, temporary observed tail). Semantic +opportunity evaluation is deferred while annotation quality is audited; +dense forecasting and raw endpoint onset/offset AP remain the current paper +scope. This does not complete P0 or establish a visual/tail-performance gain. +Earlier person-grounded/semantic strategy documents remain conditional +extensions and historical records, not descriptions of the default draft. + +The scoped [analysis experiment log](docs/local_evidence_analysis_experiment_log_2026-09-17.md) +queues local-evidence controls, tail-refresh x local-audio comparisons, context +caps, persistence, activity-error cases, full-pipeline profiling, and paired +Qwen Omni 3B/7B--MiniCPM-o 4.5 backbone-transfer controls. Transfer fixes the +local audio sidecar and audits readout, refresh, and actual history; the +MiniCPM entry point is not GPU-verified or an established matched 8-s cap. +All are pending in that log; Omni LoRA is deferred and no semantic gate is completed. + ## Current state ### Verified assets diff --git a/docs/README.md b/docs/README.md index e9d38dc..fe73777 100644 --- a/docs/README.md +++ b/docs/README.md @@ -7,6 +7,8 @@ implemented or validated. | Document | Role | |---|---| +| [manuscript_local_evidence_structure_2026-09-17.md](manuscript_local_evidence_structure_2026-09-17.md) | Current narrowed four-page manuscript scope; supersedes older semantic/person-grounded descriptions for the default draft | +| [local_evidence_analysis_experiment_log_2026-09-17.md](local_evidence_analysis_experiment_log_2026-09-17.md) | Pending local evidence, tail/audio, context, cross-backbone transfer, persistence, error and efficiency controls; Omni LoRA deferred | | [icassp2027_acceptance_storyline_2026-09-10.md](icassp2027_acceptance_storyline_2026-09-10.md) | Submission-focused paper story, evidence bar, pruning decisions and go/no-go criteria | | [global_local_omni_paper_position_2026-09-07.md](global_local_omni_paper_position_2026-09-07.md) | Evidence-backed paper thesis after the causal oracle information screen | | [progressive_feature_fusion_route_2026-09-02.md](progressive_feature_fusion_route_2026-09-02.md) | Gated route from pooled local evidence to Frame-set Attention, temporal person memory and Thinker adapters | diff --git a/docs/local_evidence_analysis_experiment_log_2026-09-17.md b/docs/local_evidence_analysis_experiment_log_2026-09-17.md new file mode 100644 index 0000000..55aef7a --- /dev/null +++ b/docs/local_evidence_analysis_experiment_log_2026-09-17.md @@ -0,0 +1,199 @@ +# Local evidence analysis：待做实验日志 + +日期：2026-09-17。状态：**设计已记录，以下实验均未在本轮执行**。 +只修改日志与手稿占位，不新增训练、提取或 evaluator 实现；未产生新的实测结果。 +当前稿件边界见 [四页稿结构决策](manuscript_local_evidence_structure_2026-09-17.md)。 + +## 本轮决定与篇幅 + +正文按 Main Comparison、Local Evidence Analysis、Streaming and Context Analysis、 +Error and Qualitative Analysis 展开，不把六类实验并列成六个小节。 +近期音频是核心，视觉是可选增量；Omni LoRA 暂缓，不预留独立小节。 +MiniROADA stage-two continuation 不是 Omni LoRA，也不替代其适配对照。 +同日追加：Qwen2.5-Omni 3B/7B 与 MiniCPM-o 4.5 的 backbone transfer 对照， +用于检验 local evidence 增量是否依赖单一骨干，不新增独立贡献或正文大节。 + +四页技术内容内优先保留：一张方法图、一张主表、一张 compact controls 表， +一张包含错误/失败案例的定性图。Detailed recipe、全部窗口/相位曲线和错误切片 +存入实验记录；正文仅选择事先指定、能回答核心假设的摘要，不按测试涨点挑选。 +结果解释文字将替换 TODO，而不是在当前四页基础上无限追加。 + +## 固定研究契约 + +- EasyCom Sessions 4--12 训练，1--3 测试；5 Hz，10 个 0.2--2.0 s horizon； + 固定 background / wearer / others 顺序和原 dense labels。 +- 新 recipe、窗口、干预延迟、场景规则和 case-selection rule 仅用训练侧确定。 + 最终沿用固定四 epoch 协议；若需改变，先记录训练侧依据与新协议版本。 +- 正式矩阵 seeds 0/1/2；记录 split、parent/checkpoint hash、运行 commit、 + feature/model/processor revision、优化器、有效学习参数、预算和 common valid pairs。 +- Training-side development 必须重建未见该 fold 的 task parent；已经用全部训练 + sessions 训练过的 parent 不能把 Session 12 宣称为 continuation 的 held-out dev。 +- Sessions 1--3 已多次被查看，后续对照如实标为 exploratory；新增 seeds 不恢复 + 独立 confirmatory test，也不把帧或十个相关 horizon 当作独立重复。 +- Dense：逐 horizon 三类 AP / mAP、wearer AP、十-horizon mean。 + Raw：source wearer OFF/ON 条件下，分别以 p(wearer)/1-p(wearer) 预测未来 endpoint。 + 报告 AP、prevalence、分母和每个 session；GT current activity 只用于评测切片。 +- 这些是 endpoint anticipation，不是 one-event-one-vote onset/offset detection。 + P0 真实数据审计仍未完成；semantic opportunity-v1 AP 已 superseded，不能复用为新证据。 + +## 实验队列：全部 pending + +| ID | 待做对照 / 审计 | 回答的问题 | 手稿位置 | +|---|---|---|---| +| E0 | 核验原始 pred/valid/target、sidecar 和 checkpoint provenance；打印代表性时间轴 | 是否比较了相同物理时间与有效样本？ | Setup / 主表说明 | +| L1 | 同 recipe、seed 和 cutoff 的 A、G、G+A、G+A+V | 音频与全局 context 是否互补？V 是否提供额外收益？ | 主表 + Local Evidence | +| L2 | 宽 global-only head，尽可能匹配 G+A 的有效学习参数；统一下游 hidden width 并报告 allocated/active 参数 | 收益是否仅来自更大 head？ | Local Evidence TODO | +| L3 | 固定 checkpoint 的 past-delayed A / V，保留对应时间与 mask 审计 | 模型是否依赖及时局部证据？ | Local Evidence TODO | +| L4 | 因果 current-activity classifier，当前预测概率重复至十个 horizon | Dense 收益是否主要是更好的当前状态识别？ | Main Comparison TODO | +| T1 | 8-s cap 下 tail refresh × local A 的四组训练对照 | 两条最新证据路径互补，还是冗余？ | Streaming 表上半部 | +| T2 | 同 block decomposition 从头 replay、future-only media intervention、cache reset；按十档 phase 分层 | cache 正确性与 tail-length dependence 是否成立？ | Streaming TODO | +| C1 | 4/8/16-s cap 的 tail-aware G+A；按 cap 重提取并分别训练 | 历史长度是否影响主方法，8 s 是否是合理折中？ | Streaming 表下半部 | +| C2 | 仅当要声称 local 分支缓解窗口敏感性时，在各 cap 增加 G 对照 | 窗口能否替代近期证据？融合是否对 cap 更稳定？ | 条件追加，不新增正文小节 | +| B0 | 审计各骨干的 causal readout、维度、刷新与实际历史范围；先短片 GPU smoke | 跨骨干样本和接口是否可比？ | Setup / Backbone transfer 说明 | +| B1 | Qwen2.5-Omni 3B/7B 各跑 G 与 G+A，保持同一份 local A | 增量是否在同系列不同规模下保留？ | Local Evidence + compact 表末三行 | +| B2 | MiniCPM-o 4.5 各跑 G 与 G+A；匹配 Qwen refresh 对照后报告增量 | 增量能否迁移到另一 Omni 家族？ | Local Evidence + compact 表末三行 | +| H1 | 预定义 activity 场景切片、正负计数、score/AP；纯负类只报告 distribution 或 calibrated FPR | 哪些可复核的活动场景仍混淆？ | Error TODO | +| Q1 | 固定规则选择 raw onset、offset、hard-negative/shared failure 的同-horizon 时间轴 | 定量差异对应什么可观察错误？ | 定性图空位 | +| S1 | 同硬件完整流水线 P50/P95 latency、吞吐、峰值显存；记录阶段分项与 block-transition cost | 逻辑 5 Hz 是否可实测在线运行，代价是什么？ | Streaming TODO | + +L1 的 A 是近期 1 s，G 是有历史的 integrated AV state。即使 recipe 和 head family +一致，也不能把 A/G 的差异纯粹归给 Thinker、语言推理或长时记忆；L2/C1 是必要限制性 +对照。闲置占位权重不能充当有效学习容量匹配。 + +L3 只使用过去输入，不能拿未来帧、双向平滑或 nearest-future 对齐作为合法干预。 +mask-only/availability 对照在缺失率与标签明显相关时追加，不先假定模型确有此 shortcut。 +局部视觉采用逐人平均，person-slot permutation 应不改变结果；不能把排列不变当成 +成功的 wrong-person grounding control。 + +## T1 四组及解释规则 + +| Omni refresh | local A off | local A on，固定 1 s / 5 Hz | +|---|---|---| +| Completed-only / 2-s update-and-hold | T1-C-G | T1-C-GA | +| Observed-tail / 200-ms refresh | T1-T-G | T1-T-GA | + +四组共同固定：source-aligned 2-s blocks、8-s maximum budget、readout/head family、 +训练暴露、目标、seed、common startup treatment 与 valid source--horizon pairs。 +各组单独训练匹配的 head，不把单纯在 test masking 的 OOD 差异写成消融收益。 + +分别记录 tail 在 A off/on 下的 AP delta、A 在两种 refresh 下的 AP delta； +四组差值之差仅作为描述性交互量，AP 非线性且帧间相关，不直接推出统计独立或机制定论。 +completed-only 未得到首个 state 的行在全部条件中统一处理并报告，不未来回填。 +因 phase 不同，历史 span 随当前块位置变化；对齐同一 cutoff/phase 并保存实际范围。 + +`tail_embedding` 是辅助 preserved-EgoSpeak residual 的 block-phase 编码，不是 tail +内容。其 on/off 只能是该辅助架构内的独立可选实验，不能替代 T1。 +同分块 replay 的数值一致性与当前尾块带来的预测收益分开报告；continuous-prefix +编码属于不同 representation condition，不作为必然数值等价的参照。 + +## 窗口与运行代价 + +C1 中 local A/V 固定 1 s，Omni cap 为预先指定的 4/8/16 s。各 cap 的 global state +必须从对应因果输入提取，不能通过裁剪缓存向量伪造窗口。8-s cap 行与 T1-T-GA +复用同一个确切 run，不重复训练后择优填写两处。 +固定 checkpoint 改 cap 若执行，单独标成 distribution-shift sensitivity；不能混入 +分别训练的主比较。C2 在没有执行前，不声称融合已降低对长 context 的依赖。 + +S1 计时包括视频读取/裁切、模态编码、尾块重算、cache/history rebuild、local evidence +和 head，不以缓存特征上的 head FPS 代替系统 FPS。固定硬件/dtype，预热、GPU 同步， +采样非边界与 block transition cutoff；记录是否含 I/O、并发/batching 和延迟积压。 +没有完整测量前只能称 5-Hz logical output，不称已验证 real-time。 + +## Backbone transfer：同日追加，全部 pending + +模型标识核对自官方模型卡： +[Qwen2.5-Omni-3B](https://huggingface.co/Qwen/Qwen2.5-Omni-3B)、 +[Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B)、 +[MiniCPM-o-4_5](https://huggingface.co/openbmb/MiniCPM-o-4_5)。 +本轮没有提取新特征、训练这些配对或验证跨骨干性能。 + +| 骨干 | Global-only | Global + local audio | 主要解释 | +|---|---|---|---| +| Qwen2.5-Omni-3B | B1-3-G | B1-3-GA | 主体参考；协议完全一致时复用 L1/T1 的确切 run | +| Qwen2.5-Omni-7B | B1-7-G | B1-7-GA | 同家族规模对照，不等于仅改变参数量的因果实验 | +| MiniCPM-o 4.5 | B2-M-G | B2-M-GA | 跨家族迁移；先过 B0，不能与未匹配 Qwen 原分数直接排名 | + +主迁移矩阵固定现有同一份 causal 1-s / 5-Hz Qwen local audio sidecar、mask 和 +下游 head family，只更换 global backbone。不能同时把 A 换成该模型自己的 encoder， +否则无法区分 global backbone 与 local encoder 的贡献。若后续试 native local A， +另设第二层实验，不混入主迁移矩阵；V 在音频配对完成后再作可选扩展。 + +每个骨干都分别训练 G/GA，固定训练暴露与预先锁定的 recipe，记录参数与逐 seed delta。 +完整记录保留绝对 M/W/On/Off、每-session/count；正文三行只摘要 +`Delta(G+A - G)` 的四项 AP/mAP 百分点增量。不只呈现涨点骨干。 +跨模型 head 不能直接复用权重；不同 hidden dimensions 用明确适配的投影，报告有效参数， +不能静默 padding 或截断。Qwen 当前 last-four-layer readout 与 MiniCPM 当前 final +normalized state 不同；B0 需冻结共同可实现的 readout 控制或明确标成接口差异， +不能把默认导出方式不一致解释成纯骨干效果。 + +B0 固定并审计相同 split/目标、physical cutoffs、模态可用性、输入采样、刷新频率、 +startup/common valid pairs、history 范围、readout 和下游训练预算；保存实际范围、 +source times、processor/model/code revision 与完整 latency/memory。 +同样称“8 s”或“5 Hz”不代表相同可见历史或相同 state freshness。 +不同预训练、编码器和 tokenization 仍有差异，因此配对增量是适配可迁移性的证据， +不是纯架构排名，也不预设 7B 更好或大模型不需要 local A。 + +MiniCPM 的现有状态见 [提取与验证说明](minicpmo_streaming_features.md)： +`native-1hz-hold` 与 `cached-partial-5hz` 已有入口，但均缺真实 GPU smoke 证据。 +原生 one-second commits 保留其 streaming-window policy；当前入口没有与 Qwen +物理 8-s budget 等价的已验证 cap。不能仅把原生缓存写成“8-s matched”。 + +- 先做 native 1-Hz hold G/GA pilot；必须另做 Qwen **1-Hz** update-and-hold G/GA， + 对齐 one-second chunk boundaries、初始化与 source-age。T1 的 2-s hold 不能替代。 + 若 history 范围暂无法匹配，仅报告各自协议内增量为 scoped exploratory transfer， + 不放进 matched 跨骨干摘要，不声称同预算的绝对性能排名。 +- 主体采用 5-Hz observed-tail 时，MiniCPM 行仅在 boundary/同分块 clean replay、 + snapshot restoration、future-only media tests 与实际 history 审计通过后准入。 + `--allow-unverified-partial-tail` 产物不能进入论文训练；失败记 pending/unsupported， + 不填 0 AP，也不拿 1-Hz 行顶替 5-Hz matched 行。 +- Qwen 7B 也需独立 causal smoke、维度/显存检查和匹配提取；旧的不同窗口特征不能 + 因模型名相同就视为本矩阵已完成。每种 refresh 分层报告，不混合平均。 + +执行优先级：主 3B 归因矩阵先完成，再做 7B 配对；MiniCPM 先 B0/native pilot， +验证与可比性门槛通过后补正式配对。若跨骨干只验证了 local audio 的增量， +不能进一步声称同一 tail 算法或视觉收益已普遍迁移。 + +## Hard negative 与定性分析规则 + +先用 participant timing facts 冻结切片，例如同 other 的 pause/resume、other-to-other +切换、overlap；场景命名不直接赋予 semantic floor ownership。 +positive/negative 仍由该 family/horizon 的原 dense endpoint 定义决定，不用 LLM verdict +强行覆盖。wearer backchannel 可以是 raw onset positive；other backchannel 是否是 +raw offset negative，取决于 wearer 在未来 endpoint 是否仍在讲话。 + +H1 保存每种 scene × family × horizon 的 Npositive/Nnegative、duration、invalid counts、 +分数分布；两类都存在时报告 AP/prevalence。纯负类 AP 记为 undefined，不记成 0； +若报告 FPR，阈值只在训练侧 calibration 选择，并同时给 matched-recall 结果与实际召回。 +原 semantic-v1 ENTRY/INT AP 不再使用；v2 真实数据与人工审核后才能恢复 semantic panel。 + +Q1 用训练侧固定的 case-selection/horizon 规则选择病例并保留失败案例。将预测定位到 +其未来 target time，同时标出实际 causal source cutoff；同一 horizon 比较 EgoSpeak、G、 +G+A，V 仅在实际跑完时加入。不能将十个 horizon 的最大值包络或未来平滑曲线与当前 GT +直接拼接。转写/participant annotations 只作为分析可视化，不回流为模型输入。 + +## 下一步顺序与交付 + +1. E0 / training-side recipe freeze，然后 L1/L2；复用已提取特征，先补最小归因证据。 +2. T1/T2，再执行 C1 和 S1；每个新提取路径先做单 session / 短片因果 smoke。 +3. L4 与 L3；若要窗口稳定性 claim，补 C2，而不仅看 G+A 窗口曲线。 +4. H1/Q1 与前述结果配套；语义审核与 v2 audit 独立保留，不因数值好坏决定是否审核。 +5. B0/B1/B2：7B 配对与 MiniCPM native pilot 可在核心冻结后推进；准入后补迁移摘要。 +6. LoRA 留作后续可选替代适配对照，不是本轮执行要求。 + +实际运行后放入 `results/__/`，至少 README、summary.json 或 +metrics.csv，保留 provenance 与协议。正文替换 TODO/--，不要把 pending 改名为 measured。 +阴性视觉、尾块或窗口结果均如实保留；三种子与逐 session 结果不等于足够的显著性证据。 + +手稿占位：[main.tex](../manuscript/main/main.tex)，主表 `tab:main`、compact controls 表 +`tab:ablation`、错误/定性图 `fig:qualitative_floor_events`。本轮没有启动 GPU 训练。 + +## 本轮文稿检查 + +本机 Tectonic 编译后的预览共四页（含参考文献）；方法仍只有一条展示公式。 +相对链接、TeX/bib 引用和 `git diff --check` 通过。四页及新表/图空位完成渲染检查， +没有 overfull 警告或可见重叠/裁切。现有 XeTeX/旧模板字体替代和 underfull 警告仍在， +这是 working preview，不是官方工具链的最终字体/篇幅验证。填入数值与分析后重新检查。 +本轮没有修改 feature/label/evaluator/训练代码，没有将任何 pending 实验标为完成。 +同日追加 backbone-transfer 短段和三行增量空位后重新编译、渲染检查全部四页， +仍为四页含参考文献；链接、TeX/bib keys、单展示公式与 `git diff --check` 再次通过。 +未修改旧运行或宣称跨骨干提取/对照已完成。 diff --git a/docs/manuscript_local_evidence_structure_2026-09-17.md b/docs/manuscript_local_evidence_structure_2026-09-17.md new file mode 100644 index 0000000..d4f430d --- /dev/null +++ b/docs/manuscript_local_evidence_structure_2026-09-17.md @@ -0,0 +1,109 @@ +# Local evidence adaptation：四页稿结构决策 + +日期：2026-09-17。状态：用户要求的内容重排，不是新模型实现或最终实验结论。 +本文更新当前稿件组织；此前战略记录保留其当时判断。 + +## 同日后续决定：分析与待做实验 + +按用户要求暂不展开 Omni LoRA。结果部分采用 Main Comparison、Local Evidence +Analysis、Streaming and Context Analysis、Error and Qualitative Analysis。 +详细队列、tail × local audio 四组、4/8/16-s context cap、persistence baseline、 +capacity/recency 控制及运行代价记录在 +[待做实验日志](local_evidence_analysis_experiment_log_2026-09-17.md)；全部 pending， +不是新增实测结果，也没有启动训练或修改实现。正文保留数值表/图空位，先控制四页篇幅。 +Hard-negative/qualitative 合并；原始活动标签与语义负例不能混用，semantic audit 仍暂缓。 +同日追加 Qwen2.5-Omni 3B/7B 与 MiniCPM-o 4.5 的 backbone-transfer 配对。 +主骨干仍为 3B；各骨干 G→G+A 固定同一份 local A，并区分 refresh/history/readout +差异。细节与准入门槛放日志，正文仅 Local Evidence 短段和 compact 表三行增量空位， +不新增独立贡献或大节。MiniCPM 提取入口未 GPU 验证，native cache 不等于 8-s cap。 + +## 中心命题与三点贡献 + +以 **causal local evidence adaptation** 组织近期音频、可选局部视觉和当前尾块读取。 +两者不是相同表示：local A/V 显式暴露近期模态证据；tail handling 使全局 Thinker state +包含截止时刻已经观测到的当前块。不能把 tail 误写为另一条已提取的内容 embedding。 + +1. **模型适配框架**：不可见 wearer 的 global--local future-activity forecasting， + 用冻结 Omni context 与近期模态证据融合，而非宣称首次提出多人 turn-taking。 +2. **流式实现**：bounded context + committed history + temporary partial tail， + 每 200 ms 对齐同一物理 cutoff。普通 cache 本身不是新算法；预测收益与实测效率待验证。 +3. **受控分析**：分别报告 dense forecasting 与 endpoint onset/offset anticipation， + 通过匹配容量、输入和刷新规则的对照定位收益，不能预写已改善语义 floor control。 + +第三项目前是研究设计，不是已通过的机制结论。近期音频是核心，视觉是可选消融； +即使正确视觉没有增量，也可如实报告，不能让稿件依赖未验证的视觉收益。 + +## 四页技术内容预算 + +官方上限为四页技术内容（含图表），可选第五页仅参考文献， +见 [ICASSP 2027 author guidelines](https://2027.ieeeicassp.org/author-guidelines/) +和 [submission instructions](https://2027.ieeeicassp.org/paper-submission-instructions/)。 +篇幅符合要求不意味着工作必然被接收。 + +| 部分 | 建议预算 | 内容 | +|---|---:|---| +| Abstract + Introduction | 约 0.8 页 | 问题、相关工作、适配动机、三项贡献 | +| Method | 约 1.1 页 | 两个子节及一张模型/流式对齐图 | +| Experimental Setup | 约 0.6 页 | split、训练和两层 threshold-free 指标 | +| Results and Analysis | 约 1.3 页 | 主表、关键机制/尾块对照、限制 | +| Conclusion | 约 0.2 页 | 支持的窄结论 | + +方法仅保留一条融合/输出公式，不展开普通 attention、mask、loss 或 cache 公式。 +最终数值表和一条 horizon/phase 曲线优先于增加候选方法。 + +## 与当前代码对齐 + +- 主体：`easycom_head.py` 的 last-four-layer concat-MLP、256D local projection、 + validity-aware feature concatenation 与 10×3 forecast head。 +- 局部：`extract_turn_local_features.py` 的近期 1s Qwen audio valid-token mean； + 可选视觉为 2FPS crop spatial mean → equal-person mean → recent valid-frame mean。 +- 流式：`streaming_turn_model.py` 的 `OpenStreamReplayExtractor`， + 2s source blocks、最多 8s presentation budget、完成块 payload 复用、临时 tail/cache + 分支及单次完成块提交。稳态 span 为约 6.2--8s，不是每行精确 8s sliding crop。 +- preserved-EgoSpeak residual 和阶段二仅为辅助架构实验，不是默认正文方法。 +- `tail_embedding` 仅存在于 `egospeak_omni_residual.py`，编码十档 block phase； + 不能归入当前 G+A head，更不是 tail-content/未来 horizon embedding。 +- from-scratch replay 的正确性参照必须使用同一 block decomposition。 + continuously encoded prefix 是独立表示对照，不能直接声称数值等价。 + +## 评测与实验边界 + +Dense：逐 horizon 三类 AP 的 mAP，再平均十个 horizon。Raw onset/offset： +按 source wearer OFF/ON 条件化，分别用 p(wearer)/1-p(wearer) 判断 future endpoint。 +多个 source rows 可属于同一物理边界，不是 event detection 或 semantic ENTRY/YIELD。 +两类 AP 的 prevalence 不同，不直接比较其绝对值；比较方法时固定共同 valid pairs。 + +Opportunity-conditioned semantic evaluation 按用户决定暂缓。旧 annotation workload 和 +协议存于 [`annotation_protocol_notes.tex`](../manuscript/main/annotation_protocol_notes.tex)， +不编入四页正文；历史数值仍在原 results/reports 记录中，不改成 human-reviewed。 +P0 真实数据审核未完成，不能据此宣称减少抢话、理解 takeover 或 addressee。 + +新超参数、窗口和 recipe 只用训练侧 folds 确定。父 checkpoint 已见过的 Session 12 +不能算其 continuation 的独立 dev。已反复查看的 Sessions 1--3 属 exploratory evidence， +新三种子只检验稳定性，不会自动恢复独立 confirmatory test 的性质。 + +最小对照：A/G/G+A；容量匹配；可选 V on/off；过去延迟的 local sensitivity； +completed-only versus observed-tail refresh（local A 固定 5Hz）；block-phase 分层。 +跨骨干对照报告同骨干 G/GA 增量；1-Hz hold 需匹配 Qwen 1-Hz 控制，不能用 2-s hold +替代；5-Hz MiniCPM 行必须通过 tail/cache/因果及 history 可比性审计。 +在没有采样同一 cutoff 的 latency 和 memory 测量前，5Hz 是逻辑输出网格，不是实测 FPS。 + +## 稿件位置 + +- [`main.tex`](../manuscript/main/main.tex)：重排正文，结果与机制结论仍显式 TODO。 +- [`method_architecture.tex`](../manuscript/main/figures/method_architecture.tex)： + 模型与流式实现统一图，移除尚未测量的 25Hz MuVAP person-set 主结构。 +- [`README.md`](../manuscript/main/README.md)：当前编译和 scope 说明。 +- [`REVIEWER_CHECKLIST.md`](../manuscript/main/REVIEWER_CHECKLIST.md)：追加当前优先项， + 旧的 semantic/track 路线保留为条件分支，不是本轮硬性实验清单。 + +提交前还需最终 author kit、作者信息、全部 TODO、实验证据与 IEEE AI-assisted drafting +披露检查。本轮仅修改文稿和图，未修改模型、目标、特征契约或训练 CLI。 + +## 本轮文档验证 + +使用本机已提供的 Tectonic 和现有未改动的 `spconf` / `IEEEbib` 模板编译， +当前 working PDF 共四页（含参考文献）；最终填入结果后须重新检查技术内容不超过四页。 +主方法仅一条展示公式。相对链接、TeX 引用和 bib keys 检查通过， +`git diff --check` 通过；方法图与所有 PDF 页完成渲染检查，并修复图中文字重叠和表格溢出。 +结果 TODO 是有意保留的研究门槛，当前 PDF 不是可直接提交的最终稿。 diff --git a/docs/omni_speaker_floor_experiment_plan_2026-08-30.md b/docs/omni_speaker_floor_experiment_plan_2026-08-30.md index c459bb0..1d96d88 100644 --- a/docs/omni_speaker_floor_experiment_plan_2026-08-30.md +++ b/docs/omni_speaker_floor_experiment_plan_2026-08-30.md @@ -1,5 +1,17 @@ # Omni 多人说话者理解与 wearer floor action：实验主线（2026-08-30） +## 2026-09-17 scoped analysis update + +用户当前先写窄化的 activity-anticipation 稿件；Omni LoRA 暂缓，不展开独立分析小节。 +局部 AV、tail refresh × local audio、context cap、activity hard-negative/qualitative +和运行代价，以及 Qwen Omni 3B/7B--MiniCPM-o 4.5 的 G/GA 配对迁移对照见 +[dated experiment log](local_evidence_analysis_experiment_log_2026-09-17.md)。 +下文保留历史 semantic/person-grounded gates；该更新没有完成 P0，没有实测新增结果， +也不授权以 raw endpoint AP 声称 semantic ENTRY/YIELD 或 addressee 理解。 +跨骨干固定同一 local A；未匹配 refresh/history/readout 或未验证的 MiniCPM tail +不能进 matched 摘要。此 scoped transfer 更新覆盖下文旧的仅 cache-diagnostic 限制， +但不把 MiniCPM 变成默认主方法，也不声称已验证通用增益。 + ## 1. 论文真正要回答的问题 本项目不再把“Qwen2.5-Omni 是否能替换 EgoSpeak 的 ResNet、wav2vec 和 diff --git a/manuscript/main/README.md b/manuscript/main/README.md index 72ff270..b96d675 100644 --- a/manuscript/main/README.md +++ b/manuscript/main/README.md @@ -1,5 +1,51 @@ # Main ICASSP manuscript +## Current structure (2026-09-17) + +The current draft uses **causal local evidence adaptation** as the single +umbrella for recent modality evidence and cutoff-aligned Omni tail access. +It supersedes the older scope description below; historical strategy docs +remain records of their dates. The new scoped decision is +[`../../docs/manuscript_local_evidence_structure_2026-09-17.md`](../../docs/manuscript_local_evidence_structure_2026-09-17.md). + +- Method has two subsections: **Model Architecture** (task, global/local + features, fusion and learning) and **Streaming Inference Architecture** + (8-s maximum context, completed-history reuse, temporary current tail, + cutoff alignment). Only one displayed fusion equation remains. +- The core is frozen Qwen Omni `G` + recent 1-s Qwen audio `A`; pooled + person vision is optional, not an established performance contribution. +- The three contributions cover global--local adaptation, its partial-tail + streaming realization, and controlled dense versus onset/offset analysis. + Caching alone is not claimed as a new algorithm; mechanism gains await + matched controls. +- The method figure now depicts that actual core rather than an unmeasured + 25-Hz MuVAP person-attention branch. Preserved-EgoSpeak residual fusion and + its block-phase embedding are auxiliary controls, not the default head. +- Semantic opportunity evaluation is deferred during annotation auditing. + Its historical protocol and workload are preserved in + [`annotation_protocol_notes.tex`](annotation_protocol_notes.tex), which is + deliberately not included in the submission PDF. Historical results remain + in the original `results/` and `reports/` records. +- Analysis uses Main Comparison, Local Evidence Analysis, Streaming and + Context Analysis, and Error and Qualitative Analysis. Omni LoRA is deferred. + Planned experiments are in the + [dated experiment log](../../docs/local_evidence_analysis_experiment_log_2026-09-17.md). +- Locked main rows and the tail-refresh x local-audio / context-cap table are + unmeasured placeholders, not zero or mixed-protocol confirmatory numbers. + Historical main-table screens are identified explicitly. Already inspected + official-test runs remain exploratory despite additional seed repetitions. +- Backbone transfer adds paired G/GA controls for Qwen2.5-Omni 3B/7B and + MiniCPM-o 4.5, fixing the same local audio sidecar. Compact table deltas are + pending and require readout/refresh/history audits; MiniCPM is not yet + GPU-verified or an established matched 8-s stream. This scoped transfer + overrides the older cache-diagnostic-only restriction below, not the 3B core. +- The body targets four technical pages with one method figure, one main + table, one compact streaming/transfer controls table, and one error/qualitative figure slot. + The optional fifth page is references only; replace TODOs rather than keep + appending, and finalize author kit and applicable AI-drafting disclosure. + +## Build and historical scope + This is the editable paper directory. It uses the unmodified official ICASSP 2026 LaTeX assets stored in `../ICASSP2026_Paper_Templates/`. When the ICASSP 2027 author kit is released, replace or add the official template and update diff --git a/manuscript/main/REVIEWER_CHECKLIST.md b/manuscript/main/REVIEWER_CHECKLIST.md index 4bd8864..a407c1a 100644 --- a/manuscript/main/REVIEWER_CHECKLIST.md +++ b/manuscript/main/REVIEWER_CHECKLIST.md @@ -2,6 +2,47 @@ This is an internal execution checklist. It is not part of the submitted PDF. +## Current priority override (2026-09-17) + +Use the [local-evidence structure decision](../../docs/manuscript_local_evidence_structure_2026-09-17.md) +for the current draft. The older semantic/person-grounded checklist below is +a conditional extension, not authorization to describe it as implemented or +a prerequisite for this narrower activity-anticipation study. + +- [x] Separate model architecture from streaming inference architecture. +- [x] Describe actual G+A fusion, optional pooled vision, and observed-tail + recomputation; retain only one displayed method equation. +- [x] Move the provisional LLM annotation protocol/workload outside the paper + body without relabeling it as human ground truth. +- [x] Reserve local-evidence, streaming/context, and combined error/qualitative + analysis slots; record pending experiments without LoRA expansion in the + [dated log](../../docs/local_evidence_analysis_experiment_log_2026-09-17.md). +- [ ] Complete a training-side-frozen A/G/G+A matrix with three seeds and common + causal-valid pairs; audit capacity and distinguish their history ranges. +- [ ] Evaluate optional vision on/off before retaining a visual-gain claim. +- [ ] Complete tail refresh x local-audio four-group comparison with enabled + local audio fixed at 5 Hz; report phase strata and same-decomposition replay. +- [ ] Re-extract/train the predeclared 4/8/16-s context-cap controls; add G at + each cap before claiming local fusion reduces context sensitivity. +- [ ] Add a predicted-current-activity persistence reference and capacity/recency + controls; do not supply GT activity as model input. +- [ ] Report scene-level activity-error counts and fixed-horizon timelines, + including a shared failure; semantic negatives cannot override raw labels. +- [ ] Report dense AP/mAP and conditional endpoint onset/offset AP separately, + including denominators, per-session results, and limits of the three-session + official test. Do not claim one-event-one-vote detection. +- [ ] Measure full extraction latency/memory before claiming real-time FPS. +- [ ] Run paired G/GA backbone-transfer controls for Qwen Omni 3B/7B and + MiniCPM-o 4.5 with the same local A; report within-backbone deltas and audit + readout, dimensions, refresh, and actual history. MiniCPM 5-Hz rows require + GPU tail/cache/causality verification; native 1-Hz pilots need Qwen 1-Hz + controls and cannot substitute for matched 5-Hz rows. The dated log overrides + the older cache-diagnostic-only scope below, not its safety checks. +- [ ] Acknowledge that already viewed Sessions 1--3 results are exploratory; + additional seeds do not create a fresh independent test. +- [ ] Resolve all TODOs, retain only supported mechanisms, and check the + four-technical-page limit, author kit, and IEEE AI-drafting disclosure. + ## Reviewer verdict and frozen paper shape The potentially publishable question is: **can a general-purpose Omni readout diff --git a/manuscript/main/annotation_protocol_notes.tex b/manuscript/main/annotation_protocol_notes.tex new file mode 100644 index 0000000..a751479 --- /dev/null +++ b/manuscript/main/annotation_protocol_notes.tex @@ -0,0 +1,196 @@ +% ARCHIVED INTERNAL NOTES: not included in the four-page submission draft. +% Source: main.tex before the 2026-09-17 local-evidence restructure. +% Provisional annotation workload and protocol notes, not model results. +% Table references refer to the previous manuscript layout. + +% Upstream 2026-09-17 merge: semantic evaluation remains deferred in main.tex. +% Protocol-v1 opportunity AP is superseded: ENTRY cannot cross another other +% onset/intervening wearer activity; multi-competitor INT must not use an +% arbitrary onset. Re-run protocol v2 and audit labels before semantic claims. +% Retained upstream semantic qualitative plan (not compiled in main.tex): +\subsection{Qualitative Floor-Event Analysis} + +We will compare EgoSpeak with the final matched model on human-confirmed +examples selected by pre-declared event subtype rather than post-hoc score +extremes. Each case will align participant-resolved activity, minimal transcript +context, the opportunity anchor and competing-risk censor, and the two models' +causally emitted wearer forecasts. The comparison will include a genuine wearer +entry, an other-to-other false-entry opportunity, and an INT-yield paired with a +non-takeover overlap; a shared failure case will be retained when space permits. + +\begin{figure*}[t] +\centering +\fbox{\parbox[c][0.115\textheight][c]{0.965\textwidth}{ +\centering +\textbf{Qualitative comparison placeholder}\\[0.7em] +\begin{tabular}{ccc} +Case A: wearer entry & Case B: other$\rightarrow$other HN +& Case C: INT-yield / failed takeover\\ +participant timeline + transcript & opportunity anchor + censor +& EgoSpeak vs. proposed forecast\\ +\end{tabular} +}} +\caption{Planned qualitative floor-event comparison. Final panels will use +human-confirmed protocol-v2 cases and show the same causal source time for both +models; the figure is a layout placeholder and contains no result claim.} +\label{fig:qualitative_floor_events} +\end{figure*} + +\begin{table}[h] +\caption{LLM pre-label workload before human adjudication. Agreement is +between two independent passes.} +\label{tab:behavior_annotation_stats} +\centering +\scriptsize +\begin{tabular}{@{}lrr@{}} +\hline +Measure & EasyCom S1--4 & Ego4D AV val \\ +\hline +Candidates & 2,128 & 3,608 \\ +AI positive & 367 & 541 \\ +AI hard negative & 1,017 & 1,174 \\ +AI excluded & 744 & 1,893 \\ +Two-pass agreement & 93.4\% & 82.1\% \\ +Human-confirmed & 0 & 0 \\ +\hline +\end{tabular} +\end{table} + +\appendix +\section{LLM-Assisted Behavioral Event Annotation} +\label{app:event_annotation} + +EasyCom's raw speech boundaries are acoustic annotations rather than floor +labels. We preserve them unchanged for dense forecasting and create a separate +behavioral-event layer. Its design principle is to anchor an observed wearer +behavior first and then retrieve the conversational opportunity that could make +that behavior meaningful. False-positive opportunities are mined independently +because, by definition, many contain no wearer transition. + +\subsection{Positive-event mining} + +\textbf{Wearer activity transitions.} +We preserve every raw wearer $0\!\rightarrow\!1$ and $1\!\rightarrow\!0$ +boundary as a semantic review candidate. Candidate construction does not merge +short silence gaps, infer within-turn resumption, or decide whether overlapping +speech constitutes takeover. It only records participant-resolved temporal and +transcript facts. The source intervals and dense targets remain unchanged. + +\textbf{Wearer entry.} +Every raw wearer onset at $t_b$ is proposed as one wearer-entry review item. +We search backward within the retrieval window for non-wearer offsets and store +each one with its response delay and factual indicators of intervening wearer +speech, intervening other onsets, and concurrent activity. These are possible +context anchors rather than deterministically admissible causes. The LLM and +human reviewer decide whether the boundary is a genuine new entry, a +within-turn resumption, a wearer backchannel, or an excluded non-event. A +preceding release is not required for a positive entry: the wearer may enter +after a release, from silence, or while another speaker remains active. When a +release clearly prompts the entry, the LLM or human may select one supplied +option; neither may invent or move a timestamp. + +\textbf{INT-yield.} +Every raw wearer offset $t_y$ is proposed as an INT-yield review item. The +generator records the current wearer activity-run start, all nearby non-wearer +onsets and offsets, activity immediately before and after $t_y$, whether an +other speaker is pre-existing or begins during wearer speech, strict and +50-ms-tolerant boundary crossing, overlap duration, the next speech onset, +transcripts, and target-of-speech metadata. Relations such as +$t_w=Latex, - flow/.style={-{Latex[length=2.0mm]}, line width=0.65pt, draw=navy}, - aux/.style={-{Latex[length=1.8mm]}, line width=0.6pt, draw=teal}, - box/.style={draw=navy, rounded corners=1.5pt, line width=0.65pt, - fill=bluefill, minimum height=7mm, align=center, inner sep=2.3pt}, - localbox/.style={box, draw=teal, fill=tealfill}, - residualbox/.style={box, draw=orange, fill=orangefill}, - labelbox/.style={align=center, text=navy}, - note/.style={font=\sffamily\tiny, text=grayline, align=center} + font=\sffamily\scriptsize, >=Latex, + flow/.style={-{Latex[length=1.8mm]},line width=0.65pt,draw=navy}, + localflow/.style={flow,draw=teal}, + box/.style={draw=navy,rounded corners=1.5pt,line width=0.65pt, + fill=bluefill,minimum height=8mm,align=center,inner sep=2.2pt}, + localbox/.style={box,draw=teal,fill=tealfill}, + note/.style={font=\sffamily\tiny,text=grayline,align=center} ] - -\node[labelbox] (av) at (0,2.65) {causal full-scene AV\\prefix $\mathcal X_{\leq t_n}$}; -\node[box, right=7mm of av, minimum width=24mm] (qwen) {frozen\\Qwen2.5-Omni}; -\node[box, right=8mm of qwen, minimum width=12mm] (g) {global\\$G_{t_n}$}; - -\node[labelbox] (audio) at (0,1.45) {mixed audio\\$\leq t_n$}; -\node[localbox, right=7mm of audio, minimum width=24mm] (cpc) {frozen causal CPC\\25-Hz processing}; -\node[localbox, right=8mm of cpc, minimum width=12mm] (u) {shared\\$U_{t_n}$}; - -\node[labelbox] (faces) at (0,0.15) {visible face crops\\$\leq t_n$}; -\node[localbox, right=7mm of faces, minimum width=24mm] (asd) {frozen causal AV/ASD\\25-Hz processing}; -\node[localbox, right=8mm of asd, minimum width=20mm] (persons) {anonymous set\\$\mathcal P_{t_n}=\{[V_i;s_i]\}$}; - -\node[box, minimum width=21mm] (base) at (8.25,2.05) {$G+U$ base head\\$Z^{\rm base}_{t_n}$}; -\node[localbox, minimum width=22mm] (attn) at (8.25,0.70) {masked person MHA\\query: $G_{t_n}+U_{t_n}$\\\scriptsize memory: $\mathcal P_{t_n}+$ NULL}; -\node[residualbox, minimum width=19mm] (res) at (10.65,0.70) {zero-init residual\\$\Delta Z_{t_n}$}; -\node[circle, draw=navy, fill=white, line width=0.7pt, - minimum size=5.5mm, inner sep=0pt] (sum) at (11.55,2.05) {$+$}; -\node[box, minimum width=21mm] (out) at (13.15,2.05) {$10\!\times\!3$ future activity\\$0.2$--$2.0$ s}; - -\draw[flow] (av) -- (qwen); -\draw[flow] (qwen) -- (g); -\draw[aux] (audio) -- (cpc); -\draw[aux] (cpc) -- (u); -\draw[aux] (faces) -- (asd); -\draw[aux] (asd) -- (persons); -\draw[flow] (g.east) -| ($(base.west)+(-0.25,0.12)$) -- (base.west); -\draw[aux] (u.east) -| ($(base.west)+(-0.25,-0.12)$) -- (base.west); -\draw[flow] (g.east) -- ++(0.3,0) |- (attn.north west); -\draw[aux] (u.east) -- ++(0.5,0) |- (attn.west); -\draw[aux] (persons.east) -- (attn.west); -\draw[flow] (base) -- (sum); -\draw[aux] (attn) -- (res); -\draw[flow, draw=orange] (res.north) |- (sum.south); -\draw[flow] (sum) -- (out); - -\node[note, below=1.5mm of base] {trained first, then frozen}; -\node[note, below=1.2mm of res] {trainable adaptation}; - -\draw[grayline, line width=0.45pt] (-0.15,-0.62) -- (14.35,-0.62); -\node[anchor=west, font=\sffamily\scriptsize\bfseries, text=navy] - at (-0.05,-0.92) {Causal streaming at cutoff $t_n$}; - -\node[anchor=east, text=navy] at (2.25,-1.45) {Omni input}; -\draw[fill=bluefill, draw=navy, line width=0.55pt] (2.45,-1.67) rectangle (4.35,-1.23); -\draw[fill=bluefill, draw=navy, line width=0.55pt] (4.35,-1.67) rectangle (6.25,-1.23); -\node[text=navy] at (3.40,-1.45) {cached 2-s blocks}; -\node[text=navy] at (5.30,-1.45) {$\cdots$}; -\draw[fill=orangefill, draw=orange, line width=0.65pt] (6.25,-1.67) rectangle (9.15,-1.23); -\node[text=orange] at (7.70,-1.45) {current partial block}; -\draw[dashed, draw=orange, line width=0.8pt] (9.15,-2.90) -- (9.15,-1.02); -\node[anchor=south, text=orange, font=\sffamily\tiny\bfseries] at (9.15,-1.02) {$t_n$}; -\node[note, anchor=west] at (9.40,-1.45) {latest observed input included\\up to 8-s rolling budget}; - -\node[anchor=east, text=teal] at (2.25,-2.15) {local encoder}; -\foreach \x in {2.55,2.85,...,8.85} { - \draw[draw=teal, line width=0.55pt] (\x,-2.33) -- (\x,-1.97); +\node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy] + at (-0.15,3.15) {Model: global context + recent evidence}; +\node[align=center,text=navy] (av) at (0.65,2.45) + {full-scene AV\\observed prefix}; +\node[box] (omni) at (3.05,2.45) {frozen Qwen2.5-Omni\\8-s maximum context}; +\node[box] (global) at (5.65,2.45) {last-four-layer\\projection $G_n$}; +\node[align=center,text=teal] (audio) at (0.65,1.28) + {recent 1-s\\mixed audio}; +\node[localbox] (ae) at (3.05,1.28) {frozen Qwen audio\\valid-token mean}; +\node[localbox] (ap) at (5.65,1.28) {local projection\\$\widetilde A_n$ + validity}; +\node[align=center,text=teal] (video) at (0.65,0.08) + {recent 1-s\\person crops}; +\node[localbox,dashed] (ve) at (3.05,0.08) + {frozen Qwen visual\\recent crop pooling}; +\node[localbox,dashed] (vp) at (5.65,0.08) + {optional projection\\$\widetilde V_n$ + validity}; +\node[box,minimum width=20mm] (fusion) at (8.35,1.28) + {feature concatenation\\trainable MLP}; +\node[box,minimum width=23mm] (out) at (11.15,1.28) + {$10\times3$ future activity\\0.2--2.0 s}; +\draw[flow] (av) -- (omni); +\draw[flow] (omni) -- (global); +\draw[localflow] (audio) -- (ae); +\draw[localflow] (ae) -- (ap); +\draw[localflow,dashed] (video) -- (ve); +\draw[localflow,dashed] (ve) -- (vp); +\draw[flow] (global.east) -- ++(0.35,0) |- (fusion.north); +\draw[localflow] (ap) -- (fusion); +\draw[localflow,dashed] (vp.east) -- ++(0.35,0) |- (fusion.south); +\draw[flow] (fusion) -- (out); +\node[note] at (10.95,0.22) + {background / wearer / aggregate others\\no semantic event or identity output}; +\draw[grayline,line width=0.45pt] (-0.15,-0.65) -- (12.5,-0.65); +\node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy] + at (-0.15,-0.98) {Stream: committed history + temporary current tail}; +\node[anchor=east,text=navy] at (2.05,-1.65) {Omni media}; +\foreach \a/\b/\lab in {2.3/3.9/2 s,3.9/5.5/2 s,5.5/7.1/2 s} { + \draw[fill=bluefill,draw=navy,line width=0.55pt] + (\a,-1.87) rectangle (\b,-1.43); + \node[text=navy] at ({(\a+\b)/2},-1.65) {\lab}; } -\draw[draw=teal, line width=0.55pt] (2.55,-2.15) -- (9.15,-2.15); -\node[note, anchor=west, text=teal] at (9.40,-2.15) {25 Hz internal states}; - -\node[anchor=east, text=navy] at (2.25,-2.72) {forecast grid}; -\draw[draw=navy, line width=0.65pt] (2.55,-2.72) -- (9.15,-2.72); -\foreach \x/\lab in {3.15/0.2,4.65/0.4,6.15/0.6,7.65/0.8,9.15/$t_n$} { - \draw[draw=navy, line width=0.75pt] (\x,-2.92) -- (\x,-2.52); - \node[below=0.5mm, font=\sffamily\tiny, text=navy] at (\x,-2.92) {\lab}; - \draw[-{Latex[length=1.4mm]}, draw=teal, line width=0.55pt] (\x,-2.32) -- (\x,-2.50); +\draw[fill=orangefill,draw=orange,line width=0.65pt] + (7.1,-1.87) rectangle (8.5,-1.43); +\node[text=orange,font=\sffamily\tiny] at (7.8,-1.65) {partial tail}; +\draw[fill=white,draw=grayline,dashed,line width=0.55pt] + (8.5,-1.87) rectangle (8.7,-1.43); +\node[note,anchor=west] at (9.0,-1.65) + {three completed blocks + observed tail\\8-s maximum; future portion excluded}; +\node[note,text=navy] at (4.7,-2.13) {reuse completed media payloads}; +\node[note,text=orange] at (7.8,-2.13) {recompute at each cutoff}; +\node[anchor=east,text=teal] at (2.05,-2.62) {local evidence}; +\draw[draw=teal,line width=1.8pt] (7.7,-2.62) -- (8.5,-2.62); +\node[note,anchor=west,text=teal] at (9.0,-2.62) + {recent 1-s audio / optional vision\\refreshed on the common 5-Hz grid}; +\node[anchor=east,text=navy] at (2.05,-3.18) {decision grid}; +\draw[draw=navy,line width=0.6pt] (2.3,-3.18) -- (8.5,-3.18); +\foreach \x in {2.42,2.58,...,8.34} { + \draw[draw=navy,line width=0.5pt] (\x,-3.28) -- (\x,-3.08); } -\node[note, anchor=west, text=navy] at (9.40,-2.72) {5 Hz fusion and prediction\\every fifth local right edge}; - +\draw[draw=navy,line width=0.9pt] (8.5,-3.3) -- (8.5,-3.06); +\draw[dashed,draw=orange,line width=0.75pt] + (8.5,-3.35) -- (8.5,-1.15); +\node[anchor=south,text=orange,font=\sffamily\tiny\bfseries] + at (8.5,-1.15) {cutoff $t_n$}; +\node[note,anchor=west,text=navy] at (9.0,-3.18) + {one prediction every 200 ms\\same cutoff and target index}; \end{tikzpicture} \end{document} diff --git a/manuscript/main/main.tex b/manuscript/main/main.tex index a2721ba..f4c997c 100644 --- a/manuscript/main/main.tex +++ b/manuscript/main/main.tex @@ -1,121 +1,102 @@ -% ICASSP 2027 content skeleton. The final paper must use the official 2027 -% author kit and contain no visible \todo{} markers. Planned experiments are -% intentionally marked rather than presented as completed work. +% ICASSP 2027 working draft: results and mechanism claims are not yet frozen. +% Four technical pages maximum; the optional fifth page contains references only. +% Use the final author kit and resolve every TODO before submission. \documentclass{article} \usepackage{spconf} \usepackage{amsmath,amssymb,graphicx,hyperref,xcolor} - +\hypersetup{hidelinks} \newcommand{\todo}[1]{\textcolor{red}{\textbf{[TODO: #1]}}} \newcommand{\vect}[1]{\boldsymbol{#1}} \setlength{\emergencystretch}{2em} - -\title{Is It My Turn? Causal Local Evidence Adaptation of Omni Models for -Egocentric Floor Forecasting} - -% TODO: Insert the final author list and affiliations after the ICASSP 2027 -% review policy and author order are finalized. +\title{Causal Local Evidence Adaptation of Omni Models for +Egocentric Speech Anticipation} \name{Internal Manuscript Draft} \address{} - \begin{document} \ninept \maketitle \begin{abstract} -Real-time audiovisual agents must decide not only what to say, but whether an -emerging floor opportunity belongs to them. This is difficult in multiparty -egocentric interaction: the target wearer is behind the camera, while the -visible people provide only indirect evidence about whether the surrounding -floor remains occupied. We study strictly causal forecasting of the wearer's -future speech activity every 200~ms. Although a general-purpose Omni model -provides rich long-context audio--visual representations, its global readout is -not explicitly trained to expose sub-second, wearer-relative floor evidence. -We therefore study a candidate causal local floor adapter that combines the global -Omni state with short-window mixed audio and an anonymous set of visible-person -observations, without assuming persistent speaker identity. On EasyCom, -\todo{report the final three-seed dense and fixed-horizon behavioral-event -results: wearer-entry and INT-yield AP, recall-matched false-positive rates, -confidence intervals, and oracle-gap recovery}. Controlled -\todo{time-shift/remove-vision/same-rate or capacity-matched} interventions -show that \todo{state the supported mechanism, without claiming addressee or -speaker identity}. These results demonstrate \todo{the narrow conclusion -actually supported by the final experiments}. +Egocentric conversational agents need timely predictions of an unseen +camera wearer's speech participation. We study causal local evidence +adaptation of a frozen Omni model, combining its integrated audiovisual +context with explicitly accessible recent audio and optional local visual +observations. The model predicts future activity every 200~ms at ten horizons +from 0.2 to 2.0~s. A bounded 8-s streaming context reuses completed audiovisual +blocks while recomputing the current partial tail at each cutoff; this exposes +newly observed evidence without waiting for the next complete block. +We evaluate dense activity forecasting and wearer onset/offset anticipation +separately, because improvements in continuous activity scores need not imply +equivalent improvements near state changes. +\todo{Insert matched multi-seed results and ablation-supported conclusions.} \end{abstract} \begin{keywords} -conversational agents, turn-taking, egocentric video, multimodal adaptation, -voice activity projection +egocentric speech anticipation, multimodal adaptation, conversational agents, +causal streaming, voice activity projection \end{keywords} \section{Introduction} \label{sec:intro} -Natural turn-taking requires a conversational system to anticipate not only -when speech will stop, but whether a floor opportunity actually belongs to it. -In a dyadic exchange, a pause after one interlocutor finishes often invites the -other to respond. In a group conversation, the same pause may instead mark a -handover between two other participants. An egocentric agent that reacts only -to silence or aggregate speech may therefore interrupt a conversation that -should continue without it. We study \emph{egocentric floor forecasting}: -continuously predicting the near-future speech participation of the -first-person wearer from strictly causal audio--visual observations. +A conversational agent must predict not only what to say, but when to +participate. In egocentric multiparty conversation, the camera wearer is +usually invisible, and the visible participants provide indirect context for +the wearer's next action. We use observed wearer speech as an imitation target, +forecasting participation rather than generating a response or committing +a turn-taking action. -Voice Activity Projection (VAP) introduced continuous prediction of future -voice activity for turn-taking \cite{ekstedt2022vap}. EgoSpeak brought this -formulation to causal, untrimmed first-person streams, using future camera- -wearer activity as an imitation target \cite{kim2025egospeak}. Complementary -work adds visual gaze, head pose, and facial behavior to predictive turn-taking -\cite{russell2025mmvap}, while MuVAP combines global acoustic dynamics with -visible face tracks to forecast visible floor holders in multiparty scenes -\cite{qi2026muvap}. Our target has a different role structure: the participant -whose future action is predicted is the one participant absent from the -visible set. The visible people are contextual evidence for the unseen -wearer's decision, not candidate output identities. +Voice Activity Projection (VAP) learns continuous future speech activity +\cite{ekstedt2022vap}. EgoSpeak extends activity forecasting to untrimmed +first-person audiovisual streams and studies modalities and context length +\cite{kim2025egospeak}. Visual predictive turn-taking +\cite{russell2025mmvap} and MuVAP's causal face-grounded multiparty projections +\cite{qi2026muvap} further motivate explicit local evidence. Our output, +however, is wearer-relative: visible people provide context, not candidate +speaker identities. We do not infer addressees or semantic floor ownership +from aggregate activity labels. -This role asymmetry exposes an adaptation problem for general-purpose Omni -models. Omni pretraining favors globally integrated representations useful for -multimodal understanding and generation. Egocentric floor control instead -depends on timely prosodic and visible-person activity at sub-second scale, -including whether other participants continue to occupy the floor. Increasing -model capacity or applying a generic parameter-efficient update does not -explicitly preserve or route this evidence to a dense wearer-relative -prediction head. The open question is therefore not merely whether an Omni -backbone can be applied to EgoSpeak, but what task structure is still required -to turn general audiovisual understanding into reliable floor control. +General-purpose Omni representations integrate audio, video, and broader +context \cite{xu2025qwenomni}, but their usefulness for dense anticipation +depends on how the prediction head accesses recent observations. +Two interfaces matter. First, a short-window modality encoder can expose +recent evidence separately from the integrated Thinker state. Second, a +blockwise stream must include its currently incomplete block if predictions +are required before the next block closes. These are representation-access +and alignment questions, not evidence that all Omni models lack turn-taking +ability. -Our candidate decomposition separates long-context understanding from -short-window floor evidence. A -causal global Omni state represents the broader scene and conversation. In -parallel, a lightweight local path encodes recent mixed audio and a changing, -anonymous set of visible-person observations. A residual adapter makes this -local evidence accessible to the common ten-horizon activity head. The primary -variant deliberately avoids cross-time identity: the current development -oracle supports timely aggregate visible-other activity but not person -continuity. Track-aware structure is considered only if it defeats no-ID, -identity-corrupted, and capacity-matched controls. +We organize both interfaces under \emph{causal local evidence adaptation}. +A frozen Omni path summarizes conversational context, while a recent-audio +path supplies explicitly accessible local evidence. Visual features are an +optional extension whose benefit must be measured rather than assumed. +The stream incorporates the observed partial tail at every decision time. +We retain EgoSpeak-compatible outputs and distinguish dense forecasting +from onset/offset anticipation. -The final paper will claim the following three contributions only if the -corresponding TODO gates pass: +The paper is organized around three contributions: \begin{itemize} - \item We identify and test a temporal--role mismatch between a global Omni - readout and causal egocentric floor forecasting for an invisible wearer. - \item We evaluate a wearer-relative causal local floor adapter that combines - long-context Omni information with short-window audio and anonymous - visible-person evidence. - \item We evaluate whether the adapter assigns lower erroneous-action scores - during multiparty floor transfers under an entry/INT protocol with point - positives and explicitly annotated hard-negative opportunity regions. + \item A global--local adaptation framework for invisible-wearer speech + anticipation, combining frozen Omni context with recent modality evidence + through a lightweight forecasting head. + \item A cutoff-aligned streaming realization with bounded context and + temporary partial-tail recomputation, separating completed-history reuse + from access to the latest observed input. + \item A controlled evaluation of dense forecasting and wearer onset/offset + anticipation, isolating local-evidence and streaming effects rather than + equating dense gains with improved semantic turn-taking. \end{itemize} +% Contributions describe implementation and study design. Mechanism claims +% require matched ablations; ordinary caching is not claimed as a new algorithm. \begin{figure*}[t] \centering -\includegraphics[width=0.98\textwidth]{figures/method_architecture.pdf} -\caption{Our strictly causal global--local architecture (top) and streaming -alignment (bottom). At every 5-Hz cutoff $t_n$, the global state includes the -latest partial 2-s Omni block, while every fifth causal 25-Hz local state -provides shared audio and an anonymous visible-person set. Masked person -attention contributes only through a zero-initialized residual to the matched -$G+U$ baseline.} +\includegraphics[width=0.96\textwidth]{figures/method_architecture.pdf} +\caption{Causal local evidence adaptation. Top: integrated Omni context and +recent audio feed a common ten-horizon head; local vision is optional. +Bottom: completed 2-s blocks are reused within an 8-s context budget, while +the observed partial tail is recomputed at each 200-ms cutoff. +No future media or temporary readout tokens enter committed history.} \label{fig:method} \end{figure*} @@ -125,287 +106,164 @@ $G+U$ baseline.} \subsection{Model Architecture} \label{sec:model_arch} -At decision time $t$, the model observes only the audiovisual stream -$\mathcal X_{\leq t}$ and predicts $K=10$ future bins spaced by -$\Delta=0.2$~s. For horizon $k$, the multilabel target -$\vect y_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes -background, camera-wearer activity, and activity by one or more other people; -wearer--other overlap is allowed. The output is -\begin{equation} - \hat{\vect Y}_t=f_\theta(\mathcal X_{\leq t}) - \in[0,1]^{K\times3}. - \label{eq:task} -\end{equation} +\textbf{Task definition.} +At cutoff $t_n=0.2n$~s, $n\geq1$, the model observes only media available by +$t_n$ and predicts activity at ten horizons, $0.2,0.4,\ldots,2.0$~s. +Each horizon has three channels in fixed order: background, camera wearer, +and aggregate others. Background indicates no speech; wearer and others may +both be active during overlap. The target is future activity, not a semantic +ENTRY/YIELD label. The tensor contains ten three-channel outputs per cutoff. -Our architecture complements globally integrated Omni context with explicitly -accessible local floor evidence (Fig.~\ref{fig:method}). Let -$\vect g_t\in\mathbb R^{d_g}$ denote the causal Qwen2.5-Omni Thinker readout -at $t$ \cite{xu2025qwenomni}. In parallel, the frozen MuVAP ASD frontend -\cite{qi2026muvap} produces one shared mixed-audio CPC state $\vect u_t$ and a -variable-size set of visible-person states -\begin{equation} - \mathcal P_t=\{\vect p_{t,i}\}_{i=1}^{N_t},\qquad - \vect p_{t,i}=[\vect v_{t,i};s_{t,i}], - \label{eq:person_state} -\end{equation} -where $\vect v_{t,i}$ is a causal visual state and $s_{t,i}$ is one -audio--visual speaking logit. The shared audio state is not duplicated in -each person token. The prediction model receives neither participant IDs nor -cross-time track identity; provided causal face boxes are used only to isolate -the representation and fusion question. +\textbf{Global and local features.} +The global path uses frozen Qwen2.5-Omni-3B \cite{xu2025qwenomni}. +We retain the final four Thinker readout-layer states, concatenate them, +and apply a layer-normalized projection with GELU to obtain a +2048-dimensional context vector $\vect g_n$. It can contain audio and visual +information; global context is not audio-free. -A matched global--audio head first maps $[\vect g_t;\vect u_t]$ to baseline -logits $\vect Z_t^{\mathrm{base}}\in\mathbb R^{K\times3}$ and an intermediate -state $\vect h_t$. The same inputs query the current person set: -\begin{equation} - \begin{aligned} - \vect q_t &= W_g\vect g_t+W_a\vect u_t,\\ - \mathcal M_t &= \{W_p\vect p_{t,i}\}_{i=1}^{N_t} - \cup\{\vect p_\varnothing\},\\ - \vect c_t &= \mathrm{MHA}(\vect q_t,\mathcal M_t,\mathcal M_t), - \end{aligned} - \label{eq:person_attention} -\end{equation} -where masking supports variable cardinality and the learned null token -$\vect p_\varnothing$ makes the attention well defined for an empty set. If -$m_t$ indicates that at least one valid person observation exists, the person -branch supplies only a residual logit correction, -\begin{equation} - \vect Z_t=\vect Z_t^{\mathrm{base}} - {}+m_t\,\mathrm{reshape}\!\left(W_r[\vect h_t;\vect c_t;m_t]\right), - \qquad W_r\vert_{\mathrm{init}}=0, - \label{eq:residual} -\end{equation} -and $\hat{\vect Y}_t=\mathrm{softmax}_c(\vect Z_t)$. Thus, after copying and -freezing the trained $G+U$ head, the complete model initially reproduces that -baseline exactly; only the person projection, query, masked attention, and -residual readout are trained in the adaptation stage. +The local audio vector $\vect a_n$ is extracted independently from the most +recent 1~s of mixed monaural audio, clipped to the available prefix at startup. +The frozen Qwen audio encoder processes this cropped waveform, and valid +output tokens are averaged into a 2048-dimensional vector. Cropping precedes +preprocessing, preventing access to future source samples. +This exposes recent evidence without another Thinker pass; it does not +separate participants or explicitly estimate prosody. -We use the EgoSpeak/LSTR multi-positive cross-entropy over causal-valid rows, -\begin{equation} - \begin{aligned} - \mathcal L&=-\frac{1}{M}\sum_{t,k,c}m_{t,k}y^c_{t,k} - \log\mathrm{softmax}_c(\vect Z_{t,k}),\\ - M&=\sum_{t,k}m_{t,k}. - \end{aligned} - \label{eq:loss} -\end{equation} -For an overlap target, both active channels contribute a log-probability term. -No transcript, target-of-speech annotation, close-microphone signal, speech -label, or future observation is supplied to the forecasting model. The -wearer forecast is interpreted only at evaluation time: it is an entry proxy -when the wearer is silent and its complement is a yield proxy when the wearer -is active, not a trained event hazard or committed controller action. +An optional visual vector $\vect v_n$ uses causal person crops sampled at +2~FPS. Spatial tokens are averaged within each crop, visible people receive +equal weight within a frame, and valid frames are averaged over the recent +1~s. The output is another 2048-dimensional vector. Provided boxes isolate +the representation question, not an online tracking system. Pooling does not +preserve identity or establish active-speaker or addressee grounding. -\subsection{Causal Streaming Architecture} +\textbf{Feature fusion and learning.} +Each enabled local vector is layer-normalized, projected to 256 dimensions +with GELU, and multiplied by its validity indicator. The indicator is also +concatenated into the head, distinguishing missing observations from valid +zero-valued evidence. The common forecast is +\begin{equation} + \hat{\vect Y}_n = + \mathrm{softmax}_{c}\! + \left(F_\theta\!\left[ + \vect g_n;\widetilde{\vect a}_n;m_n^{a}; + \widetilde{\vect v}_n;m_n^{v}\right]\right), + \label{eq:fusion} +\end{equation} +with visual terms omitted in the audio-core variant. +$F_\theta$ is a layer-normalized MLP with GELU, dropout, and a 30-logit output. +The same head family supports global-only, local-only, and fused controls; +parameter counts are reported because input dimensions differ. + +The encoders and Thinker are frozen; only the layer projection, local +projections, and head are trained. This is \emph{frozen-backbone adaptation}, +not end-to-end Omni fine-tuning. We use EgoSpeak/LSTR multi-positive +cross-entropy over causal-valid source--horizon pairs. Both active channels +contribute during overlap; no semantic-event objective is added. +Transcripts, participant speech labels, target-of-speech metadata, and LLM +annotations are never model inputs. + +\subsection{Streaming Inference Architecture} \label{sec:causal_streaming} -We distinguish encoder processing granularity from the common forecasting -rate. Predictions are emitted at $t_n=n\Delta$, $n\geq1$, while every -representation used at $t_n$ is restricted to media available by that physical -cutoff. +\textbf{Bounded history.} +The Omni path processes source-aligned 2-s audiovisual blocks within a maximum +8-s presentation budget. After warm-up, it retains three completed blocks +plus the current block, whose observed duration ranges from 0.2 to 2.0~s. +The available span therefore varies with phase: 8~s is a maximum, not an +exactly sliding crop. Older blocks are dropped at block transitions. +Retained completed payloads are replayed to rebuild the bounded language-model +cache; completed modality payloads are reused rather than re-encoded. +All caches reset at session or source discontinuities. -The global path uses an 8-s rolling causal presentation budget with -source-aligned 2-s audiovisual blocks. Completed blocks are cached and -replayed, while the currently incomplete block is a temporary tail. At every -200-ms cutoff, the tail is rebuilt with all newly observed audio and video and -included in computing $\vect g_{t_n}$; when it reaches 2~s, it is committed to -the reusable history. Consequently, the Omni state is not held constant for -2~s: it includes the latest available content at every decision cutoff without -observing media after $t_n$. We retain the final four Thinker query-layer -states and fuse them in the matched head with a concatenation MLP. +\textbf{Current-tail handling.} +At each 200-ms cutoff, the current block is cropped at the physical cutoff +and only observed audio and video are encoded. This incomplete block is the +\emph{tail}. It is appended to a temporary copy of the completed-history cache, +followed by a readout query to obtain the Thinker state. +The temporary tail and query are not committed. When a block completes, +its media payload is committed once; the next cutoff starts a new tail. +The state is thus refreshed at 5~Hz instead of being held until a 2-s block +closes. -The local frontend operates causally at 25~Hz. State $j$ consumes the -half-open interval $[j/25,(j+1)/25)$ and is timestamped at its right edge. -Mixed audio is encoded once as a continuous shared CPC stream, while each -available face crop is processed by the causal visual and audio--visual -speaking modules. Ten-second extraction chunks may prepend up to 2~s of past -context, never future context. At a 5-Hz forecasting cutoff, we select the -local state whose right edge equals $t_n$; hence every fifth 25-Hz state aligns -with one Omni prediction row. The two paths therefore describe the same -causal prefix: the Omni path preserves integrated conversational context, -whereas the local path exposes fine-grained audio and visible-person floor -evidence. Fusion and prediction occur only on the common 5-Hz grid. +Completed history and the tail are represented in one integrated Omni state, +not a separate tail-content or learned block-phase embedding. +% The auxiliary preserved-EgoSpeak residual has a learned phase embedding; +% its on/off ablation is separate from the core stream described here. +\textbf{Alignment and availability.} +Recent audio and optional vision are refreshed on the common 5-Hz grid and +joined by cutoff and target index, never by nearest future observation. +Only video frames available before the cutoff and truncated audio enter +preprocessing. No future completed block is backfilled into earlier rows. +Tail-aware refresh ensures access to observed input, not a measured +real-time throughput or predictive gain. Correctness, refresh effects, +and latency are assessed separately in Section~\ref{sec:analysis}. -\section{Experiments} +\section{Experimental Setup} \label{sec:experiments} -\subsection{Dataset and Data Preparation} +\subsection{Data and Training Protocol} -We evaluate on EasyCom following the EgoSpeak split, using Sessions 4--12 as -the training/development pool and Sessions 1--3 only for final evaluation -\cite{donley2021easycom}. The participant-resolved speech annotations are -converted into two parallel representations. First, they are downsampled to -5~Hz to construct the EgoSpeak-compatible ten-horizon targets for background, -camera-wearer, and aggregate non-wearer activity. Second, the original -participant timeline is retained to derive adjudicated interaction events for -behavioral evaluation. We additionally apply the annotation workflow to the -3,608 Ego4D AV validation candidates as a protocol stress test; Ego4D is not -used to train, tune, or compare the forecasting models reported here. +We use EasyCom \cite{donley2021easycom} with the EgoSpeak official split: +Sessions~4--12 for training and Sessions~1--3 for testing. +Targets follow the common 5-Hz grid and class order. +Training-side folds determine recipe changes; a session seen by a parent +checkpoint is not held-out validation for its continuation. +The final recipe is fixed before training on all nine training sessions. +Already inspected test runs are exploratory and are not used to choose +learning rates, windows, checkpoints, or ablations. -\textbf{Behavioral event annotation.} -We use every observed wearer activity transition as a high-recall semantic -candidate. For each wearer onset, nearby preceding non-wearer releases, the -intervening silence gap, and concurrent activity are retained as contextual -facts; the onset is labeled as wearer entry only when the subsequent wearer -speech constitutes a genuine floor take, including re-entry while another -speaker is active. For each wearer offset, nearby non-wearer onsets and offsets, -pre-existing and newly active speakers, overlap durations, and subsequent -speakers are retained without a timing-based takeover gate; the offset is -labeled as INT-yield only when a non-wearer genuinely takes over and the wearer -relinquishes the floor. We let an LLM jointly inspect -participant-resolved timing and transcripts to provide a semantic pre-label, -followed by mandatory human adjudication. Separately, we mine regions with no -wearer response that nevertheless constitute false-positive opportunities: -same-speaker pauses and other-to-other transitions for entry, and backchannels or -non-takeover overlaps for INT. Only human-confirmed positives and hard negatives -enter evaluation. Appendix~\ref{app:event_annotation} gives the full protocol. -Table~\ref{tab:behavior_annotation_stats} reports the current annotation -workload compactly; the EasyCom snapshot covers Sessions 1--4 only and is not -the final train/test annotation inventory. +The primary comparison includes the released EgoSpeak MiniROADA A+V reference, +Qwen audio encoder-only, global Omni $G$, audio-core $G+A$, and optional +$G+A+V$. Preserved-EgoSpeak-plus-Omni residual fusion is an auxiliary +architecture control, not the default method. Zero initialization guarantees +initial baseline recovery, not non-degradation after training. +Omni LoRA is deferred; the core uses frozen-backbone adaptation. +Qwen Omni 7B and MiniCPM-o 4.5 are planned transfer controls, not the core. -\begin{table}[h] -\caption{LLM pre-label workload before human adjudication. Agreement is -between two independent passes.} -\label{tab:behavior_annotation_stats} -\centering -\scriptsize -\begin{tabular}{@{}lrr@{}} -\hline -Measure & EasyCom S1--4 & Ego4D AV val \\ -\hline -Candidates & 2,128 & 3,608 \\ -AI positive & 367 & 541 \\ -AI hard negative & 1,017 & 1,174 \\ -AI excluded & 744 & 1,893 \\ -Two-pass agreement & 93.4\% & 82.1\% \\ -Human-confirmed & 0 & 0 \\ -\hline -\end{tabular} -\end{table} +\todo{Insert the locked recipe, provenance, trainable counts, and common +valid-pair counts. Report three seeds and per-session results.} -Participant identity is used only to construct these evaluation annotations -and is never provided to the prediction model. Provided person boxes are used -only for the causal visible-person input; transcripts, target-of-speech labels, -close-microphone audio, and future observations are unavailable to the model. -\todo{Report duration and event statistics: entry positives/hard negatives, -INT positives/hard negatives, excluded candidates, and audited-region counts -for train/development and test. Freeze the event-construction and region-audit -protocol before reporting semantic entry/INT results; until then, retain the -timing-derived proxy terminology.} +\subsection{Evaluation} -\subsection{Model Configuration and Training} +\textbf{Dense forecasting.} +For each horizon, valid frame scores are pooled across evaluated sessions +to compute AP for background, wearer, and others. Their arithmetic mean is +horizon mAP; averaging the ten horizon mAPs yields mean mAP. +Wearer AP is also reported separately. Comparisons use the intersection of +valid source--horizon pairs. -Our main backbone is Qwen2.5-Omni-3B. We compare a frozen global Omni readout, -matched PEFT/LoRA adaptation, global features with recent local audio, a -parameter- and history-matched temporal local branch, and the proposed causal -local audio--visual adapter. Qwen2.5-Omni-7B is used as a capacity-scaling -control. All learned variants use identical causal examples, prediction -horizons, prediction heads, and the multi-positive cross-entropy objective in -Eq.~\eqref{eq:loss}. +\textbf{Raw onset/offset anticipation.} +Onset evaluation uses source rows where the wearer is silent. A speaking +future endpoint is positive, scored by predicted wearer probability. +Offset evaluation uses source rows where the wearer is active. A silent +future endpoint is positive, scored by one minus predicted wearer probability. +Current ground-truth activity defines evaluation subsets only, not inputs. +We report AP per horizon and its ten-horizon mean for each direction. -The diagnostic streaming ablation additionally uses MiniCPM-o 4.5 -\cite{cui2026minicpmo45}. Its native cached stream is evaluated at its native -one-second update interval: a state becomes available to the 200-ms row whose -cutoff equals the chunk end and is then causally held until the next update. A -Qwen row subjected to the same 1-Hz update-and-hold schedule separates -refresh-rate effects from backbone effects. A MiniCPM-o -5-Hz cache row is admissible only if a cached branch at every cutoff agrees -with full causal-prefix recomputation within a frozen numerical tolerance and -passes future-input intervention tests. Thus the native 1-Hz condition is a -strictly causal held-state baseline, not a claim of native 5-Hz refresh. - -Model selection is performed using training-side data only. Event operating -points are selected from session-level out-of-fold predictions over -Sessions~4--12; the final model is then retrained on all nine sessions before -evaluation on Sessions~1--3. Final results are averaged over three random -seeds. -\todo{Freeze context length, local window/rates, optimizer, epochs, LoRA -configuration, trainable parameter counts, and inference latency.} - -\subsection{Evaluation Protocol} - -We first retain dense future-activity evaluation for direct comparison with -EgoSpeak, reporting AP at the 0.2--2.0~s horizons and their mean mAP. -As a threshold-free diagnostic, we also report wearer OFF$\rightarrow$ON and -ON$\rightarrow$OFF anticipation AP; these raw boundaries are timing proxies, -not semantic EOT or INT labels. - -For dense-only models, behavioral evaluation is opportunity-conditioned and -uses the model's original wearer-activity forecasts rather than treating them -as semantic intent probabilities. An entry opportunity begins at a reviewed -non-wearer release. It resolves positively when the wearer enters, or -negatively when the same non-wearer resumes or a different non-wearer takes the -next floor. From the first causal 5-Hz state that observes the release, the -entry score is the maximum projected wearer probability before that competing -resolution, capped at 2.0~s. A selected-release positive is excluded if another -non-wearer onset occurs before or exactly at the wearer entry, or if wearer -activity already occurred after the selected release. Positive entries without -a reviewer-selected release are reported as unanchored and excluded from this -matched-opportunity analysis; they remain available to the separate raw-onset -analysis. - -An INT opportunity begins when a non-wearer starts while the wearer holds the -floor. It resolves positively when the reviewed behavior is an INT-yield, and -negatively when the non-wearer contribution is a backchannel or failed -takeover and the wearer retains the floor. The yield score is the maximum of -one minus the projected wearer probability over the next 0.2--0.6~s, censored -strictly at the competing non-wearer offset. All prediction values are read -diagonally from one causal source row of the target-aligned forecast tensor. -No input after the observed action outcome is used. For current INT-positive -annotations, which do not select a competing onset explicitly, the -implementation evaluates only a unique supplied non-wearer interval crossing -the wearer offset; cases with multiple eligible intervals are excluded and -counted for audit rather than assigned an arbitrary onset. - -The headline entry hard negatives are same-speaker mid-turn pauses and other- -to-other transfers. Wearer backchannels are not pooled into headline false- -entry rate: their high wearer probability is correct under the dense training -target. We instead retain entry-versus-backchannel as a separate semantic- -selectivity diagnostic. Thus, the semantic annotations define which matched -opportunities are evaluated, while every model is scored solely through the -same EgoSpeak-compatible dense output. A capacity-matched downstream probe, -following MuVAP~\cite{qi2026muvap}, is secondary and is not needed for this -zero-shot activity-derived comparison. - -\textbf{False-positive scoring rule.} -A false positive is a hard-negative opportunity whose opportunity score -exceeds the frozen threshold, regardless of whether the wearer subsequently -exhibits an unrelated later onset or offset. Each opportunity contributes one -score, and competing-risk censoring prevents later conversational outcomes -from changing its label. Thus, false-positive evaluation includes situations -in which the correct wearer behavior is to remain silent or retain the floor. - -We report threshold-free proxy AP for entry and INT. The headline behavioral -operating point additionally reports false-positive rate at matched recall -$R=0.8$; thresholds are selected independently for entry and INT from training- -side out-of-fold predictions. The other-to-other hard-negative subset tests -whether local visual evidence specifically reduces erroneous wearer action -scores during multiparty floor transfers. \todo{Freeze the 2.0-s entry and -0.6-s yield caps using training-side data, audit selected-release and competing- -onset coverage, and verify identical censored/valid denominators across methods.} - -A deployment-style committed-event analysis with one-to-one matching and -signed latency is secondary and is omitted unless a threshold, persistence, -refractory period, matching interval, and competing-risk rule are all frozen -before test evaluation. The fixed-horizon protocol itself makes no detection- -latency claim. +These are \emph{endpoint state-change anticipation} metrics, not one-event- +one-vote timestamp detection: multiple source rows can correspond to one +boundary, and reversals before the endpoint are not captured. +Dense and conditional AP have different prevalence and are not directly +comparable in absolute value. Positive counts and denominators are reported. +AP is threshold-free and requires no calibration session. +\textbf{Scope.} +Semantic ENTRY/YIELD opportunity evaluation is deferred during annotation +auditing. Provisional results and annotation workload are retained outside +the submission body. Activity transitions do not establish intent, takeover, +or floor ownership; we make no semantic control, false-trigger, or event +detection latency claim. \section{Results and Analysis} -\label{sec:results} +\label{sec:analysis} \subsection{Main Comparison} -Table~\ref{tab:main} places the released EgoSpeak checkpoint beside the -currently available Omni screening runs and reserves a row for the final causal -local audiovisual model. Historical unmatched runs are identified explicitly -and cannot support confirmatory attribution. The final comparison will use the -same split, causal inputs, targets, training/selection protocol, and three -random seeds; event thresholds will be selected on training-side calibration -data and frozen before test evaluation. \todo{Freeze all baseline -implementations and verify identical targets, causal validity, and evaluation.} +Table~\ref{tab:main} retains exploratory references and reserves the locked +comparison. Historical runs have unmatched training/selection protocols; +the three-seed F1 screen establishes stability, not matched superiority. +\todo{Fill the locked rows with three-seed mean and sample standard deviation.} \begin{table*}[t] \caption{EasyCom Sessions~1--3 dense future-activity and raw-boundary @@ -440,348 +298,142 @@ Omni $G{+}A{+}$pooled-$V^{\dagger}$ Current-frame F1$^{\ddagger}$ & $81.78{\pm}0.16$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & $64.95{\pm}0.09$ & -- & --\\ -$G+$causal local AV (ours) +Locked local $A$ + & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- + & -- & -- & --\\ +Locked Omni $G$ + & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- + & -- & -- & --\\ +Locked $G+A$ + & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- + & -- & -- & --\\ +Locked $G+A+V$ (optional) & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & --\\ \hline \end{tabular}} \end{table*} -\textbf{Dense forecasting.} -The EgoSpeak baseline reaches 74.20\% 0.2-s mAP and 61.19\% mean mAP; its raw -wearer-onset/offset mean APs are 24.49/38.02\%. Opportunity-conditioned values -remain pending a protocol-v2 rerun and human review; continuous wearer-score -plots do not alone establish semantic floor-yield selectivity. -The proposed local AV adapter improves 0.2-s mAP from -\todo{$X$ to $Y$} over the strongest matched baseline and improves -mean mAP by \todo{$Z$ points}. -The gain is largest at \todo{short prediction horizons / transition-adjacent -frames / specify actual observation}, indicating that local audiovisual -evidence primarily improves \todo{the interpretation supported by the data}. +Alongside these models, a planned causal current-activity classifier repeats +its predicted probabilities across horizons as a persistence reference. +It uses no ground-truth current activity as input. +\todo{Report its dense and endpoint APs; distinguish state recognition from +anticipation.} -\textbf{Annotated behavior.} -At matched wearer-entry recall $R=0.8$, the proposed adapter reduces entry FPR -from \todo{$X$} to \todo{$Y$}; at matched INT-yield recall, it changes INT FPR -from \todo{$A$} to \todo{$B$}. Its entry/INT opportunity APs are -\todo{$E/I$}. These recall-matched opportunity error rates test whether lower -scores on confusing floor situations are obtained without suppressing -legitimate entry or yield events. +\subsection{Local Evidence Analysis} -\textbf{Comparison with adaptation baselines.} -Matched PEFT yields \todo{result}, whereas adding recent local audio alone -yields \todo{result}. -The parameter- and history-matched temporal branch achieves -\todo{result}. -The remaining gap between this control and the proposed adapter therefore -suggests that \todo{correctly timed audiovisual local evidence contributes -beyond additional temporal capacity}. -\todo{If this difference is not statistically supported, state instead that -the local temporal branch accounts for most of the gain.} +The $A/G/G+A/G+A+V$ comparisons separate recent audio, integrated context, +and optional vision. We will audit parameter counts and train a wider +global-only head as a capacity control. These branches expose different +history ranges, so the comparison is not a pure Thinker-versus-encoder test. +Past-delayed local inputs provide separate fixed-checkpoint sensitivity +tests, not substitutes for retrained modality-removal controls. +\todo{Insert audio and visual marginal deltas in wearer, onset, and offset +AP; retain negative findings even if overall mAP increases.} -\subsection{Qualitative Floor-Event Analysis} +\textbf{Backbone transfer.} +We reserve paired $G/G+A$ controls for Qwen2.5-Omni 3B/7B and MiniCPM-o 4.5, +fixing the same recent-audio features. Within-backbone deltas test transfer; +absolute scores also reflect pretraining and interface differences. +Readout, refresh, and actual history must be audited. Native 1-Hz hold needs +a matched Qwen 1-Hz control; MiniCPM 5-Hz tail rows require GPU verification. -We will compare EgoSpeak with the final matched model on human-confirmed -examples selected by pre-declared event subtype rather than post-hoc score -extremes. Each case will align participant-resolved activity, minimal transcript -context, the opportunity anchor and competing-risk censor, and the two models' -causally emitted wearer forecasts. The comparison will include a genuine wearer -entry, an other-to-other false-entry opportunity, and an INT-yield paired with a -non-takeover overlap; a shared failure case will be retained when space permits. +\subsection{Streaming and Context Analysis} + +Table~\ref{tab:ablation} reserves a tail-refresh $\times$ local-audio +comparison. Completed-only states update at 2-s closures and are causally +held between them; tail-aware states use the observed partial block at every +cutoff. Enabled local audio remains 5~Hz. Common startup rows and valid pairs +are used. This tests whether the two paths are complementary or redundant. + +\begin{table}[t] +\caption{Planned controls. M/W: mean mAP/wearer AP; On/Off: mean endpoint +AP (percent). Transfer reports $G+A-G$ deltas (points) after interface audits. +Dashes are unmeasured, not zero.} +\label{tab:ablation} +\centering +\scriptsize +\setlength{\tabcolsep}{3pt} +\begin{tabular}{@{}lcccc@{}} +\hline +Condition & M & W & On & Off\\ +\hline +\multicolumn{5}{l}{\emph{Tail refresh $\times$ local audio; 8-s cap}}\\ +Completed-only $G$ & -- & -- & -- & --\\ +Completed-only $G+A$ & -- & -- & -- & --\\ +Tail-aware $G$ & -- & -- & -- & --\\ +Tail-aware $G+A$ & -- & -- & -- & --\\ +\hline +\multicolumn{5}{l}{\emph{Context cap; tail-aware $G+A$}}\\ +4~s & -- & -- & -- & --\\ +8~s & -- & -- & -- & --\\ +16~s & -- & -- & -- & --\\ +\hline +\multicolumn{5}{l}{\emph{Backbone transfer; paired $G+A-G$ deltas}}\\ +Qwen2.5-Omni-3B & -- & -- & -- & --\\ +Qwen2.5-Omni-7B & -- & -- & -- & --\\ +MiniCPM-o 4.5 & -- & -- & -- & --\\ +\hline +\end{tabular} +\end{table} + +The planned 4/8/16-s context-cap comparison keeps local evidence at 1~s +and reports actual history spans and block phases. Each cap requires its +own extracted states and matched training; a fixed-checkpoint context +intervention is a separate distribution-shift diagnostic. +Cache reuse is checked against the same block decomposition, with +future-media interventions and reset checks. +\todo{Insert tail/audio interaction, window and phase results, and full +pipeline P50/P95 latency, throughput, and peak memory.} + +\subsection{Error and Qualitative Analysis} + +We will stratify activity errors around other-speaker pauses, other-to-other +transitions, and overlap, reporting positive/negative counts at fixed +horizons. Pure-negative slices receive score distributions, not AP; +any FPR uses training-side calibration. A wearer backchannel can be a raw +onset positive despite being a semantic ENTRY negative. Semantic +interpretations remain deferred pending annotation audit and protocol v2. +\todo{Insert scene-level counts and score/AP differences, plus one shared +failure; do not infer floor ownership from activity errors.} + +Figure~\ref{fig:qualitative_floor_events} reserves cases chosen by +pre-declared rules, aligning source cutoffs and corresponding future target +times at the same horizon for EgoSpeak and the selected ablations. \begin{figure*}[t] \centering -\fbox{\parbox[c][0.115\textheight][c]{0.965\textwidth}{ +\fbox{\parbox[c][0.065\textheight][c]{0.965\textwidth}{ \centering -\textbf{Qualitative comparison placeholder}\\[0.7em] -\begin{tabular}{ccc} -Case A: wearer entry & Case B: other$\rightarrow$other HN -& Case C: INT-yield / failed takeover\\ -participant timeline + transcript & opportunity anchor + censor -& EgoSpeak vs. proposed forecast\\ -\end{tabular} +\textbf{Error and qualitative comparison placeholder}\\[0.5em] +Wearer onset \qquad Wearer offset \qquad Unchanged state / shared failure\\ +Activity timeline + source cutoff + same-horizon wearer forecasts }} -\caption{Planned qualitative floor-event comparison. Final panels will use -human-confirmed protocol-v2 cases and show the same causal source time for both -models; the figure is a layout placeholder and contains no result claim.} +\caption{Planned onset, offset, and hard-negative/shared-failure timelines. +Show raw activity and same-horizon forecasts, not an envelope of ten horizons. +This placeholder contains no result; semantic panels await audited labels.} \label{fig:qualitative_floor_events} \end{figure*} -\subsection{Information Opportunity} - -Before designing the deployable adapter, we measured whether local floor -information was useful at all using an analysis-only oracle. -The oracle receives perfectly timed visible-other speech activity through a -non-deployable pathway and is never used in the main comparison. - -On development Session~12, aggregate visible activity improved 0.2-s mAP by -6.97 points and ten-horizon mean mAP by 2.91 points over the matched -global-plus-local-audio baseline. -In contrast, visibility alone and one-second-delayed activity remained close -to the audio baseline, while person-specific histories did not outperform -identity-shuffled histories. - -These controls indicate that the useful signal is primarily -\emph{timely local floor evidence}, rather than participant identity or -static visibility. -The proposed adapter is therefore designed to approximate this information -using only causally available audiovisual observations. - -\todo{After completing all seeds, report mean $\pm$ confidence interval here. -If the oracle effect is unstable, describe it qualitatively and omit the exact -numbers from the four-page version.} - -\subsection{Mechanism Analysis} - -To determine whether the gain arises from correctly aligned visual floor -evidence rather than additional parameters or generic visual context, and to -separate streaming refresh from representation family, we evaluate the -controls in Table~\ref{tab:ablation}. - -\begin{table}[t] -\caption{Mechanism and streaming/cache controls. The streaming panel uses -frozen global states and capacity-matched downstream heads; all rows share -targets and evaluation. FPRs are measured at matched recall $R=0.8$. -Cross-family raw scores are diagnostic.} -\label{tab:ablation} -\centering -\small -\resizebox{\columnwidth}{!}{% -\begin{tabular}{lcccc} -\hline -Condition & Mean mAP $\uparrow$ & Entry FPR $\downarrow$ -& O$\rightarrow$O FPR $\downarrow$ & INT FPR $\downarrow$\\ -\hline -\multicolumn{5}{l}{\emph{Local-evidence controls (Qwen2.5-Omni-3B)}}\\ -Remove vision & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -Time-shift vision & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -Full-frame vision & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -Correct local AV & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -\hline -\multicolumn{5}{l}{\emph{Streaming/cache controls}}\\ -Qwen, full causal prefix at 5 Hz (no cache) - & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -Qwen, cached 5-Hz update & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -Qwen, cached 1-Hz + causal hold & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -MiniCPM-o, native cached 1-Hz + causal hold - & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -MiniCPM-o, audited cached 5-Hz branch - & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -\hline -\end{tabular}} -\end{table} - -Removing local vision changes \todo{entry/O$\rightarrow$O/INT FPR by $X/Y/Z$}, -while shifting the visual stream by \todo{$\Delta t$} produces \todo{result}. -Replacing person-local evidence with full-frame vision produces \todo{result}. -If aligned local AV improves estimates of surrounding floor occupancy, its -largest selective reduction should appear on the O$\rightarrow$O hard-negative -subset. Together, these comparisons \todo{support / do not support} that -mechanism beyond generic visual context. - -\todo{Report paired intervals and the supported interpretation. If correct -and shifted/removed vision do not separate, drop the audiovisual -evidence-routing claim. If dense mAP improves but event behavior does not, do -not claim improved floor control. If legitimate onset or yield degrades, do -not describe lower false entry as success.} - -\subsection{Scaling and Streaming Controls} - -Increasing the frozen Omni backbone from 3B to 7B yields -\todo{$X$ points} in 0.2-s mAP and \todo{entry/INT FPR of $Y/Z$} at matched -recall. -The proposed 3B adapter \todo{remains better than / approaches / is surpassed -by} the 7B scaling control. - -This result suggests that \todo{architectural access to local causal evidence -provides a complementary benefit to backbone scaling}. -Because both backbones belong to the Qwen2.5-Omni family, we do not claim that -this conclusion generalizes to all omni-modal models. - -The streaming rows answer two different questions. First, full causal-prefix -recomputation and streaming cache reuse at the same 5-Hz Qwen cutoffs test -whether cache/chunk semantics change the representation; equivalent outputs -make this an efficiency control rather than a distinct predictive method. -Second, the matched Qwen 5-Hz and 1-Hz cached rows test whether an advantage is -explained by more frequent causal state refresh rather than the proposed local -evidence route. For native 1-Hz streams, the state emitted at the end of each -one-second chunk is never backfilled into earlier 200-ms rows; source gaps and -session boundaries reset all modality and language-model caches. MiniCPM-o is -a secondary cross-family diagnostic under the same 1-Hz schedule, not a -same-frequency architecture ranking. Its 5-Hz cached-branch row is omitted if -cached/full-prefix equivalence, cutoff invariance, or reset audits fail. - -\todo{Freeze native chunk boundaries, initialization of rows before the first -complete chunk, cache-reset rules, the cached/full-prefix tolerance, and the -future-media intervention test. Report within-backbone 5-Hz versus 1-Hz -deltas plus cache/no-cache throughput and memory; do not interpret unmatched -Qwen--MiniCPM raw scores as a pure backbone effect.} - \subsection{Limitations} -The results support \todo{precise supported claim}, but several limitations -remain. -First, \todo{EasyCom size/session diversity limitation}. -Second, the adapter relies on \todo{state the actual visible-person box or -tracking assumption}. -Third, \todo{failure condition: occlusion, overlapping speakers, rapid floor -exchange, etc.} remains challenging. - -\todo{Replace Fig.~\ref{fig:qualitative_floor_events} with compact, pre-selected -human-confirmed timelines. Examples illustrate aggregate rates and never -establish the claim by themselves.} +EasyCom has only three official test sessions. The backbone remains frozen, +and local audio averages tokens rather than explicitly modeling prosody. +Optional pooling does not establish identity or addressee understanding. +Tail recomputation may increase cost despite completed-block reuse. +Visual and tail-mechanism conclusions require the matched controls above. \section{Conclusion} -\label{sec:conclusion} -This study asks whether a general-purpose Omni representation can be adapted -for causal floor forecasting of an invisible egocentric wearer. The candidate -method separates long-context Omni understanding from short-window local floor -evidence without assuming persistent speaker identity. \todo{State the final -learned-method improvement, matched-recall entry/INT false-positive rates, -fixed-horizon event AP, causal-control evidence, and main limitation. If no learned -variant passes these tests, reframe the submission as an information audit or -do not make the method claim.} - -\appendix -\section{LLM-Assisted Behavioral Event Annotation} -\label{app:event_annotation} - -EasyCom's raw speech boundaries are acoustic annotations rather than floor -labels. We preserve them unchanged for dense forecasting and create a separate -behavioral-event layer. Its design principle is to anchor an observed wearer -behavior first and then retrieve the conversational opportunity that could make -that behavior meaningful. False-positive opportunities are mined independently -because, by definition, many contain no wearer transition. - -\subsection{Positive-event mining} - -\textbf{Wearer activity transitions.} -We preserve every raw wearer $0\!\rightarrow\!1$ and $1\!\rightarrow\!0$ -boundary as a semantic review candidate. Candidate construction does not merge -short silence gaps, infer within-turn resumption, or decide whether overlapping -speech constitutes takeover. It only records participant-resolved temporal and -transcript facts. The source intervals and dense targets remain unchanged. - -\textbf{Wearer entry.} -Every raw wearer onset at $t_b$ is proposed as one wearer-entry review item. -We search backward within the retrieval window for non-wearer offsets and store -each one with its response delay and factual indicators of intervening wearer -speech, intervening other onsets, and concurrent activity. These are possible -context anchors rather than deterministically admissible causes. The LLM and -human reviewer decide whether the boundary is a genuine new entry, a -within-turn resumption, a wearer backchannel, or an excluded non-event. A -preceding release is not required for a positive entry: the wearer may enter -after a release, from silence, or while another speaker remains active. When a -release clearly prompts the entry, the LLM or human may select one supplied -option; neither may invent or move a timestamp. - -\textbf{INT-yield.} -Every raw wearer offset $t_y$ is proposed as an INT-yield review item. The -generator records the current wearer activity-run start, all nearby non-wearer -onsets and offsets, activity immediately before and after $t_y$, whether an -other speaker is pre-existing or begins during wearer speech, strict and -50-ms-tolerant boundary crossing, overlap duration, the next speech onset, -transcripts, and target-of-speech metadata. Relations such as -$t_w