Freeze targeted introduction storyline

2026-09-24 17:15:16 +08:00
parent 98b3ad2d69
commit 9721e6c409
3 changed files with 143 additions and 100 deletions
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -21,36 +21,46 @@
 跨过下一次 other onset，yield 不能跨过下一次 other offset。在该问题修正前，事件结果
 只能称为 timing-derived proxy。

-## Targeted manuscript 的固定故事主线
+## Targeted manuscript 的 Introduction 冻结叙事

-写作入口是**第一视角会话代理的 when-to-speak 预测**：相机佩戴者通常不在画面里，
-模型须从严格因果的多人音视频流预测其未来讲话活动。先概述活跃的 turn-taking/VAP、
-多方对话和流式语音交互研究，再准确定位缺口：第一视角 AV 已有 EgoSpeak、
-3M-Transformer、AV-CONV 等先例，不能宣称“首个 egocentric AV turn-taking”；
-当前要研究的是单个 wearer 的 5 Hz 未来活动预测中，Omni 全局状态能否及时暴露
-局部视听证据。相关工作按轮次预测、第一视角视听、流式 Omni/编码器适配三组组织，
-以约 25--30 篇经过核对的文献支撑，而非堆砌引用。具体边界见
-[`docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md`](docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md)。
+以下四段的顺序、问题定义和论证主线已经确定。除非用户明确要求，后续不再重排
+Introduction 的故事或改换核心研究问题；可继续核实文献、修正证据边界、精炼学术措辞和
+调整排版。以 [`manuscript/targeted/main.tex`](manuscript/targeted/main.tex) 的 Introduction
+为正文基准，不把计划写成实测。

-方法叙事是两个**截止时刻对齐的接口**，不是声称所有 Omni 都缺乏局部感知：
+1. **任务与动机。** 从第一视角会话代理能够回答现场问题、补充语境、辅助佩戴者出发，
+   说明它不仅要决定说什么，还要决定何时开始、等待、停止说话。判断依赖严格因果的
+   音频、可见行为和语言线索；人数与角色随场景变化。两名可见的人彼此交谈不应触发
+   代理回复，且 wearer 通常不在画面中。因此关注的是 wearer 的未来讲话活动，而非
+   当前可见说话人身份、addressee 或语义 EOT。
+2. **相关工作与精确缺口。** 原先分开的相关工作段落合为一个自然段，不加黑体引导词。
+   连贯比较 VAD 的语音活动、双人 VAP 和 Easy Turn/FastTurn 的轮次预测、双人 AV、
+   固定三人和可变人数研究、MuVAP 的第三方静态视角及人脸轨迹、缺乏视觉关系信息的
+   device-directed speech，再承认 EgoSpeak 已在 EasyCom/Ego4D 做第一视角 wearer
+   未来语音预测，以及 3M-Transformer、AV-CONV、SICNet 和 active-speaker 工作的
+   不同目标。缺口是：自然多人、单个第一视角、严格因果的流中，**集成式流模型如何
+   保留预测截止时刻附近的证据，以预测不可见 wearer 的未来活动**。不得声称首个
+   egocentric AV turn-taking；文献和 novelty 边界见
+   [`docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md`](docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md)。
+3. **Omni 进展与局限。** 说明模态编码器、共享语言模型状态提供全局音视频与语义上下文；
+   具体介绍 Qwen2.5-Omni 3B/7B、分块处理、Thinker/Talker。本研究实测的是冻结 3B
+   Thinker readout，7B 仅为计划控制。在本项目配置中，原生 2 秒音频块相对 200 ms
+   预测频率较粗：只复用完成块会使局部证据变旧，强行每 200 ms 提交则改变原生块表示。
+   多层集成状态可能不如编码器特征直接暴露瞬时线索，但仍有全局上下文优势；不能
+   泛称所有 Omni 模型都缺乏局部感知。
+4. **本文观点与三项贡献。** 先提出保留 Omni 全局状态、同时在每个因果截止点显式接入
+   近期模态证据的观点，再以冒号引出三项贡献：(1) SWE（short-window encoder
+   evidence，短窗编码器证据）将近期音频与可选的因果人物框视觉特征接入冻结 Omni
+   预测头；“短窗”指时间局部，不是浅层网络或任务级端到端训练。(2) PTR（partial-tail
+   refresh，部分尾块刷新）在复用已完成块时只查询未完成块已观测前缀，不把它提交到
+   持久历史；其收益须由相同 cutoff、相同上下文上限的配对实验验证。(3) EasyCom
+   细致分析与 Ego4D 同协议迁移评估。当前只有 EasyCom 已归档筛查结果；Ego4D 分数、
+   匹配 PTR 消融和可靠事件级行为结论仍待完成，不能写成已验证的提升。

- **SWE（short-window encoder evidence，短窗编码器证据）**：把最近 1 秒的音频
-  编码特征及可选的因果人物框视觉特征直接接到冻结 Omni readout 的预测头。
-  “短窗/时间局部”指观测范围，不称为浅层网络特征，也不把冻结骨干加 head 写成
-  端到端训练。
- **PTR（partial-tail refresh，部分尾块刷新）**：在复用已完成块的同时，于每个
-  200 ms 截止时刻只查询当前未完成 2 秒块的已观测前缀。2 秒是本项目 Qwen 提取
-  的块设置，相对 200 ms 决策间隔较粗；不能推广为所有 Omni 模型的固定块长。
-  PTR 的独立准确率收益和运行延迟须经相同 cutoff、相同上下文上限的配对实验验证。
-
-论文按 Introduction/Related Work → Method → Experiments → Results and Analysis →
-Conclusion 展开。EasyCom 用官方 Sessions 4--12/1--3 划分做 0.2--2.0 秒十个
-horizon mAP、raw onset/offset、局部音视频与 oracle/时序消融；Ego4D 用于同协议
-迁移检验。**当前只有 EasyCom 已归档筛查分数；Ego4D 测试分数、匹配的 PTR
-消融和可靠的事件级行为结论均未完成**。稿件可写明 Ego4D 实验协议与待填表，
-不得写成已获得跨数据集提升；raw endpoint AP 不能替代语义 EOT/INT。将实测、
-历史非匹配 screen、分析用 privileged oracle、待测方案明确分开。targeted 正文见
-[`manuscript/targeted/main.tex`](manuscript/targeted/main.tex)。
+后续 Method → Experiments → Results and Analysis → Conclusion 顺着这条主线展开。
+EasyCom 遵循官方 Sessions 4--12/1--3 划分；十个 0.2--2.0 秒 horizon 的活动预测、
+raw onset/offset、局部音视频和 oracle/时序消融应区分实测、历史非匹配 screen、
+仅用于分析的 privileged oracle 与待测方案。raw endpoint AP 不等同语义 EOT/INT。

 ## 不可破坏的研究契约

--- a/manuscript/targeted/main.tex
+++ b/manuscript/targeted/main.tex
@@ -54,82 +54,80 @@ streaming, voice activity projection, conversational agents
 \section{Introduction}
 \label{sec:intro}

-An egocentric conversational agent must decide \emph{when} to speak even
-though its camera wearer is usually invisible.  It hears a mixed conversation
-and sees other participants, but must anticipate its own speech from the
-observed stream before a turn boundary is known (Fig.~\ref{fig:overview}).
-This decision differs from recognizing who is speaking now: the relevant
-evidence may be a brief pause, overlap, or response by someone in view, while
-the target is the wearer's \emph{future} activity.  It also differs from
-predicting a semantic end of turn or selecting an addressee.  We therefore
-study a deliberately narrower and measurable question: at each 200-ms
-cutoff, how well can a causal observer forecast background, wearer, and
-aggregate-other activity over the next 0.2--2.0~s?
-Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic
-extensions, backchannel prediction, online end-of-turn detection, and event
-benchmarks address increasingly realistic timing decisions
-\cite{ekstedt2022vap,russell2025mmvap,elmers2025triadic,
-lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}.
-Their input viewpoints and target decisions, however, are not interchangeable
-with forecasting the future activity of an unseen wearer.
-The distinction matters when comparing systems: a correct current-speaker
-label or an utterance-final response can conceal errors in the first few
-hundred milliseconds, precisely where a wearable agent must decide whether
-to wait or begin speaking.
+An egocentric conversational agent could answer situated questions, supply
+context, and assist its wearer during face-to-face conversation. To
+participate naturally, it must decide not only \emph{what} to say but when to
+begin, wait, or stop speaking. This requires interpreting mixed audio,
+visible people's behavior, and linguistic content as the number and roles of
+interlocutors change. A pause is not necessarily an invitation: two people
+visible to the wearer may be talking to each other, and their turn exchange
+should not trigger the agent's response. Since the wearer is normally
+off-camera, the relevant target is not simply which visible face speaks now,
+but whether the wearer is about to speak or cease speaking.
+Both boundaries matter: the agent should withhold an entry during a
+third-party exchange and yield when another participant takes the floor.
+From a moving first-person view, face visibility and overlapping voices can
+change independently; neither a silence threshold nor one detected face
+provides a fixed map of who is interacting with the wearer. These timing
+errors arise before response content is considered.

-\textbf{Egocentric audiovisual conversation.}
-EgoSpeak is the closest wearer-centered predecessor: it extracts audio and
-RGB features with specialized encoders, fuses their temporal evidence, and
-forecasts wearer speech on EasyCom and Ego4D \cite{kim2025egospeak,
-donley2021easycom,grauman2022ego4d}.  Other first-person work addresses
-multi-view next-speaker prediction, speaking/listening graphs, or addressee
-detection \cite{serj2024threem,jia2024avconv,kong2024sicnet}.
-Egocentric active-speaker methods \cite{jiang2022asl,huh2025egoasd,
-clarke2025asd} and long--short person modeling \cite{wang2024loconet}
-recover present speaker evidence; audiovisual cues can also help turn
-prediction in noise \cite{oconnorrussell2025noise}.  These studies make a
-broad ``first audiovisual turn-taking'' claim untenable.  What remains
-less examined is how a single causal first-person stream should expose
-recent audio--visual evidence to forecast the invisible wearer's activity.
-EgoSpeak establishes that compact audio and video representations can be
-fused for this task, but it does not answer whether a large integrated Omni
-representation makes those cutoff-local cues readily available to a small
-forecasting head.  Conversely, work on active speakers or addressees need
-not solve wearer anticipation: the visible interlocutor and the predicted
-speaker occupy different roles.  This leaves an architectural question,
-rather than a claim that audiovisual evidence itself is new.
+Prior work addresses complementary parts of this decision. Voice activity
+detection (VAD) detects speech presence \cite{sohn1999vad}; dyadic voice
+activity projection (VAP) predicts future speaker activity
+\cite{ekstedt2022vap}, while Easy Turn and FastTurn combine acoustic and
+linguistic evidence for spoken-agent turn detection
+\cite{li2026easyturn,wang2026fastturn}. Audiovisual turn and backchannel
+prediction exploits visible cues but is commonly evaluated with two
+interlocutors \cite{russell2025mmvap,lin2025backchannel,
+oconnorrussell2025noise}. Triadic VAP extends audio forecasting to fixed
+three-person conversations
+\cite{elmers2025triadic}; online three-party EOT has also used controlled
+multimodal sensor data \cite{lee2024eot}. MuVAP handles a variable number of
+speakers using single-camera video and face tracks, but studies visible
+speakers from a
+static third-party view \cite{qi2026muvap}. Device-directed speech detection
+can combine acoustics, ASR output, and text to ask whether an utterance
+addresses an assistant, yet the cited model does not observe visual
+relations among interlocutors \cite{wagner2024ddsd}. EgoSpeak directly
+forecasts the unseen wearer's future speech from causal
+first-person audio--video on EasyCom and Ego4D \cite{kim2025egospeak,
+donley2021easycom,grauman2022ego4d}. Multi-view next-speaker prediction,
+first-person speaking/listening graphs, and addressee detection provide
+related but distinct targets \cite{serj2024threem,jia2024avconv,
+kong2024sicnet}. Egocentric active-speaker localization likewise concerns
+current speaker evidence \cite{jiang2022asl,huh2025egoasd,clarke2025asd}.
+Together, these studies leave open how an integrated streaming model should
+retain cutoff-local evidence for forecasting the unseen wearer's activity in
+a natural multiparty, first-person stream. This is a representation-and-timing
+question, not a claim to first egocentric turn prediction. Our activity
+labels do not establish addressee or semantic floor ownership.

-\textbf{Streaming Omni and temporal access.}
-Omni systems integrate modalities and support streaming interaction
-\cite{xu2025qwenomni,cui2026minicpmo45,tian2026roma,yan2026duplug,
-yu2025salmonnomni,chen2024videollmonline,wang2025omnimmi}.  Encoder-side
-audio--visual adaptation offers another route to accessible modality evidence
-\cite{lin2023lavish,rouditchenko2024whisperflamingo}.  Qwen2.5-Omni uses
-block-wise audio processing \cite{xu2025qwenomni}; in our extraction, a 2-s
-block is long relative to a 200-ms decision interval.  A streaming interface
-alone therefore does not specify how fresh the integrated representation is
-at each forecast cutoff.  We study two complementary, cutoff-aligned
-interventions: \emph{short-window encoder evidence} (SWE) directly exposes
-recent modality features outside the Thinker state, while \emph{partial-tail
-refresh} (PTR) queries the observed prefix of the incomplete current block.
-SWE denotes temporal locality, not shallow encoder layers; the argument
-concerns this pipeline, not an intrinsic deficit of every Omni model.
-These interventions target different bottlenecks.  SWE asks whether recent
-encoder evidence improves prediction when the global state is held fixed;
-PTR asks whether the same integrated backbone benefits from a fresher readout
-without admitting future media.  A gain from one does not establish a gain
-from the other.  In particular, the effect of PTR requires a matched
-completed-only comparison at the same context cap, rather than comparison
-with a longer or shorter history.
+Omni models couple modality encoders to a shared language-model state, so
+audio, video, and text can inform streaming interaction and longer-range
+semantic context \cite{xu2025qwenomni,tian2026roma,chen2024videollmonline}.
+Qwen2.5-Omni has released 3B and 7B variants. Its block-wise encoders and
+time-aligned audio--video interleaving feed a Thinker for integrated
+understanding; a separate Talker generates speech \cite{xu2025qwenomni}.
+We use frozen 3B Thinker readouts; 7B is a planned transfer control, not a
+measured result. The native audio attention block spans 2~s, whereas our
+wearer forecast updates every 200~ms. Holding only completed blocks leaves
+stale context between boundaries; forcing 200-ms commits changes the native
+block construction and may alter its representation. Neither implies that
+all Omni models lack local evidence. An integrated, multi-layer state can
+carry broader audiovisual and linguistic context, yet transient cues may be
+less accessible to a small head than encoder-side features themselves
+\cite{lin2023lavish,rouditchenko2024whisperflamingo}.

-This framing yields three concrete contributions.  First, we formulate a
-causal, 5-Hz wearer-activity forecast and specify SWE and PTR as separable
-ways to expose local evidence to a frozen Omni model.  Second, on EasyCom we
-compare global, recent-audio, and pooled-visual features, then use privileged
-past visible activity only as an information audit, not a deployable input.
-Third, we specify cross-dataset, endpoint, and hard-slice tests.  EasyCom
-screening is the only measured evidence; matched PTR, Ego4D transfer, and
-event behavior remain open.
+We therefore retain the global Thinker state while exposing recent modality
+evidence at each causal cutoff (Fig.~\ref{fig:overview}). EasyCom screens
+suggest that global context and local audio are complementary; gains from
+pooled vision and isolated PTR effects remain unproven. This study makes
+three contributions: (1) short-window encoder evidence (SWE) supplies recent
+audio and optional person-crop vision to a frozen-Omni forecasting head;
+(2) partial-tail refresh (PTR) queries the observed current-block prefix
+without committing it to persistent history; and (3) detailed EasyCom
+analyses accompany a specified Ego4D transfer protocol. Matched PTR,
+Ego4D scores, and event behavior remain pending.

 \begin{figure}[t]
 \centering
--- a/manuscript/targeted/references.bib
+++ b/manuscript/targeted/references.bib
@@ -7,6 +7,41 @@
  doi       = {10.21437/Interspeech.2022-10955}
 }

+@article{sohn1999vad,
+  author  = {Jongseo Sohn and Nam Soo Kim and Wonyong Sung},
+  title   = {A Statistical Model-Based Voice Activity Detection},
+  journal = {IEEE Signal Processing Letters},
+  volume  = {6},
+  number  = {1},
+  pages   = {1--3},
+  year    = {1999},
+  doi     = {10.1109/97.736233}
+}
+
+@inproceedings{li2026easyturn,
+  author    = {Guojian Li and Chengyou Wang and Hongfei Xue and Shuiyuan Wang and Dehui Gao and Zihan Zhang and Yuke Lin and Wenjie Li and Longshuai Xiao and Zhonghua Fu and Lei Xie},
+  title     = {{Easy Turn}: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systems},
+  booktitle = {Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
+  year      = {2026},
+  url       = {https://arxiv.org/abs/2509.23938}
+}
+
+@article{wang2026fastturn,
+  author  = {Chengyou Wang and Hongfei Xue and Mingchen Shao and Chunjiang He and Jingbin Hu and Shuiyuan Wang and Bo Wu and Yuyu Ji and Jimeng Zheng and Ruofei Chen and Zhou Zhu and Lei Xie},
+  title   = {{FastTurn}: Unifying Acoustic and Streaming Semantic Cues for Low-Latency and Robust Turn Detection},
+  journal = {arXiv preprint arXiv:2604.01897},
+  year    = {2026}
+}
+
+@inproceedings{wagner2024ddsd,
+  author    = {Dominik Wagner and Alexander Churchill and Siddharth Sigtia and Panayiotis Georgiou and Matt Mirsamadi and Aarshee Mishra and Erik Marchi},
+  title     = {A Multimodal Approach to Device-Directed Speech Detection with Large Language Models},
+  booktitle = {Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
+  pages     = {10451--10455},
+  year      = {2024},
+  doi       = {10.1109/ICASSP48485.2024.10446224}
+}
+
@inproceedings{kim2025egospeak,
  author    = {Junhyeok Kim and Min Soo Kim and Jiwan Chung and Jungbin Cho and Jisoo Kim and Sungwoong Kim and Gyeongbo Sim and Youngjae Yu},
  title     = {{EgoSpeak}: Learning When to Speak for Egocentric Conversational Agents in the Wild},