Freeze targeted introduction storyline

manuscript/targeted/main.tex

@@ -54,82 +54,80 @@ streaming, voice activity projection, conversational agents
 \section{Introduction}
 \label{sec:intro}
 
-An egocentric conversational agent must decide \emph{when} to speak even
-though its camera wearer is usually invisible.  It hears a mixed conversation
-and sees other participants, but must anticipate its own speech from the
-observed stream before a turn boundary is known (Fig.~\ref{fig:overview}).
-This decision differs from recognizing who is speaking now: the relevant
-evidence may be a brief pause, overlap, or response by someone in view, while
-the target is the wearer's \emph{future} activity.  It also differs from
-predicting a semantic end of turn or selecting an addressee.  We therefore
-study a deliberately narrower and measurable question: at each 200-ms
-cutoff, how well can a causal observer forecast background, wearer, and
-aggregate-other activity over the next 0.2--2.0~s?
-Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic
-extensions, backchannel prediction, online end-of-turn detection, and event
-benchmarks address increasingly realistic timing decisions
-\cite{ekstedt2022vap,russell2025mmvap,elmers2025triadic,
-lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}.
-Their input viewpoints and target decisions, however, are not interchangeable
-with forecasting the future activity of an unseen wearer.
-The distinction matters when comparing systems: a correct current-speaker
-label or an utterance-final response can conceal errors in the first few
-hundred milliseconds, precisely where a wearable agent must decide whether
-to wait or begin speaking.
+An egocentric conversational agent could answer situated questions, supply
+context, and assist its wearer during face-to-face conversation. To
+participate naturally, it must decide not only \emph{what} to say but when to
+begin, wait, or stop speaking. This requires interpreting mixed audio,
+visible people's behavior, and linguistic content as the number and roles of
+interlocutors change. A pause is not necessarily an invitation: two people
+visible to the wearer may be talking to each other, and their turn exchange
+should not trigger the agent's response. Since the wearer is normally
+off-camera, the relevant target is not simply which visible face speaks now,
+but whether the wearer is about to speak or cease speaking.
+Both boundaries matter: the agent should withhold an entry during a
+third-party exchange and yield when another participant takes the floor.
+From a moving first-person view, face visibility and overlapping voices can
+change independently; neither a silence threshold nor one detected face
+provides a fixed map of who is interacting with the wearer. These timing
+errors arise before response content is considered.
 
-\textbf{Egocentric audiovisual conversation.}
-EgoSpeak is the closest wearer-centered predecessor: it extracts audio and
-RGB features with specialized encoders, fuses their temporal evidence, and
-forecasts wearer speech on EasyCom and Ego4D \cite{kim2025egospeak,
-donley2021easycom,grauman2022ego4d}.  Other first-person work addresses
-multi-view next-speaker prediction, speaking/listening graphs, or addressee
-detection \cite{serj2024threem,jia2024avconv,kong2024sicnet}.
-Egocentric active-speaker methods \cite{jiang2022asl,huh2025egoasd,
-clarke2025asd} and long--short person modeling \cite{wang2024loconet}
-recover present speaker evidence; audiovisual cues can also help turn
-prediction in noise \cite{oconnorrussell2025noise}.  These studies make a
-broad ``first audiovisual turn-taking'' claim untenable.  What remains
-less examined is how a single causal first-person stream should expose
-recent audio--visual evidence to forecast the invisible wearer's activity.
-EgoSpeak establishes that compact audio and video representations can be
-fused for this task, but it does not answer whether a large integrated Omni
-representation makes those cutoff-local cues readily available to a small
-forecasting head.  Conversely, work on active speakers or addressees need
-not solve wearer anticipation: the visible interlocutor and the predicted
-speaker occupy different roles.  This leaves an architectural question,
-rather than a claim that audiovisual evidence itself is new.
+Prior work addresses complementary parts of this decision. Voice activity
+detection (VAD) detects speech presence \cite{sohn1999vad}; dyadic voice
+activity projection (VAP) predicts future speaker activity
+\cite{ekstedt2022vap}, while Easy Turn and FastTurn combine acoustic and
+linguistic evidence for spoken-agent turn detection
+\cite{li2026easyturn,wang2026fastturn}. Audiovisual turn and backchannel
+prediction exploits visible cues but is commonly evaluated with two
+interlocutors \cite{russell2025mmvap,lin2025backchannel,
+oconnorrussell2025noise}. Triadic VAP extends audio forecasting to fixed
+three-person conversations
+\cite{elmers2025triadic}; online three-party EOT has also used controlled
+multimodal sensor data \cite{lee2024eot}. MuVAP handles a variable number of
+speakers using single-camera video and face tracks, but studies visible
+speakers from a
+static third-party view \cite{qi2026muvap}. Device-directed speech detection
+can combine acoustics, ASR output, and text to ask whether an utterance
+addresses an assistant, yet the cited model does not observe visual
+relations among interlocutors \cite{wagner2024ddsd}. EgoSpeak directly
+forecasts the unseen wearer's future speech from causal
+first-person audio--video on EasyCom and Ego4D \cite{kim2025egospeak,
+donley2021easycom,grauman2022ego4d}. Multi-view next-speaker prediction,
+first-person speaking/listening graphs, and addressee detection provide
+related but distinct targets \cite{serj2024threem,jia2024avconv,
+kong2024sicnet}. Egocentric active-speaker localization likewise concerns
+current speaker evidence \cite{jiang2022asl,huh2025egoasd,clarke2025asd}.
+Together, these studies leave open how an integrated streaming model should
+retain cutoff-local evidence for forecasting the unseen wearer's activity in
+a natural multiparty, first-person stream. This is a representation-and-timing
+question, not a claim to first egocentric turn prediction. Our activity
+labels do not establish addressee or semantic floor ownership.
 
-\textbf{Streaming Omni and temporal access.}
-Omni systems integrate modalities and support streaming interaction
-\cite{xu2025qwenomni,cui2026minicpmo45,tian2026roma,yan2026duplug,
-yu2025salmonnomni,chen2024videollmonline,wang2025omnimmi}.  Encoder-side
-audio--visual adaptation offers another route to accessible modality evidence
-\cite{lin2023lavish,rouditchenko2024whisperflamingo}.  Qwen2.5-Omni uses
-block-wise audio processing \cite{xu2025qwenomni}; in our extraction, a 2-s
-block is long relative to a 200-ms decision interval.  A streaming interface
-alone therefore does not specify how fresh the integrated representation is
-at each forecast cutoff.  We study two complementary, cutoff-aligned
-interventions: \emph{short-window encoder evidence} (SWE) directly exposes
-recent modality features outside the Thinker state, while \emph{partial-tail
-refresh} (PTR) queries the observed prefix of the incomplete current block.
-SWE denotes temporal locality, not shallow encoder layers; the argument
-concerns this pipeline, not an intrinsic deficit of every Omni model.
-These interventions target different bottlenecks.  SWE asks whether recent
-encoder evidence improves prediction when the global state is held fixed;
-PTR asks whether the same integrated backbone benefits from a fresher readout
-without admitting future media.  A gain from one does not establish a gain
-from the other.  In particular, the effect of PTR requires a matched
-completed-only comparison at the same context cap, rather than comparison
-with a longer or shorter history.
+Omni models couple modality encoders to a shared language-model state, so
+audio, video, and text can inform streaming interaction and longer-range
+semantic context \cite{xu2025qwenomni,tian2026roma,chen2024videollmonline}.
+Qwen2.5-Omni has released 3B and 7B variants. Its block-wise encoders and
+time-aligned audio--video interleaving feed a Thinker for integrated
+understanding; a separate Talker generates speech \cite{xu2025qwenomni}.
+We use frozen 3B Thinker readouts; 7B is a planned transfer control, not a
+measured result. The native audio attention block spans 2~s, whereas our
+wearer forecast updates every 200~ms. Holding only completed blocks leaves
+stale context between boundaries; forcing 200-ms commits changes the native
+block construction and may alter its representation. Neither implies that
+all Omni models lack local evidence. An integrated, multi-layer state can
+carry broader audiovisual and linguistic context, yet transient cues may be
+less accessible to a small head than encoder-side features themselves
+\cite{lin2023lavish,rouditchenko2024whisperflamingo}.
 
-This framing yields three concrete contributions.  First, we formulate a
-causal, 5-Hz wearer-activity forecast and specify SWE and PTR as separable
-ways to expose local evidence to a frozen Omni model.  Second, on EasyCom we
-compare global, recent-audio, and pooled-visual features, then use privileged
-past visible activity only as an information audit, not a deployable input.
-Third, we specify cross-dataset, endpoint, and hard-slice tests.  EasyCom
-screening is the only measured evidence; matched PTR, Ego4D transfer, and
-event behavior remain open.
+We therefore retain the global Thinker state while exposing recent modality
+evidence at each causal cutoff (Fig.~\ref{fig:overview}). EasyCom screens
+suggest that global context and local audio are complementary; gains from
+pooled vision and isolated PTR effects remain unproven. This study makes
+three contributions: (1) short-window encoder evidence (SWE) supplies recent
+audio and optional person-crop vision to a frozen-Omni forecasting head;
+(2) partial-tail refresh (PTR) queries the observed current-block prefix
+without committing it to persistent history; and (3) detailed EasyCom
+analyses accompany a specified Ego4D transfer protocol. Matched PTR,
+Ego4D scores, and event behavior remain pending.
 
 \begin{figure}[t]
 \centering

manuscript/targeted/references.bib

@@ -7,6 +7,41 @@
   doi       = {10.21437/Interspeech.2022-10955}
 }
 
+@article{sohn1999vad,
+  author  = {Jongseo Sohn and Nam Soo Kim and Wonyong Sung},
+  title   = {A Statistical Model-Based Voice Activity Detection},
+  journal = {IEEE Signal Processing Letters},
+  volume  = {6},
+  number  = {1},
+  pages   = {1--3},
+  year    = {1999},
+  doi     = {10.1109/97.736233}
+}
+
+@inproceedings{li2026easyturn,
+  author    = {Guojian Li and Chengyou Wang and Hongfei Xue and Shuiyuan Wang and Dehui Gao and Zihan Zhang and Yuke Lin and Wenjie Li and Longshuai Xiao and Zhonghua Fu and Lei Xie},
+  title     = {{Easy Turn}: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systems},
+  booktitle = {Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
+  year      = {2026},
+  url       = {https://arxiv.org/abs/2509.23938}
+}
+
+@article{wang2026fastturn,
+  author  = {Chengyou Wang and Hongfei Xue and Mingchen Shao and Chunjiang He and Jingbin Hu and Shuiyuan Wang and Bo Wu and Yuyu Ji and Jimeng Zheng and Ruofei Chen and Zhou Zhu and Lei Xie},
+  title   = {{FastTurn}: Unifying Acoustic and Streaming Semantic Cues for Low-Latency and Robust Turn Detection},
+  journal = {arXiv preprint arXiv:2604.01897},
+  year    = {2026}
+}
+
+@inproceedings{wagner2024ddsd,
+  author    = {Dominik Wagner and Alexander Churchill and Siddharth Sigtia and Panayiotis Georgiou and Matt Mirsamadi and Aarshee Mishra and Erik Marchi},
+  title     = {A Multimodal Approach to Device-Directed Speech Detection with Large Language Models},
+  booktitle = {Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
+  pages     = {10451--10455},
+  year      = {2024},
+  doi       = {10.1109/ICASSP48485.2024.10446224}
+}
+
 @inproceedings{kim2025egospeak,
   author    = {Junhyeok Kim and Min Soo Kim and Jiwan Chung and Jungbin Cho and Jisoo Kim and Sungwoong Kim and Gyeongbo Sim and Youngjae Yu},
   title     = {{EgoSpeak}: Learning When to Speak for Egocentric Conversational Agents in the Wild},