wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Polish targeted manuscript framing and attribution

Browse Source
This commit is contained in:
Xin Wang
2026-09-24 15:44:25 +08:00
parent e09ee1c432
commit 15f1b779b3
2 changed files with 97 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
manuscript/targeted/README.md
View File

@@ -15,6 +15,10 @@ interaction, then studies two cutoff-aligned interfaces:
- **partial-tail refresh (PTR)**: query only the observed prefix of the current
two-second block at each 200-ms cutoff, reusing completed payloads.
SWE names the short temporal support of encoder evidence, not a shallow
network layer. The two-second chunk refers to this Qwen extraction setup,
not a universal Omni-model setting.
The evidence remains a causal global--local **audit**, not a claim that all
Omni models lack local perception or that PTR's independent gain is proven:

175
manuscript/targeted/main.tex
View File

@@ -19,30 +19,31 @@
\renewcommand{\dblfloatpagefraction}{0.75}
\title{Egocentric Speech Anticipation with Omni Models:\\
Short-Window Evidence and Partial-Tail Refresh}
\name{\shortstack{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu,\\
\name{\shortstack{Xin Wang, Dingtian Yan, Changdong Liu, Shengxiang Hu, \\
Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}}
\address{Shanghai Ideal Information Industry (Group) Co., Ltd., China Telecom}
\address{China Telecom Corporation Limited Shanghai Branch}
\begin{document}
\ninept
\maketitle
\begin{abstract}
An egocentric conversational agent must anticipate when its invisible wearer
will speak while observing a causal multiparty audio--video stream. Although
turn-taking prediction and omni-modal assistants have advanced, the temporal
accessibility of local evidence in an integrated Omni state remains unclear.
We study two interfaces: a short-window encoder-evidence path that exposes
recent audio and optional person-crop vision to a frozen-Omni forecasting
head, and partial-tail refresh that queries the observed prefix of an
incomplete audiovisual block at each 200-ms cutoff. On EasyCom, the
exploratory global-plus-audio screen obtains 81.72\% 0.2-s mAP and 65.11\%
ten-horizon mean mAP, versus 74.20/61.19\% for released EgoSpeak. Pooled
vision adds only 0.29 mean-mAP points, whereas an analysis-only causal
visible-activity oracle reveals additional information. A separate chunk
screen suggests sensitivity to state freshness, but does not isolate the
partial-tail effect. These results motivate finer local evidence access;
they do not establish semantic floor control or cross-dataset transfer.
An egocentric conversational agent must anticipate when its unseen camera
wearer will speak from a causal multiparty audio--video stream. Turn-taking
models and streaming Omni assistants address related problems, but the
cutoff-level availability of local evidence in an integrated Omni state is
underexplored for this wearer-centered task. We examine two interfaces:
short-window encoder evidence (SWE) supplies recent audio and optional
person-crop vision directly to a frozen-Omni forecasting head; partial-tail
refresh (PTR) queries only the observed prefix of the current audiovisual
block at each 200-ms cutoff. On EasyCom, the exploratory global-plus-audio
screen reaches 81.72\% mAP at 0.2~s and 65.11\% over ten horizons, versus
74.20/61.19\% for released EgoSpeak. Pooled vision adds 0.29 mean-mAP
points, while an analysis-only visible-activity audit exposes further local
information. A separate chunk screen suggests sensitivity to state
freshness but does not isolate PTR. We specify an Ego4D transfer protocol;
its scores remain pending. The evidence supports causal activity
anticipation, not semantic floor control or established cross-dataset gains.
\end{abstract}
\begin{keywords}
@@ -53,16 +54,17 @@ streaming, voice activity projection, conversational agents
\section{Introduction}
\label{sec:intro}
An egocentric conversational agent must decide \emph{when} to speak while its
camera wearer is usually invisible. It sees other participants and hears a
mixed conversation, but must predict the wearer's own future speech before a
turn boundary is known (Fig.~\ref{fig:task}). Continuous turn-taking models
and voice-activity projection (VAP) show the value of anticipatory rather than
silence-triggered decisions \cite{ekstedt2022vap,russell2025mmvap,
elmers2025triadic,lin2025backchannel,lee2024eot,qi2026muvap,
jiang2026turnbench}. Yet these tasks differ in sensor placement, visible
speakers, dialogue roles, and whether the target is a future activity state
or a semantic intervention.
An egocentric conversational agent must decide \emph{when} to speak even
though its camera wearer is usually invisible. It hears a mixed conversation
and sees other participants, but must anticipate its own speech from the
observed stream before a turn boundary is known (Fig.~\ref{fig:task}).
Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic
extensions, backchannel prediction, online end-of-turn detection, and event
benchmarks address increasingly realistic timing decisions
\cite{ekstedt2022vap,russell2025mmvap,elmers2025triadic,
lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}.
Their input viewpoints and target decisions, however, are not interchangeable
with forecasting the future activity of an unseen wearer.
\begin{figure}[H]
\centering
@@ -75,40 +77,42 @@ not semantic floor ownership.}
\end{figure}
\textbf{Egocentric audiovisual conversation.}
EgoSpeak forecasts wearer speech from first-person RGB and audio using
specialized encoders and a temporal fusion model \cite{kim2025egospeak};
EasyCom and Ego4D supply complementary egocentric conversation settings
\cite{donley2021easycom,grauman2022ego4d}. Multi-view next-speaker
anticipation, egocentric speaking/listening graphs, and addressee detection
study related but different targets \cite{serj2024threem,jia2024avconv,
kong2024sicnet}. Egocentric active-speaker localization and detection
\cite{jiang2022asl,huh2025egoasd,clarke2025asd} and long--short person
modeling \cite{wang2024loconet} estimate who is speaking now; they do not by
themselves forecast an unseen wearer's future activity. Visual cues can help
turn prediction in noise \cite{oconnorrussell2025noise}, but whether pooled
vision conveys the relevant local floor evidence here is empirical.
EgoSpeak is the closest wearer-centered predecessor: it extracts audio and
RGB features with specialized encoders, fuses their temporal evidence, and
forecasts wearer speech on EasyCom and Ego4D \cite{kim2025egospeak,
donley2021easycom,grauman2022ego4d}. Other first-person work addresses
multi-view next-speaker prediction, speaking/listening graphs, or addressee
detection \cite{serj2024threem,jia2024avconv,kong2024sicnet}.
Egocentric active-speaker methods \cite{jiang2022asl,huh2025egoasd,
clarke2025asd} and long--short person modeling \cite{wang2024loconet}
recover present speaker evidence; audiovisual cues can also help turn
prediction in noise \cite{oconnorrussell2025noise}. These studies make a
broad ``first audiovisual turn-taking'' claim untenable. What remains
less examined is how a single causal first-person stream should expose
recent audio--visual evidence to forecast the invisible wearer's activity.
\textbf{Omni and local evidence.}
Omni systems integrate modalities and streaming interaction
\textbf{Streaming Omni and temporal access.}
Omni systems integrate modalities and support streaming interaction
\cite{xu2025qwenomni,cui2026minicpmo45,tian2026roma,yan2026duplug,
yu2025salmonnomni,chen2024videollmonline,wang2025omnimmi}. Encoder-side
audio--visual adaptation offers another route to accessible modality evidence
\cite{lin2023lavish,rouditchenko2024whisperflamingo}. However, neither
large-context integration nor a nominally streaming interface guarantees a
fresh representation at every 200-ms prediction cutoff. In our Qwen2.5-Omni
extraction, completed 2-s blocks and an incomplete current block have
different update rules. We therefore test two distinct interfaces:
\emph{short-window encoder evidence} (SWE) exposes recent modality features
outside the integrated Thinker state, and \emph{partial-tail refresh} (PTR)
queries only the observed current-block prefix. Neither term implies that
all Omni models use the same chunk duration or lack local perception.
\cite{lin2023lavish,rouditchenko2024whisperflamingo}. Qwen2.5-Omni uses
block-wise audio processing \cite{xu2025qwenomni}; in our extraction, a 2-s
block is long relative to a 200-ms decision interval. A streaming interface
alone therefore does not specify how fresh the integrated representation is
at each forecast cutoff. We study two complementary, cutoff-aligned
interventions: \emph{short-window encoder evidence} (SWE) directly exposes
recent modality features outside the Thinker state, while \emph{partial-tail
refresh} (PTR) queries the observed prefix of the incomplete current block.
SWE denotes temporal locality, not shallow encoder layers; the argument
concerns this pipeline, not an intrinsic deficit of every Omni model.
We contribute a causal implementation of SWE and PTR for wearer activity
anticipation; an EasyCom audit separating global, local-audio, pooled-visual,
and privileged information conditions; and a protocol for EasyCom and Ego4D
transfer with endpoint and hard-slice diagnostics. Matched tail and Ego4D
scores are still pending, so our measured claims remain confined to the
available EasyCom screens.
Our contributions are a causal SWE/PTR interface for 5-Hz wearer activity
anticipation; an EasyCom analysis separating global, recent-audio,
pooled-visual, and privileged visible-activity information; and a specified
EasyCom--Ego4D evaluation with horizon, endpoint, and hard-slice diagnostics.
Only EasyCom screening scores are currently available: matched PTR effects,
Ego4D transfer, and event-level behavior remain open tests.
\begin{figure}[t]
\centering
@@ -125,8 +129,9 @@ temporarily queries the observed current-block prefix while reusing completed
\subsection{Task and Short-Window Encoder Evidence}
At cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and
predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s. Each
At the 5-Hz cutoff $t_n=0.2n$~s, the model observes only media available by
$t_n$ and predicts activity at $t_n+h$, where
$h\in\{0.2,0.4,\ldots,2.0\}$~s. Each
horizon has three channels in fixed order: background, camera wearer, and
aggregate others. Wearer and others may overlap; no output identifies an
individual speaker, addressee, or semantic end of turn.
@@ -134,12 +139,13 @@ individual speaker, addressee, or semantic end of turn.
The global feature $\vect g_n$ concatenates the final four frozen
Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It
already contains audio and video context; $\G$ is not audio-free. SWE
adds encoder-side features without requiring the Thinker state to preserve
their fine timing. Local audio $\vect a_n$ crops the latest one-second mixed
provides a separate short-window path so the head need not recover all
cutoff-local detail from the integrated state. Local audio $\vect a_n$ crops
the latest one-second mixed
waveform \emph{before} preprocessing, runs the frozen Qwen audio encoder, and
averages valid tokens. Optional $\vect v_n$ averages frozen vision tokens
inside provided person boxes, then across people and valid causal frames in
the latest second. This pooling has no identity or mouth-motion trajectory;
inside provided person boxes, then across people and valid causal 2-FPS
frames in the latest second. This pooling has no identity or mouth-motion trajectory;
provided boxes do not imply an online detector.
Each enabled local feature is layer-normalized, projected to 256 dimensions,
@@ -160,14 +166,16 @@ is frozen-backbone adaptation rather than end-to-end training.
\subsection{Causal Partial-Tail Refresh}
PTR uses source-aligned 2-s blocks under an 8-s maximum presentation cap
(Fig.~\ref{fig:overview}). After warm-up, three completed blocks and the
observed current tail are available; span varies with block phase. Reusing
(Fig.~\ref{fig:overview}). After warm-up, three completed blocks and the
observed current tail are available; span varies with block phase. Reusing
completed modality payloads, we crop the tail at $t_n$ and query it in a
temporary cache. The tail and query are not committed. Only completed
temporary cache. The tail and query are not committed. Only completed
blocks enter persistent history; session and source discontinuities reset
caches. Thus each 5-Hz readout sees the observed prefix without future
backfill. PTR is a causal access rule, not by itself a proven accuracy or
real-time latency improvement.
caches. Thus each 5-Hz readout sees the observed prefix without future
backfill. To test whether access policy matters, the matched design crosses
completed-only versus PTR readouts with $\A$ on/off at an identical 8-s cap;
the separate 2--12-s cap sweep tests context length. Those comparisons are
pending, so PTR is a causal access rule rather than a demonstrated gain.
\subsection{Visible-Activity Information Audit}
@@ -333,7 +341,7 @@ Full path & -- & -- & -- \\
\label{sec:experiments}
\textbf{Data and protocol.}
We use EasyCom with the official EgoSpeak split: Sessions~4--12 for training
We use EasyCom with the EgoSpeak official split: Sessions~4--12 for training
and Sessions~1--3 for testing. Inputs and targets share a 5-Hz grid. Heads
are trained for four fixed epochs on all training sessions without test-time
checkpoint selection. The deployable feature matrix is an archived
@@ -341,7 +349,7 @@ single-seed screen; the information audit uses seeds 0, 1, and 2 with an
identical fixed recipe. We therefore report standard deviations only for the
latter and treat small differences in the former descriptively.
\textbf{Ego4D transfer protocol.}
\textbf{Ego4D transfer protocol (pending results).}
Ego4D audiovisual clips \cite{grauman2022ego4d} use the existing EgoSpeak
manifest and targets, with 346 training and 87 test clips. The same 5-Hz
cutoffs and local 1-s encoder windows define $\G$, $\G+\A$, $\G+\V$, and
@@ -368,8 +376,8 @@ already sufficient. Because that screen used Session~12 model selection
whereas the historical pooled row did not, their difference is diagnostic,
not a matched causal attribution.
\textbf{Six pre-specified control families.}
We freeze six pending control families: (L) capacity and recency, (T) tail
\textbf{Pre-specified analyses.}
We freeze six control families: (L) capacity and recency, (T) tail
refresh crossed with local audio, (C) six context caps, (B) paired backbone
transfer, (H/Q) error slices and timelines, and (S) full-path efficiency.
They share the official split and physical cutoffs; the 8-s T/C row reuses
@@ -381,7 +389,7 @@ one run. The source retains unmeasured hypotheses and falsifiers.
\subsection{EasyCom: Local Encoder Evidence}
Table~\ref{tab:main} shows that recent encoder-side audio $\A$ is strong at
0.2~s while integrated $\G$ is slightly stronger across all horizons.
0.2~s while integrated $\G$ is slightly stronger on the ten-horizon mean.
Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 over
$\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at
0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23
@@ -389,8 +397,10 @@ points, onset AP by only 1.47: dense gains do not imply equal improvement
at both activity boundaries.
The $\G$-only head has more trainable parameters and longer history than the
$\A$-only head. Thus this screen shows predictive complementarity, not a
capacity-matched proof that the Thinker lacks a specific local cue.
$\A$-only head. Thus this screen is consistent with predictive
complementarity, not a capacity-matched proof that the Thinker lacks a
specific local cue. Individual $\A$ and $\G$ horizon curves are unavailable
under the same causal-valid summary and are not reconstructed in Table~\ref{tab:main}.
% Retained as a planning aid; illustrative, unmeasured patterns do not belong
% in the page-limited submission preview.
@@ -541,13 +551,14 @@ next other offset. Behavioral claims await a real-timeline evaluator audit.
\section{Conclusion}
On EasyCom, a short-window audio-encoder path complements the frozen Omni
readout for future wearer activity, most clearly at speech offset. Pooled
vision recovers little of the timely visible-activity information exposed by
the analysis-only audit. Partial-tail refresh provides a causal 5-Hz
interface, but its independent predictive and efficiency effects await
matched tests. Ego4D transfer and censored event evaluation remain necessary
before claiming generalizable or improved semantic floor behavior.
For egocentric when-to-speak forecasting, cutoff-aligned access to recent
encoder evidence is a useful complement to frozen Omni context on EasyCom;
the strongest measured boundary difference is at wearer speech offset.
Pooled vision recovers little of the timely visible-activity information
revealed by the privileged audit. PTR supplies a causal 5-Hz readout rule,
but its independent predictive and efficiency effects still require matched
tests. Ego4D transfer and censored event evaluation remain necessary before
claiming generalizable or improved semantic floor behavior.
\section{Acknowledgment}
OpenAI Codex assisted with drafting and figure layout; the authors remain