wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add targeted ICASSP manuscript draft

Browse Source
This commit is contained in:
Xin Wang
2026-09-23 07:54:36 +08:00
parent dcbf60a360
commit ba162f44a0
9 changed files with 643 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

326
manuscript/targeted/main.tex Normal file
View File

@@ -0,0 +1,326 @@
% Acceptance-target ICASSP 2027 draft grounded only in archived measurements.
% The official 2027 author kit should replace the 2026-compatible style assets
% before submission. Four technical pages plus an optional reference-only page.
\documentclass{article}
\usepackage{spconf}
\usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor}
\hypersetup{hidelinks}
\newcommand{\vect}[1]{\boldsymbol{#1}}
\newcommand{\G}{\mathrm{G}}
\newcommand{\A}{\mathrm{A}}
\newcommand{\V}{\mathrm{V}}
\setlength{\emergencystretch}{2em}
\title{Does an Omni Model Know When the Wearer Will Speak?\\
A Causal Global--Local Audit of Egocentric Speech Anticipation}
\name{Authors withheld in this working draft}
\address{}
\begin{document}
\ninept
\maketitle
\begin{abstract}
An egocentric conversational agent must anticipate the future speech of an
invisible camera wearer from a causal multiparty audio--video stream. We ask
whether a frozen general-purpose Omni state exposes the sub-second evidence
needed by this task. Our cutoff-aligned model combines an 8-s Qwen2.5-Omni
context with the most recent 1-s audio and predicts three activity channels at
ten horizons from 0.2 to 2.0~s. In a single-seed EasyCom screen, this model reaches
81.72\% mAP at 0.2~s and 65.11\% averaged over horizons, compared with
74.20\% and 61.19\% for released EgoSpeak. The gain is larger for wearer
offset anticipation (+7.23 points) than onset anticipation (+1.47). Pooled
visual features add only 0.29 mean-mAP points. A separate three-seed,
strictly causal information audit explains this gap: privileged past activity
of visible participants adds 4.83 points at 0.2~s and 1.77 mean-mAP points,
whereas visibility, one-second-delayed activity, and identity shuffling show
that timeliness matters more than identity continuity. The results isolate an
evidence-access bottleneck: recent acoustic evidence is already useful, while
current generic visual pooling does not recover the available visible-floor
signal.
\end{abstract}
\begin{keywords}
egocentric speech anticipation, multimodal foundation models, causal
streaming, voice activity projection, conversational agents
\end{keywords}
\section{Introduction}
\label{sec:intro}
Conversational agents must decide not only \emph{what} to say, but also
\emph{when} to participate. This decision is particularly asymmetric in an
egocentric stream: the camera wearer is normally invisible, yet the model must
forecast that wearer's behavior from mixed audio and observations of other
people. A useful predictor must update before a turn boundary is known and
must not use media from after its decision cutoff.
Voice Activity Projection (VAP) casts turn taking as continuous prediction of
future activity \cite{ekstedt2022vap}. EgoSpeak brings this objective to
untrimmed first-person video and defines the EasyCom protocol used here
\cite{kim2025egospeak,donley2021easycom}. Other work predicts future speakers
from synchronized egocentric sensors \cite{serj2024threem}, reconstructs
egocentric conversational graphs \cite{jia2024avconv}, or grounds causal
multiparty projections in visible face tracks \cite{qi2026muvap}. Those
systems establish that neither multiparty audiovisual modeling nor
first-person speech initiation is itself a novelty. Our question is narrower:
does a general-purpose Omni readout make timely evidence accessible for
forecasting the unseen wearer?
Qwen2.5-Omni integrates long-form audio and video in a Thinker representation
\cite{xu2025qwenomni}. Such integration is attractive for conversational
context, but a blockwise state can blur or delay the short acoustic and visual
cues needed at 200-ms intervals. We therefore separate an integrated global
state $\G$ from explicit recent evidence $\A$ and optional $\V$, and treat
their value as an empirical question rather than assuming that a larger
backbone solves temporal access.
This paper makes three contributions. First, we formulate a strictly causal
global--local audit for invisible-wearer activity forecasting, with a common
5-Hz cutoff and ten future horizons. Second, we realize the audit with a
bounded Omni stream that combines completed history with a temporary observed
tail and a separate one-second audio path. Third, we distinguish deployable
feature gains from an analysis-only visible-activity oracle. This reveals a
specific negative result---generic pooled or current-frame person features do
not recover most of the available local-floor information---and identifies
timely activity estimation, rather than identity continuity, as the next
modeling target.
\begin{figure*}[t]
\centering
\includegraphics[width=0.97\textwidth]{figures/system_overview.pdf}
\caption{Global--local causal audit. At each 200-ms cutoff, the prediction
path fuses an integrated Omni state with explicit recent audio and optional
person-crop vision. Completed 2-s blocks are reused while only the observed
partial tail is temporarily recomputed. The purple branch is a privileged
past-activity audit used to test information value; it is never a deployable
input and contains no future activity.}
\label{fig:overview}
\end{figure*}
\section{Method}
\label{sec:method}
\subsection{Task and Causal Inputs}
At cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and
predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s. Each
horizon has three channels in fixed order: background, camera wearer, and
aggregate others. Wearer and others may overlap; no output identifies an
individual speaker, addressee, or semantic end of turn.
The global feature $\vect g_n$ is extracted from frozen Qwen2.5-Omni-3B. We
concatenate the final four Thinker readout states and project them to 2048
dimensions. The readout contains both audio and video context; $\G$ is not an
audio-free condition. The local audio feature $\vect a_n$ is obtained by
cropping the most recent one-second mixed waveform \emph{before}
preprocessing, encoding it with the frozen Qwen audio encoder, and averaging
valid output tokens. An optional visual feature $\vect v_n$ averages spatial
tokens inside provided person boxes, people within each causal frame, and
valid frames from the most recent second. It preserves neither identity nor
mouth-motion trajectories, and provided boxes do not constitute an online
detector.
Each enabled local feature is layer-normalized, projected to 256 dimensions,
and paired with a validity bit. The trainable head is
\begin{equation}
\widehat{\vect Y}_{n}=
\operatorname{softmax}_{c}F_{\theta}
\bigl[\vect g_n;\widetilde{\vect a}_n;m_n^a;
\widetilde{\vect v}_n;m_n^v\bigr],
\label{eq:fusion}
\end{equation}
where omitted modalities remove their terms and $F_\theta$ is a
layer-normalized MLP with a 30-logit output. We train only the projections and
head with the EgoSpeak multi-positive cross-entropy over causal-valid
source--horizon pairs. The Omni and modality encoders remain frozen, so this
is frozen-backbone adaptation rather than end-to-end training.
\subsection{Cutoff-Aligned Streaming}
The global stream uses source-aligned 2-s blocks and an 8-s maximum
presentation budget. After warm-up, three completed blocks and the observed
part of the current block are available; the exact span varies with block
phase. Completed modality payloads are reused when the bounded language-model
cache is rebuilt. At every cutoff, the incomplete current block is cropped at
$t_n$, appended to a temporary cache, and queried for $\vect g_n$. Neither
the temporary tail nor the query is committed. At a block boundary the
completed payload is committed once and a new tail begins. Sessions and
source discontinuities reset all caches. Thus the logical state is refreshed
at 5~Hz without backfilling a completed block into earlier rows.
\subsection{Visible-Activity Information Audit}
To distinguish missing information from ineffective learned vision, we build
analysis-only features from transcript-derived \emph{past} activity. At each
cutoff, visible non-wearer participants receive two seconds of binary history
sampled on the same 5-Hz grid. The preparation fails if its aggregate
activity disagrees with the released target. We compare: (i) visibility with
activity zeroed, (ii) per-person histories, (iii) identities shuffled while
preserving active-person counts at every lag, (iv) histories delayed by one
second, and (v) one OR-pooled aggregate-visible history. These conditions
never expose future activity, but their speech labels make them privileged
diagnostics, not deployable features or upper bounds.
\section{Experiments}
\label{sec:experiments}
\textbf{Data and protocol.}
We use EasyCom with the official EgoSpeak split: Sessions~4--12 for training
and Sessions~1--3 for testing. Inputs and targets share a 5-Hz grid. Heads
are trained for four fixed epochs on all training sessions without test-time
checkpoint selection. The deployable feature matrix is an archived
single-seed screen; the information audit uses seeds 0, 1, and 2 with an
identical fixed recipe. We therefore report standard deviations only for the
latter and treat small differences in the former descriptively.
\textbf{Metrics.}
At each horizon we compute AP for the three activity channels over common
causal-valid rows, then average classes to obtain mAP and average all ten
horizons to obtain mean mAP. We also report wearer AP at 0.2~s. For endpoint
diagnostics, onset AP conditions source rows on the wearer being silent and
scores future speech with $p(\mathrm{wearer})$; offset AP conditions on wearer
speech and scores future silence with $1-p(\mathrm{wearer})$. These
threshold-free quantities average ten horizons. They are not one-event-one-
vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding.
\textbf{Comparisons.}
We evaluate released EgoSpeak MiniROADA A+V; recent audio $\A$; global Omni
$\G$; $\G+\A$; and $\G+\A+$pooled-$\V$. A three-seed current-frame
frame-set attention screen tests whether a richer person aggregation is
already sufficient. Because that screen used Session~12 model selection
whereas the historical pooled row did not, their difference is diagnostic,
not a matched causal attribution.
\section{Results}
\label{sec:results}
\begin{table*}[t]
\caption{EasyCom Sessions~1--3 results (\%). $\mathrm{mAP}_{.2}$ and
$\mathrm{WAP}_{.2}$ are dense 0.2-s metrics; Mean averages dense mAP over ten
horizons. On/Off are raw wearer endpoint AP, not semantic event scores.
Deployable rows are archived single-seed screens; $\pm$ is shown only for the
three-seed F1 screen. Dashes denote unavailable measurements.}
\label{tab:main}
\centering
\scriptsize
\setlength{\tabcolsep}{5.0pt}
\begin{tabular}{@{}lccccc@{}}
\toprule
Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
& Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
\midrule
EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
Omni $\G+\A+$pooled-$\V$ & \textbf{82.027} & \textbf{85.086}
& \textbf{65.400} & \textbf{26.019} & 45.371 \\\\
Current-frame person set F1 & $81.777{\pm}0.156$ & --
& $64.951{\pm}0.095$ & -- & -- \\\\
\bottomrule
\end{tabular}
\end{table*}
\subsection{Global Context and Recent Audio Are Complementary}
Table~\ref{tab:main} shows that $\A$ is strong at 0.2~s, while $\G$ is
slightly stronger when mAP is averaged to 2.0~s. Their fusion is better than
$\A$ at every horizon: the mean gain is 4.60 points, with gains of 2.03 at
0.2~s and 4.47 at 2.0~s. It also improves every class when averaged over
horizons (+8.72 background, +2.43 wearer, and +2.65 others). Relative to
released EgoSpeak, $\G+\A$ improves 0.2-s mAP by 7.52 points and mean mAP by
3.92 points. The endpoint effect is asymmetric: offset AP rises by 7.23
points, while onset AP rises by 1.47. Dense forecasting gains therefore do
not imply equal progress at both behavior boundaries.
The global-only head has more trainable parameters and a longer context than
the local-only head, so this comparison demonstrates predictive
complementarity, not that semantic reasoning alone causes the gain. A wider
history-matched control is required for a representation-level causal claim.
\subsection{Generic Vision Does Not Recover the Available Signal}
Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
+0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when
averaged over horizons; onset/offset changes are +0.059/+0.120. A richer
current-frame person-set model is stable across seeds but reaches lower mean
mAP than the historical pooled row under a non-matched selection protocol.
These results do not support a claim of learned speaker grounding.
\begin{table}[t]
\caption{Three-seed privileged-information audit (mean $\pm$ sample standard
deviation, \%). Every row includes $\G+\A$; activity conditions use only
past labels. Bold marks the best diagnostic, not a deployable model.}
\label{tab:oracle}
\centering
\scriptsize
\setlength{\tabcolsep}{3.2pt}
\begin{tabular}{@{}lccc@{}}
\toprule
Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
\midrule
None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
& $\mathbf{66.951{\pm}.022}$ \\\\
\bottomrule
\end{tabular}
\end{table}
Table~\ref{tab:oracle} locates the missing information. Aggregate visible
activity improves over the audio baseline by 4.826 points at 0.2~s and 1.775
mean-mAP points. Visibility alone changes mean mAP by only +0.040, and
one-second-delayed activity by +0.164. Thus availability or extra parameters
cannot explain the oracle gain. Per-person activity exceeds shuffled identity
by only 0.200 mean-mAP points and is itself 0.291 below the aggregate
condition. The current target therefore supports timely visible-floor
activity as useful information, but not the necessity of identity continuity.
The majority of the oracle's 0.2-s macro gain comes from background and
others; wearer AP rises by only 0.572. This is why neither the oracle nor a
dense mAP gain establishes fewer false wearer entries.
\subsection{What the Audit Establishes}
Three conclusions survive the controls. First, a general-purpose global
state and a one-second acoustic readout provide different useful timescales.
Second, generic visual pooling has not converted visible people into reliable
local floor evidence. Third, perfect timely aggregate activity contains
recoverable information, whereas identity contributes little under the
current target. A dedicated causal active-speaker front end---designed for
mouth motion, blur, and short audio--visual synchrony as in egocentric ASD
\cite{huh2025egoasd}---is consequently a better-motivated next step than a
larger identity-aware fusion module.
The evidence remains bounded. EasyCom has only three official test sessions;
the deployable matrix is single seed and has been examined repeatedly. The
oracle uses labels and is not a performance upper bound. Provided boxes omit
online detection cost, and logical 5-Hz output is not a measured throughput
claim. Finally, aggregate others cannot supply addressee or semantic
interruption ground truth. Confirmatory behavioral claims require a frozen
training-side operating point, competing-risk event attribution, and
multi-seed matched controls.
\section{Conclusion}
We presented a strictly causal global--local audit of egocentric wearer speech
anticipation. Frozen Omni context and recent audio are strongly complementary,
especially for future wearer cessation. In contrast, pooled or current-frame
person features do not recover the local-floor signal revealed by a causal
visible-activity oracle. The actionable conclusion is deliberately narrow:
the next bottleneck is timely, learnable local activity evidence, not simply
more multimodal capacity or persistent identity. This distinction provides a
clearer target for future egocentric conversational agents and prevents dense
forecasting gains from being mistaken for semantic floor control.
\section{Acknowledgment}
OpenAI Codex assisted with substantive drafting and figure layout in the
abstract, introduction, and presentation of the archived analysis; the
authors remain responsible for verification, experimental claims, and the
final manuscript.
\bibliographystyle{IEEEbib}
\bibliography{references}
\end{document}