wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update targeted ICASSP 2027 manuscript and results

Browse Source
This commit is contained in:
Xin Wang
2026-09-24 11:07:48 +08:00
parent 6de4873b45
commit 36b63bc283
11 changed files with 536 additions and 227 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
manuscript/targeted/Makefile
View File

@@ -2,15 +2,16 @@ LATEXMK ?= latexmk
.PHONY: all clean figure .PHONY: all clean figure
all: figures/system_overview.pdf all: figures/system_overview_onecolumn.pdf figures/task_overview.pdf
mkdir -p build mkdir -p build
$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=build main.tex $(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=build main.tex
figure: figures/system_overview.pdf figure: figures/system_overview_onecolumn.pdf figures/task_overview.pdf
figures/system_overview.pdf: figures/system_overview.tex figures/%.pdf: figures/%.tex
$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=figures $< $(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=figures $<
clean: clean:
$(LATEXMK) -C -outdir=build main.tex $(LATEXMK) -C -outdir=build main.tex
$(LATEXMK) -c -outdir=figures figures/system_overview.tex $(LATEXMK) -c -outdir=figures figures/system_overview_onecolumn.tex
$(LATEXMK) -c -outdir=figures figures/task_overview.tex

37
manuscript/targeted/README.md
View File

@@ -14,8 +14,21 @@ that is the strongest complete story supported by archived evidence:
- a three-seed causal oracle shows that timely visible-participant activity is - a three-seed causal oracle shows that timely visible-participant activity is
informative, while identity continuity is not currently necessary. informative, while identity continuity is not currently necessary.
The draft also reflects two seed-0 exploratory screens reported in
`../main/main.tex`: no-box full-frame F3 (Table 1) and a global-only
chunk-construction comparison (Table 2, panel B). Neither supplies
matched visual-motion evidence, endpoint gains, or the pending T/C factorial.
The result directories cited by the main draft for F3 and its G+A recheck
are absent from this checkout, so their underlying logs have not been
independently audited here.
No semantic ENTRY/YIELD, addressee, online-detector, or real-time-throughput No semantic ENTRY/YIELD, addressee, online-detector, or real-time-throughput
claim is made. The deployable matrix is explicitly single-seed and exploratory. claim is made. The deployable matrix is explicitly single-seed and exploratory.
The six-group L/T/C/B/H-Q/S table remains in `main.tex` as an
**illustrative, unmeasured planning aid**, excluded from the page-limited
PDF. Its directional statements are hypotheses and falsifiers, not EasyCom
scores. Replace or remove every row after the corresponding experiment,
including results that contradict the working story.
## Build ## Build
@@ -24,13 +37,25 @@ cd manuscript/targeted
make make
``` ```
The editable sources are `main.tex`, `references.bib`, and The editable sources are `main.tex`, `references.bib`,
`figures/system_overview.tex`. Generated files are written to `build/`. `figures/task_overview.tex`, and `figures/system_overview_onecolumn.tex`.
Generated LaTeX files are written to `build/`.
The build currently uses the repository's unmodified ICASSP 2026 style assets The preview is four technical pages plus a reference-only fifth page. Both
as a compatibility preview. Replace the `template_dir` in `latexmkrc` with the the task diagram (Fig. 1) and the system diagram (Fig. 2) occupy one column.
official ICASSP 2027 author kit before submission and repeat all page/layout The current PDF reserves five numbered result tables: measured deployable
checks. and oracle screens share Table 1; L controls extend its first panel. The
unmeasured T/C matrix and a separate measured chunk screen share Table 2;
B, H, and S occupy Tables 3--5. Pending cells are explicitly unmeasured.
Figure 3 is an H/Q case slot, not a selected or measured example. Its
cutoff-marked timelines must be inserted only after the slice protocol and
case-selection rule are frozen. The single-column system figure is generated
from figures/system_overview_onecolumn.tex; the earlier full-width source
is retained for comparison.
The build uses the repository's ICASSP 2027 LaTeX assets in
`../ICASSP2027_Paper_Templates/`, selected by `latexmkrc`. The supplied
`spconf.sty` and `IEEEbib.bst` match the 2026 copies.
## Evidence sources ## Evidence sources

19
manuscript/targeted/REVIEWER_REPORT.md
View File

@@ -78,10 +78,23 @@ the science.
causal active-speaker/floor estimator and require correct-time evidence to causal active-speaker/floor estimator and require correct-time evidence to
beat delayed, removed, and capacity-matched controls. Otherwise retain the beat delayed, removed, and capacity-matched controls. Otherwise retain the
honest audio-local paper and remove visual-method language. honest audio-local paper and remove visual-method language.
5. Rebuild with the official 2027 kit, keep technical content within four 5. Recheck the four-page technical limit after inserting results. The current
pages, put only references on an optional fifth page, and retain the IEEE build uses the 2027 kit and puts only references on page five; retain the
AI-assisted drafting disclosure. IEEE AI-assisted drafting disclosure.
Passing gates 1--3 would support a solid ICASSP empirical paper. Passing gate Passing gates 1--3 would support a solid ICASSP empirical paper. Passing gate
4 with a meaningful matched-recall behavior gain would support the stronger 4 with a meaningful matched-recall behavior gain would support the stronger
method paper originally envisioned. method paper originally envisioned.
## Review 4: expanded five-page working preview
The layout now uses four pages for technical content and a fifth page for
references only. The Introduction includes a causal task timeline; the
method figure separates deployable inputs from privileged audit labels. The
results presentation has measured main/oracle tables and a clearly labeled
six-group **illustrative scenario** with explicit falsifiers. This improves
readability and makes the intended experimental logic reviewable, but it does
not raise the scientific verdict above borderline: L/T/C/B/H-Q/S values and
full-pipeline efficiency remain unmeasured, and the scenario cannot be cited
as evidence. Re-review after gates 1--4 and replacement of the scenario table
with actual per-seed/per-session results.

BIN
manuscript/targeted/figures/system_overview.pdf
View File

Binary file not shown.

133
manuscript/targeted/figures/system_overview.tex
View File

@@ -1,92 +1,71 @@
\documentclass[tikz,border=2pt]{standalone} \documentclass[tikz,border=2pt]{standalone}
\usepackage{amsmath} \usepackage{amsmath}
\usetikzlibrary{arrows.meta,calc,positioning} \usetikzlibrary{arrows.meta,positioning,calc}
\definecolor{navy}{HTML}{19324A} \definecolor{navy}{HTML}{19324A}
\definecolor{bluefill}{HTML}{EAF1F7} \definecolor{bluefill}{HTML}{EAF1F7}
\definecolor{teal}{HTML}{0B7A75} \definecolor{teal}{HTML}{087F78}
\definecolor{tealfill}{HTML}{E5F4F2} \definecolor{tealfill}{HTML}{E6F4F1}
\definecolor{orange}{HTML}{C96A12} \definecolor{orange}{HTML}{B85D0F}
\definecolor{orangefill}{HTML}{FFF0DF} \definecolor{orangefill}{HTML}{FFF0DF}
\definecolor{purple}{HTML}{6E4B8B} \definecolor{grayline}{HTML}{687782}
\definecolor{purplefill}{HTML}{F1EBF6}
\definecolor{grayline}{HTML}{697782}
\begin{document} \begin{document}
\begin{tikzpicture}[ \begin{tikzpicture}[
font=\sffamily\scriptsize, >=Latex, font=\sffamily\scriptsize,>=Latex,
flow/.style={-{Latex[length=1.7mm]},line width=0.65pt,draw=navy}, flow/.style={-{Latex[length=1.6mm]},line width=.65pt,draw=navy},
localflow/.style={flow,draw=teal}, localflow/.style={flow,draw=teal},
auditflow/.style={flow,draw=purple,dashed}, block/.style={draw=navy,fill=bluefill,rounded corners=1.6pt,
box/.style={draw=navy,rounded corners=1.5pt,line width=0.65pt, line width=.6pt,minimum height=8.5mm,align=center,inner sep=2pt},
fill=bluefill,minimum height=8mm,align=center,inner sep=2.2pt}, local/.style={block,draw=teal,fill=tealfill},
localbox/.style={box,draw=teal,fill=tealfill}, hint/.style={font=\sffamily\tiny,text=grayline,align=center}]
auditbox/.style={box,draw=purple,fill=purplefill,dashed},
note/.style={font=\sffamily\tiny,text=grayline,align=center}
]
\node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy] \node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy]
at (-0.1,3.2) {Prediction path: integrated history plus explicit recent evidence}; at (0,3.12) {Deployable prediction path at the same causal cutoff $t_n$};
\node[align=center,text=navy] (av) at (0.7,2.45) \node[block,minimum width=25mm] (history) at (1.35,2.26)
{observed full-scene\\audio--video prefix}; {observed AV history\\8-s maximum budget};
\node[box] (omni) at (3.25,2.45) {frozen Qwen2.5-Omni\\8-s maximum context}; \node[block,minimum width=25mm] (omni) at (4.15,2.26)
\node[box] (global) at (5.95,2.45) {last-four-layer\\global state $G_n$}; {frozen Omni Thinker\\last-four-layer state};
\node[align=center,text=teal] (audio) at (0.7,1.23) \node[local,minimum width=25mm] (audio) at (1.35,1.13)
{most recent 1-s\\mixed audio}; {last 1-s mixed audio\\cropped at $t_n$};
\node[localbox] (ae) at (3.25,1.23) {frozen audio encoder\\valid-token mean}; \node[local,minimum width=25mm] (ae) at (4.15,1.13)
\node[localbox] (ap) at (5.95,1.23) {local projection\\$\widetilde A_n$ + validity}; {frozen audio encoder\\valid-token mean};
\node[align=center,text=teal] (video) at (0.7,0.02) \node[local,dashed,minimum width=25mm] (video) at (1.35,0)
{recent causal\\person crops}; {optional recent frames\\provided person boxes};
\node[localbox,dashed] (ve) at (3.25,0.02) \node[local,dashed,minimum width=25mm] (ve) at (4.15,0)
{frozen visual encoder\\crop / person pooling}; {frozen visual encoder\\pooled person tokens};
\node[localbox,dashed] (vp) at (5.95,0.02) \node[block,minimum width=20mm] (head) at (7.15,1.13)
{optional projection\\$\widetilde V_n$ + validity}; {normalized features\\+ validity $\to$ MLP};
\node[box,minimum width=21mm] (fusion) at (8.65,1.23) \node[block,minimum width=26mm] (out) at (10.4,1.13)
{cutoff-aligned\\fusion head}; {$10\times3$ forecast\\0.2--2.0-s horizons};
\node[box,minimum width=24mm] (out) at (11.55,1.23) \draw[flow] (history) -- (omni);
{$10\!\times\!3$ activity forecast\\0.2--2.0 s};
\draw[flow] (av) -- (omni);
\draw[flow] (omni) -- (global);
\draw[localflow] (audio) -- (ae); \draw[localflow] (audio) -- (ae);
\draw[localflow] (ae) -- (ap);
\draw[localflow,dashed] (video) -- (ve); \draw[localflow,dashed] (video) -- (ve);
\draw[localflow,dashed] (ve) -- (vp); \draw[flow] (omni.east) -- ++(.38,0) |- (head.north);
\draw[flow] (global.east) -- ++(0.30,0) |- (fusion.north); \draw[localflow] (ae) -- (head);
\draw[localflow] (ap) -- (fusion); \draw[localflow,dashed] (ve.east) -- ++(.38,0) |- (head.south);
\draw[localflow,dashed] (vp.east) -- ++(0.30,0) |- (fusion.south); \draw[flow] (head) -- (out);
\draw[flow] (fusion) -- (out); \node[hint,text=navy] at (10.4,.17)
\node[note] at (11.45,0.18) {background / wearer / aggregate others};
{background / invisible wearer / aggregate others}; \draw[draw=grayline,line width=.45pt] (0,-.62)--(11.85,-.62);
\draw[grayline,line width=0.45pt] (-0.1,-0.62) -- (12.9,-0.62);
\node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy] \node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy]
at (-0.1,-0.94) {Strictly causal stream at cutoff $t_n$}; at (0,-.94) {Bounded streaming state: no media after $t_n$};
\node[anchor=east,text=navy] at (2.05,-1.62) {Omni media}; \node[anchor=east,text=navy] at (1.9,-1.62) {Omni blocks};
\foreach \a/\b in {2.3/3.85,3.85/5.4,5.4/6.95} { \foreach \a/\b in {2.12/3.55,3.55/4.98,4.98/6.41} {
\draw[fill=bluefill,draw=navy,line width=0.55pt] \draw[draw=navy,fill=bluefill,line width=.55pt]
(\a,-1.84) rectangle (\b,-1.40); (\a,-1.84) rectangle (\b,-1.39);
\node[text=navy,font=\sffamily\tiny] at ({(\a+\b)/2},-1.62) {completed 2 s}; \node[font=\sffamily\tiny,text=navy] at ({(\a+\b)/2},-1.615) {completed};
} }
\draw[fill=orangefill,draw=orange,line width=0.65pt] \draw[draw=orange,fill=orangefill,line width=.65pt]
(6.95,-1.84) rectangle (8.45,-1.40); (6.41,-1.84) rectangle (7.85,-1.39);
\node[text=orange,font=\sffamily\tiny] at (7.70,-1.62) {observed tail}; \node[font=\sffamily\tiny,text=orange] at (7.13,-1.615) {observed tail};
\draw[dashed,draw=orange,line width=0.75pt] \draw[draw=orange,dashed,line width=.7pt] (7.85,-2.43)--(7.85,-1.12);
(8.45,-2.95) -- (8.45,-1.10); \node[anchor=south,font=\sffamily\tiny\bfseries,text=orange]
\node[anchor=south,text=orange,font=\sffamily\tiny\bfseries] at (7.85,-1.12) {$t_n$};
at (8.45,-1.10) {cutoff $t_n$}; \node[hint,anchor=west] at (8.16,-1.62)
\node[note,anchor=west] at (8.85,-1.62) {reuse completed payloads;\\recompute temporary tail only};
{replay completed payloads; recompute only\\the temporary tail; exclude future media}; \node[anchor=east,text=teal] at (1.9,-2.33) {local 1 s};
\node[anchor=east,text=teal] at (2.05,-2.31) {local evidence}; \draw[draw=teal,line width=2pt] (7.14,-2.33)--(7.85,-2.33);
\draw[draw=teal,line width=1.8pt] (7.65,-2.31) -- (8.45,-2.31); \node[hint,anchor=west,text=teal] at (8.16,-2.33)
\node[note,anchor=west,text=teal] at (8.85,-2.31) {recent evidence refreshed every 200 ms};
{one-second causal window, refreshed every 200 ms}; \node[hint,anchor=west] at (2.12,-2.74)
\node[anchor=east,text=navy] at (2.05,-2.86) {decision grid}; {Privileged past-activity labels are evaluated separately as an information audit; they never enter this prediction path.};
\draw[draw=navy,line width=0.6pt] (2.3,-2.86) -- (8.45,-2.86);
\foreach \x in {2.42,2.58,...,8.34} {
\draw[draw=navy,line width=0.5pt] (\x,-2.95) -- (\x,-2.77);
}
\node[auditbox,minimum width=42mm] (oracle) at (11.02,-2.78)
{analysis only: past visible-person activity\\aggregate / per-person / delayed / shuffled};
\draw[auditflow] (oracle.north) -- ++(0,0.32) -| (fusion.south);
\node[note,text=purple] at (11.02,-3.35)
{privileged information audit; never a deployable model input};
\end{tikzpicture} \end{tikzpicture}
\end{document} \end{document}

BIN
manuscript/targeted/figures/system_overview_onecolumn.pdf Normal file
View File

Binary file not shown.

50
manuscript/targeted/figures/system_overview_onecolumn.tex Normal file
View File

@@ -0,0 +1,50 @@
\documentclass[tikz,border=2pt]{standalone}
\usetikzlibrary{arrows.meta,positioning}
\definecolor{navy}{HTML}{19324A}
\definecolor{bluefill}{HTML}{EAF1F7}
\definecolor{teal}{HTML}{087F78}
\definecolor{tealfill}{HTML}{E6F4F1}
\definecolor{orange}{HTML}{B85D0F}
\definecolor{orangefill}{HTML}{FFF0DF}
\begin{document}
\begin{tikzpicture}[
font=\sffamily\small,
>=Latex,
box/.style={draw=navy,fill=bluefill,rounded corners=1.5pt,
align=center,minimum height=7mm,inner sep=2pt},
local/.style={box,draw=teal,fill=tealfill},
flow/.style={-{Latex[length=1.6mm]},draw=navy,line width=.6pt}]
\node[box,text width=20mm] (global) at (1.12,3.9)
{Observed AV\\8-s maximum};
\node[box,text width=22mm] (thinker) at (4.13,3.9)
{Frozen Thinker\\readout $G$};
\node[local,text width=20mm] (audio) at (1.12,2.9)
{Recent 1-s\\mixed audio};
\node[local,text width=22mm] (aenc) at (4.13,2.9)
{Frozen encoder\\local $A$};
\node[local,dashed,text width=20mm] (video) at (1.12,1.9)
{Past person\\crops (optional)};
\node[local,dashed,text width=22mm] (venc) at (4.13,1.9)
{Frozen encoder\\pooled $V$};
\node[box,text width=18mm] (head) at (7.05,2.9)
{Validities +\\trained head};
\draw[flow] (global) -- (thinker);
\draw[flow,draw=teal] (audio) -- (aenc);
\draw[flow,draw=teal,dashed] (video) -- (venc);
\draw[flow] (thinker.east) -- ++(.18,0) |- (head.north);
\draw[flow,draw=teal] (aenc) -- (head);
\draw[flow,draw=teal,dashed] (venc.east) -- ++(.18,0) |- (head.south);
\node[box,text width=32mm] (out) at (5.7,.88)
{$10\times3$ future activity\\0.2--2.0-s horizons};
\draw[flow] (head.south) -- (out.north);
\draw[draw=navy,line width=.5pt] (0,.12) -- (8.25,.12);
\node[anchor=west,text=navy] at (0,-.28) {Completed blocks};
\foreach \x in {0,1.04,2.08} {
\draw[draw=navy,fill=bluefill] (\x,-.93) rectangle ++(.96,.35);
}
\draw[draw=orange,fill=orangefill] (3.12,-.93) rectangle ++(.96,.35);
\node[anchor=west,text=orange] at (4.25,-.76) {Observed tail};
\node[anchor=west,text=navy] at (0,-1.33)
{Reuse completed payloads; query a temporary tail at $t_n$.};
\end{tikzpicture}
\end{document}

BIN
manuscript/targeted/figures/task_overview.pdf Normal file
View File

Binary file not shown.

39
manuscript/targeted/figures/task_overview.tex Normal file
View File

@@ -0,0 +1,39 @@
\documentclass[tikz,border=2pt]{standalone}
\usepackage{amsmath}
\usetikzlibrary{arrows.meta,calc}
\definecolor{navy}{HTML}{19324A}
\definecolor{teal}{HTML}{087F78}
\definecolor{orange}{HTML}{B85D0F}
\definecolor{light}{HTML}{EAF1F7}
\definecolor{warm}{HTML}{FFF0DF}
\begin{document}
\begin{tikzpicture}[font=\sffamily\scriptsize,>=Latex]
\node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy] at (0,2.48)
{Predict the invisible wearer's future activity};
\draw[draw=navy,fill=light,rounded corners=2pt] (0,1.25) rectangle (2.55,1.95);
\node[align=center,text=navy] at (1.275,1.60)
{camera + mixed audio\\visible people, unseen wearer};
\draw[-{Latex[length=1.7mm]},draw=navy,line width=.7pt]
(2.65,1.60)--(3.25,1.60);
\draw[draw=navy,line width=.7pt] (3.32,1.28)--(3.32,1.94);
\node[anchor=south,font=\sffamily\tiny\bfseries,text=navy] at (3.32,1.97)
{decision $t_n$};
\draw[draw=orange,fill=warm,rounded corners=2pt] (4.02,1.25) rectangle (7.6,1.95);
\node[align=center,text=orange] at (5.81,1.60)
{10 future states, every 0.2 s\\background / wearer / others};
\draw[draw=navy,line width=.55pt] (.35,.66)--(7.55,.66);
\foreach \x in {.35,1.08,1.81,2.54,3.27,4.00,4.73,5.46,6.19,6.92,7.55}
{\draw[draw=navy,line width=.5pt] (\x,.57)--(\x,.75);}
\draw[draw=teal,line width=2pt] (.35,.94)--(3.27,.94);
\node[anchor=east,font=\sffamily\tiny,text=teal] at (.25,.94) {input};
\draw[draw=orange,line width=2pt] (3.27,.94)--(7.55,.94);
\node[anchor=west,font=\sffamily\tiny,text=orange] at (7.65,.94) {target};
\node[font=\sffamily\tiny,text=navy] at (3.27,.36) {$t_n$};
\node[font=\sffamily\tiny,text=navy] at (4.0,.36) {$+.2$ s};
\node[font=\sffamily\tiny,text=navy] at (7.55,.36) {$+2.0$ s};
\node[anchor=west,font=\sffamily\tiny,text=navy] at (.35,.02)
{ENTRY-like: wearer silent $\to$ active};
\node[anchor=west,font=\sffamily\tiny,text=navy] at (4.0,.02)
{YIELD-like: wearer active $\to$ silent};
\end{tikzpicture}
\end{document}

2
manuscript/targeted/latexmkrc
View File

@@ -1,5 +1,5 @@
my $path_separator = $^O eq 'MSWin32' ? ';' : ':'; my $path_separator = $^O eq 'MSWin32' ? ';' : ':';
my $template_dir = '../ICASSP2026_Paper_Templates'; my $template_dir = '../ICASSP2027_Paper_Templates';
$ENV{'TEXINPUTS'} = $ENV{'TEXINPUTS'} =
$template_dir . $path_separator . ($ENV{'TEXINPUTS'} // ''); $template_dir . $path_separator . ($ENV{'TEXINPUTS'} // '');

474
manuscript/targeted/main.tex
View File

@@ -1,42 +1,46 @@
% Acceptance-target ICASSP 2027 draft grounded only in archived measurements. % Acceptance-target ICASSP 2027 draft grounded only in archived measurements.
% The official 2027 author kit should replace the 2026-compatible style assets % Uses the repository's ICASSP 2027 author kit.
% before submission. Four technical pages plus an optional reference-only page. % Four technical pages plus an optional reference-only page.
\documentclass{article} \documentclass{article}
\usepackage{spconf} \usepackage{spconf}
\usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor} \usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor,float}
\hypersetup{hidelinks} \hypersetup{hidelinks}
\newcommand{\vect}[1]{\boldsymbol{#1}} \newcommand{\vect}[1]{\boldsymbol{#1}}
\newcommand{\G}{\mathrm{G}} \newcommand{\G}{\mathrm{G}}
\newcommand{\A}{\mathrm{A}} \newcommand{\A}{\mathrm{A}}
\newcommand{\V}{\mathrm{V}} \newcommand{\V}{\mathrm{V}}
\setlength{\emergencystretch}{2em} \setlength{\emergencystretch}{2em}
\setcounter{topnumber}{5}
\setcounter{dbltopnumber}{5}
\renewcommand{\topfraction}{0.95}
\renewcommand{\dbltopfraction}{0.95}
\renewcommand{\textfraction}{0.05}
\renewcommand{\floatpagefraction}{0.75}
\renewcommand{\dblfloatpagefraction}{0.75}
\title{Does an Omni Model Know When the Wearer Will Speak?\\ \title{Does an Omni Model Know When the Wearer Will Speak?\\
A Causal Global--Local Audit of Egocentric Speech Anticipation} A Causal Global--Local Audit of Egocentric Speech Anticipation}
\name{Authors withheld in this working draft} \name{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu,
\address{} Xiaoju Dai, Li Zhang, and Guorun Chen}
\address{School of Computer Science, Example University, Shanghai, China}
\begin{document} \begin{document}
\ninept \ninept
\maketitle \maketitle
\begin{abstract} \begin{abstract}
An egocentric conversational agent must anticipate the future speech of an We test whether a frozen Omni readout exposes sub-second evidence for
invisible camera wearer from a causal multiparty audio--video stream. We ask forecasting an invisible camera wearer's speech from causal multiparty
whether a frozen general-purpose Omni state exposes the sub-second evidence audio--video. Our 5-Hz model combines 8-s Qwen2.5-Omni context and recent
needed by this task. Our cutoff-aligned model combines an 8-s Qwen2.5-Omni 1-s audio to predict three activity channels at ten 0.2--2.0-s horizons.
context with the most recent 1-s audio and predicts three activity channels at On EasyCom, a seed-0 screen reaches 81.72\% mAP at 0.2~s and 65.11\%
ten horizons from 0.2 to 2.0~s. In a single-seed EasyCom screen, this model reaches mean mAP, versus 74.20/61.19\% for released EgoSpeak. Wearer offset AP
81.72\% mAP at 0.2~s and 65.11\% averaged over horizons, compared with improves by 7.23 points, onset AP by 1.47. Pooled vision adds only 0.29
74.20\% and 61.19\% for released EgoSpeak. The gain is larger for wearer mean-mAP points. A three-seed causal information audit finds that privileged
offset anticipation (+7.23 points) than onset anticipation (+1.47). Pooled past visible-person activity adds 4.83 points at 0.2~s and 1.77 mean-mAP
visual features add only 0.29 mean-mAP points. A separate three-seed, points; visibility and delayed activity add little. A separate no-box
strictly causal information audit explains this gap: privileged past activity full-frame screen reaches 82.38/65.54\%, but unmatched training and head
of visible participants adds 4.83 points at 0.2~s and 1.77 mean-mAP points, capacity prevent visual attribution. Timely evidence access, not semantic
whereas visibility, one-second-delayed activity, and identity shuffling show floor understanding, remains the supported conclusion.
that timeliness matters more than identity continuity. The results isolate an
evidence-access bottleneck: recent acoustic evidence is already useful, while
current generic visual pooling does not recover the available visible-floor
signal.
\end{abstract} \end{abstract}
\begin{keywords} \begin{keywords}
@@ -51,8 +55,18 @@ Conversational agents must decide not only \emph{what} to say, but also
\emph{when} to participate. This decision is particularly asymmetric in an \emph{when} to participate. This decision is particularly asymmetric in an
egocentric stream: the camera wearer is normally invisible, yet the model must egocentric stream: the camera wearer is normally invisible, yet the model must
forecast that wearer's behavior from mixed audio and observations of other forecast that wearer's behavior from mixed audio and observations of other
people. A useful predictor must update before a turn boundary is known and people. As Fig.~\ref{fig:task} illustrates, prediction precedes any known
must not use media from after its decision cutoff. turn boundary and excludes media after the decision cutoff.
\begin{figure}[H]
\centering
\includegraphics[width=\columnwidth]{figures/task_overview.pdf}
\caption{Task, not an observed event example. At each 5-Hz decision cutoff,
only the past audio--video prefix is input; ten future activity states are
targets. ENTRY-like and YIELD-like refer to wearer activity transitions,
not semantic floor ownership.}
\label{fig:task}
\end{figure}
Voice Activity Projection (VAP) casts turn taking as continuous prediction of Voice Activity Projection (VAP) casts turn taking as continuous prediction of
future activity \cite{ekstedt2022vap}. EgoSpeak brings this objective to future activity \cite{ekstedt2022vap}. EgoSpeak brings this objective to
@@ -74,28 +88,22 @@ state $\G$ from explicit recent evidence $\A$ and optional $\V$, and treat
their value as an empirical question rather than assuming that a larger their value as an empirical question rather than assuming that a larger
backbone solves temporal access. backbone solves temporal access.
This paper makes three contributions. First, we formulate a strictly causal We contribute a cutoff-aligned global--local audit, a bounded Omni stream
global--local audit for invisible-wearer activity forecasting, with a common with observed-tail refresh and recent audio, and a separation of deployable
5-Hz cutoff and ten future horizons. Second, we realize the audit with a features from an analysis-only visible-activity oracle. Generic pooled or
bounded Omni stream that combines completed history with a temporary observed current-frame person features recover little of the available activity signal;
tail and a separate one-second audio path. Third, we distinguish deployable timely activity estimation is the next modeling target.
feature gains from an analysis-only visible-activity oracle. This reveals a
specific negative result---generic pooled or current-frame person features do
not recover most of the available local-floor information---and identifies
timely activity estimation, rather than identity continuity, as the next
modeling target.
\begin{figure*}[t] \begin{figure}[t]
\centering \centering
\includegraphics[width=0.97\textwidth]{figures/system_overview.pdf} \includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
\caption{Global--local causal audit. At each 200-ms cutoff, the prediction \caption{Deployable global--local path and bounded stream. At each 200-ms
path fuses an integrated Omni state with explicit recent audio and optional cutoff, the head fuses the frozen integrated Omni state with explicit recent
person-crop vision. Completed 2-s blocks are reused while only the observed audio and optional pooled person-crop vision. Completed 2-s payloads are
partial tail is temporarily recomputed. The purple branch is a privileged reused; only the observed partial tail is temporarily recomputed. Privileged
past-activity audit used to test information value; it is never a deployable past-activity audit labels are kept outside this graph and are not inputs.}
input and contains no future activity.}
\label{fig:overview} \label{fig:overview}
\end{figure*} \end{figure}
\section{Method} \section{Method}
\label{sec:method} \label{sec:method}
@@ -137,16 +145,13 @@ is frozen-backbone adaptation rather than end-to-end training.
\subsection{Cutoff-Aligned Streaming} \subsection{Cutoff-Aligned Streaming}
The global stream uses source-aligned 2-s blocks and an 8-s maximum Figure~\ref{fig:overview} uses source-aligned 2-s blocks under an 8-s
presentation budget. After warm-up, three completed blocks and the observed presentation cap. After warm-up, three completed blocks and the observed
part of the current block are available; the exact span varies with block current tail are available; span varies with block phase. Completed
phase. Completed modality payloads are reused when the bounded language-model modality payloads are reused. At each cutoff, the tail is cropped at $t_n$
cache is rebuilt. At every cutoff, the incomplete current block is cropped at and queried in a temporary cache, without committing it. Only completed
$t_n$, appended to a temporary cache, and queried for $\vect g_n$. Neither blocks are committed; session or source discontinuities reset caches. This
the temporary tail nor the query is committed. At a block boundary the refreshes the logical state at 5~Hz without backfilling earlier rows.
completed payload is committed once and a new tail begins. Sessions and
source discontinuities reset all caches. Thus the logical state is refreshed
at 5~Hz without backfilling a completed block into earlier rows.
\subsection{Visible-Activity Information Audit} \subsection{Visible-Activity Information Audit}
@@ -161,6 +166,148 @@ second, and (v) one OR-pooled aggregate-visible history. These conditions
never expose future activity, but their speech labels make them privileged never expose future activity, but their speech labels make them privileged
diagnostics, not deployable features or upper bounds. diagnostics, not deployable features or upper bounds.
\begin{table*}[t]
\caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B:
three-seed privileged past-activity audit, which is not deployable. Mean is
ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events.
F3 is a no-box, two-frame causal full-frame screen; it and the historical
deployable rows are exploratory seed-0 runs with unmatched head/training
details. F1 has three seeds. Dashes are unavailable, not zero.}
\label{tab:main}
\centering
\normalsize
\renewcommand{\arraystretch}{0.87}
\setlength{\tabcolsep}{5.0pt}
\begin{tabular}{@{}lccccc@{}}
\toprule
Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
& Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
\midrule
EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086
& 65.400 & 26.019 & 45.371 \\\\
No-box full-frame F3 & 82.380 & -- & 65.540 & -- & -- \\\\
Current-frame person set F1 & $81.777{\pm}0.156$ & --
& $64.951{\pm}0.095$ & -- & -- \\\\
Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\
Past-delayed $\A$ (L3) & -- & -- & -- & -- & -- \\\\
Current-state persistence (L4) & -- & -- & -- & -- & -- \\\\
\bottomrule
\end{tabular}
\par\smallskip
\emph{Panel B: privileged past-activity audit (mean $\pm$ sample SD).}\par\smallskip
\normalsize
\setlength{\tabcolsep}{3.2pt}
\begin{tabular}{@{}lccc@{}}
\toprule
Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
\midrule
None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
& $\mathbf{66.951{\pm}.022}$ \\\\
\bottomrule
\end{tabular}
\end{table*}
\begin{table*}[t]
\caption{Chunk and context audit. Panel A is an unmeasured matched T/C
matrix: each row pairs $\G$ and $\G+\A$ on common valid cutoffs. Panel B is
a separate exploratory seed-0 global-only screen (10 epochs, 8-s cap,
2-FPS video); its rows are not entries in Panel A. Dashes are pending.}
\label{tab:stream_context}
\centering\normalsize
\renewcommand{\arraystretch}{0.87}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{@{}llccccc@{}}
\toprule
Group & Refresh / cap & $\G$ mAP & $\G+\A$ mAP & $\Delta$ mAP
& $\Delta$ On & $\Delta$ Off \\
\midrule
T & Completed-only, 8 s & -- & -- & -- & -- & -- \\
T & Observed-tail, 8 s & -- & -- & -- & -- & -- \\
\midrule
C & Tail-aware, 2 s & -- & -- & -- & -- & -- \\
C & Tail-aware, 4 s & -- & -- & -- & -- & -- \\
C & Tail-aware, 6 s & -- & -- & -- & -- & -- \\
C & Tail-aware, 8 s & -- & -- & -- & -- & -- \\
C & Tail-aware, 10 s & -- & -- & -- & -- & -- \\
C & Tail-aware, 12 s & -- & -- & -- & -- & -- \\
\bottomrule
\end{tabular}
\par\smallskip
\emph{Panel B: reported 200-ms chunk screen; dense mAP (\%).}\par\smallskip
\begin{tabular}{@{}lccc@{}}
\toprule
Global feature construction & 0.2-s mAP & Mean mAP & Processor calls \\
\midrule
Native 2-s completed hold & 59.62 & 55.19 & 9,539 \\
Non-native 200-ms commits & 67.05 & 57.84 & 95,453 \\
Completed blocks + observed tail & 70.85 & 59.17 & -- \\
\bottomrule
\end{tabular}
\end{table*}
\begin{table}[t]
\caption{B result slots: paired within-backbone local-audio gains (unmeasured).
MiniCPM requires a matched readout/refresh/history audit before comparison.}
\label{tab:backbone}
\centering\normalsize
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}lccc@{}}
\toprule
Backbone & $\G$ & $\G+\A$ & $\Delta$ \\
\midrule
Qwen Omni 3B & -- & -- & -- \\
Qwen Omni 7B & -- & -- & -- \\
MiniCPM-o 4.5 & -- & -- & -- \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{H result slots: predeclared activity slices (unmeasured). Report
positive/negative counts and per-session scores; AP is undefined for a
pure-negative slice, which instead uses calibrated FPR.}
\label{tab:hard_slices}
\centering\normalsize
\setlength{\tabcolsep}{2.5pt}
\begin{tabular}{@{}lccc@{}}
\toprule
Slice & $N_+/N_-$ & AP & FPR \\
\midrule
Wearer onset & -- & -- & -- \\
Wearer offset & -- & -- & -- \\
Other pause (negative) & -- & n/a & -- \\
Overlap (negative) & -- & n/a & -- \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{S result slots: full-path cost on the same hardware (unmeasured).
Include I/O and block transitions; logical 5 Hz is not measured throughput.}
\label{tab:efficiency}
\centering\normalsize
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}lccc@{}}
\toprule
Stage & P50 ms & P95 ms & Peak GB \\
\midrule
Completed block & -- & -- & -- \\
Observed tail & -- & -- & -- \\
Local path + head & -- & -- & -- \\
Full path & -- & -- & -- \\
\bottomrule
\end{tabular}
\end{table}
\section{Experiments} \section{Experiments}
\label{sec:experiments} \label{sec:experiments}
@@ -191,36 +338,16 @@ already sufficient. Because that screen used Session~12 model selection
whereas the historical pooled row did not, their difference is diagnostic, whereas the historical pooled row did not, their difference is diagnostic,
not a matched causal attribution. not a matched causal attribution.
\textbf{Six pre-specified control families.}
We freeze six pending control families: (L) capacity and recency, (T) tail
refresh crossed with local audio, (C) six context caps, (B) paired backbone
transfer, (H/Q) error slices and timelines, and (S) full-path efficiency.
They share the official split and physical cutoffs; the 8-s T/C row reuses
one run. The source retains unmeasured hypotheses and falsifiers.
\section{Results} \section{Results}
\label{sec:results} \label{sec:results}
\begin{table*}[t]
\caption{EasyCom Sessions~1--3 results (\%). $\mathrm{mAP}_{.2}$ and
$\mathrm{WAP}_{.2}$ are dense 0.2-s metrics; Mean averages dense mAP over ten
horizons. On/Off are raw wearer endpoint AP, not semantic event scores.
Deployable rows are archived single-seed screens; $\pm$ is shown only for the
three-seed F1 screen. Dashes denote unavailable measurements.}
\label{tab:main}
\centering
\scriptsize
\setlength{\tabcolsep}{5.0pt}
\begin{tabular}{@{}lccccc@{}}
\toprule
Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
& Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
\midrule
EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
Omni $\G+\A+$pooled-$\V$ & \textbf{82.027} & \textbf{85.086}
& \textbf{65.400} & \textbf{26.019} & 45.371 \\\\
Current-frame person set F1 & $81.777{\pm}0.156$ & --
& $64.951{\pm}0.095$ & -- & -- \\\\
\bottomrule
\end{tabular}
\end{table*}
\subsection{Global Context and Recent Audio Are Complementary} \subsection{Global Context and Recent Audio Are Complementary}
Table~\ref{tab:main} shows that $\A$ is strong at 0.2~s, while $\G$ is Table~\ref{tab:main} shows that $\A$ is strong at 0.2~s, while $\G$ is
@@ -238,6 +365,70 @@ the local-only head, so this comparison demonstrates predictive
complementarity, not that semantic reasoning alone causes the gain. A wider complementarity, not that semantic reasoning alone causes the gain. A wider
history-matched control is required for a representation-level causal claim. history-matched control is required for a representation-level causal claim.
% Retained as a planning aid; illustrative, unmeasured patterns do not belong
% in the page-limited submission preview.
\iffalse
\begin{table*}[t]
\caption{Six-group \textbf{illustrative scenario, not measured results}.
These are conditional, pre-run interpretations for the pending matrix, not
EasyCom scores. $\Delta$ denotes paired mean-mAP change on common valid
cutoffs; signs and shape must be replaced by measured values even if contrary
to the story. L/T/C/B require seeds 0--2 and per-session reporting.}
\label{tab:scenario}
\centering
\normalsize
\renewcommand{\arraystretch}{1.12}
\setlength{\tabcolsep}{4.2pt}
\begin{tabular}{@{}p{0.095\textwidth}p{0.29\textwidth}p{0.29\textwidth}p{0.235\textwidth}@{}}
\toprule
Group & Frozen comparison & Illustrative pattern \emph{if observed} & Falsifier / interpretation limit \\
\midrule
L1--4 & $\A,\G,\G+\A,\G+\A+\V$; matched-width $\G$;
past-delayed local cues; current-state persistence
& $\Delta(\G+\A-\G)>0$ after width matching; delayed $\A$ loses
near-horizon benefit; persistence does not recover the gain
& Width-only or persistence matches fusion, or delay does not hurt:
do not attribute gain to timely distinct evidence. \\
\addlinespace[2.5pt]
T1--2 & 2-s completed-only vs 200-ms observed-tail refresh,
crossed with $\A$ on/off; causal replay and phase audit
& Tail helps most without $\A$; local $\A$ still adds at each refresh;
future-only media never changes prior readouts
& No independent local gain under tail refresh implies redundancy;
replay/leakage failure invalidates timing claims. \\
\addlinespace[2.5pt]
C1--2 & Genuine 2/4/6/8/10/12-s feature extraction; fixed 1-s $\A$;
add $\G$-only at each cap for a window-stability claim
& Fused performance plateaus around mid-range caps while
$\G$-only remains more cap-sensitive
& If long history substitutes for $\A$, local-evidence claim narrows;
without C2, no robustness comparison is justified. \\
\addlinespace[2.5pt]
B0--2 & Paired $\G$ and $\G+\A$ on Qwen 3B/7B and, after causal
interface audit, MiniCPM-o 4.5
& Positive within-backbone paired $\Delta$ on all admissible rows;
absolute backbone ranking is not inferred
& Failed causal/history match is unsupported, not zero gain;
one negative paired row defeats universal transfer. \\
\addlinespace[2.5pt]
H1/Q1 & Predefined onset/offset and other-pause/overlap slices;
same-horizon, cutoff-marked case selection
& Offset gains coexist with hard wearer-onset negatives; examples
explain, rather than replace, counted slice statistics
& Pure-negative slice AP is undefined; selected examples cannot
establish reduced false-entry rate. \\
\addlinespace[2.5pt]
S1 & End-to-end p50/p95 latency and peak memory, including I/O,
encoding, tail recomputation, local path, and boundaries
& A feasible throughput--quality tradeoff can be plotted
against 200-ms output cadence
& If p95 exceeds cadence, describe logical 5-Hz prediction only,
not real-time deployment. \\
\bottomrule
\end{tabular}
\end{table*}
\fi
\subsection{Generic Vision Does Not Recover the Available Signal} \subsection{Generic Vision Does Not Recover the Available Signal}
Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
@@ -245,32 +436,15 @@ Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
averaged over horizons; onset/offset changes are +0.059/+0.120. A richer averaged over horizons; onset/offset changes are +0.059/+0.120. A richer
current-frame person-set model is stable across seeds but reaches lower mean current-frame person-set model is stable across seeds but reaches lower mean
mAP than the historical pooled row under a non-matched selection protocol. mAP than the historical pooled row under a non-matched selection protocol.
These results do not support a claim of learned speaker grounding. These results do not support a claim of learned speaker grounding. A separate
seed-0 no-box full-frame F3 screen reaches 82.38\% at 0.2~s and 65.54\%
mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same
causal-valid dense protocol. F3 adds a two-frame 2-FPS token history and a
shared GRU, while head capacity and training batch differ; raw boundary
metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point
difference does not isolate visual motion or floor behavior.
\begin{table}[t] Table~\ref{tab:main}, panel B, locates the missing information. Aggregate visible
\caption{Three-seed privileged-information audit (mean $\pm$ sample standard
deviation, \%). Every row includes $\G+\A$; activity conditions use only
past labels. Bold marks the best diagnostic, not a deployable model.}
\label{tab:oracle}
\centering
\scriptsize
\setlength{\tabcolsep}{3.2pt}
\begin{tabular}{@{}lccc@{}}
\toprule
Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
\midrule
None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
& $\mathbf{66.951{\pm}.022}$ \\\\
\bottomrule
\end{tabular}
\end{table}
Table~\ref{tab:oracle} locates the missing information. Aggregate visible
activity improves over the audio baseline by 4.826 points at 0.2~s and 1.775 activity improves over the audio baseline by 4.826 points at 0.2~s and 1.775
mean-mAP points. Visibility alone changes mean mAP by only +0.040, and mean-mAP points. Visibility alone changes mean mAP by only +0.040, and
one-second-delayed activity by +0.164. Thus availability or extra parameters one-second-delayed activity by +0.164. Thus availability or extra parameters
@@ -282,45 +456,73 @@ The majority of the oracle's 0.2-s macro gain comes from background and
others; wearer AP rises by only 0.572. This is why neither the oracle nor a others; wearer AP rises by only 0.572. This is why neither the oracle nor a
dense mAP gain establishes fewer false wearer entries. dense mAP gain establishes fewer false wearer entries.
\subsection{What the Audit Establishes} \begin{figure}[t]
\centering
\fbox{\begin{minipage}[c][18mm][c]{0.94\columnwidth}
\centering
\textbf{Q1 case slots: examples pending}\\[2pt]
Onset / other-pause hard negative \quad|\quad Offset / overlap\\
Show source cutoff, observed activity and same-horizon wearer scores.
\end{minipage}}
\caption{Reserved H/Q case figure. Select examples by a frozen rule after
the slice counts in Table~\ref{tab:hard_slices} are available. No case or
behavioral improvement is asserted by this placeholder.}
\label{fig:hq_cases}
\end{figure}
Three conclusions survive the controls. First, a general-purpose global \subsection{Pending Controls and Limits}
state and a one-second acoustic readout provide different useful timescales.
Second, generic visual pooling has not converted visible people into reliable
local floor evidence. Third, perfect timely aggregate activity contains
recoverable information, whereas identity contributes little under the
current target. A dedicated causal active-speaker front end---designed for
mouth motion, blur, and short audio--visual synchrony as in egocentric ASD
\cite{huh2025egoasd}---is consequently a better-motivated next step than a
larger identity-aware fusion module.
The evidence remains bounded. EasyCom has only three official test sessions; Tables~\ref{tab:stream_context}--\ref{tab:efficiency} reserve result slots
the deployable matrix is single seed and has been examined repeatedly. The for T/C, B, H, and S; Fig.~\ref{fig:hq_cases} reserves Q cases. Panel B of
oracle uses labels and is not a performance upper bound. Provided boxes omit Table~\ref{tab:stream_context} separately reports a global-only chunk screen:
online detection cost, and logical 5-Hz output is not a measured throughput the 200-ms commits beat 2-s hold after startup exclusion but require about
claim. Finally, aggregate others cannot supply addressee or semantic ten times as many processor calls across 12 sessions. Freshness and chunk
interruption ground truth. Confirmatory behavioral claims require a frozen construction both change, and the older tail bank is not extraction-matched;
training-side operating point, competing-risk event attribution, and these values do not fill the pending T/C factorial or establish real-time
multi-seed matched controls. throughput. L requires a
trained width-matched global control and past-delayed features on identical
valid pairs; T crosses tail refresh with local audio; C re-extracts each
causal context cap and pairs it with a global-only run. B compares the
within-backbone audio increment only after readout and history audits.
H/Q reports slice counts and cutoff-marked examples; S measures full-path
latency and memory. The oracle uses privileged labels, provided boxes omit
online detection, and aggregate others cannot establish addressee or
semantic floor ownership.
\subsection{Decision Rules and Reproducibility}
The unit of comparison is the same physical cutoff and future target time,
not an arbitrary feature row. For every new condition we will archive
input-source timestamps, common valid-pair counts, seeds, per-session AP,
trainable parameter counts, processor/model revisions, checkpoint hashes,
and prediction scores. Frames and horizons are correlated, not independent
replicates.
We retain the fixed four-epoch recipe for the matched matrix. Any loss
weight, threshold, context selection, or case-selection rule is frozen
using training-side evidence before looking at Sessions~1--3. Because
these test sessions have already informed this research program, even
new seed runs remain exploratory rather than restoring an unseen test set.
The raw endpoint AP in Table~\ref{tab:main} may motivate a behavioral
hypothesis, but an ENTRY/YIELD error-rate claim additionally requires
one-event-one-vote attribution with response windows censored at the next
other onset and yield windows at the next other offset. Until that
competing-risk evaluator passes real-timeline audit, we report only dense
forecasting and endpoint anticipation.
\section{Conclusion} \section{Conclusion}
We presented a strictly causal global--local audit of egocentric wearer speech In this strictly causal audit, frozen Omni context and recent audio improve
anticipation. Frozen Omni context and recent audio are strongly complementary, wearer activity forecasting, especially future speech cessation. Generic
especially for future wearer cessation. In contrast, pooled or current-frame pooled vision does not recover the timely visible-activity signal exposed by
person features do not recover the local-floor signal revealed by a causal the analysis-only oracle. Matched controls and censored event evaluation
visible-activity oracle. The actionable conclusion is deliberately narrow: remain necessary before claiming improved floor behavior.
the next bottleneck is timely, learnable local activity evidence, not simply
more multimodal capacity or persistent identity. This distinction provides a
clearer target for future egocentric conversational agents and prevents dense
forecasting gains from being mistaken for semantic floor control.
\section{Acknowledgment} \section{Acknowledgment}
OpenAI Codex assisted with substantive drafting and figure layout in the OpenAI Codex assisted with drafting and figure layout; the authors remain
abstract, introduction, and presentation of the archived analysis; the responsible for all experimental claims and final content.
authors remain responsible for verification, experimental claims, and the
final manuscript.
\clearpage
\bibliographystyle{IEEEbib} \bibliographystyle{IEEEbib}
\bibliography{references} \bibliography{references}
\end{document} \end{document}