Update targeted ICASSP 2027 manuscript and results
This commit is contained in:
@@ -2,15 +2,16 @@ LATEXMK ?= latexmk
|
||||
|
||||
.PHONY: all clean figure
|
||||
|
||||
all: figures/system_overview.pdf
|
||||
all: figures/system_overview_onecolumn.pdf figures/task_overview.pdf
|
||||
mkdir -p build
|
||||
$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=build main.tex
|
||||
|
||||
figure: figures/system_overview.pdf
|
||||
figure: figures/system_overview_onecolumn.pdf figures/task_overview.pdf
|
||||
|
||||
figures/system_overview.pdf: figures/system_overview.tex
|
||||
figures/%.pdf: figures/%.tex
|
||||
$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=figures $<
|
||||
|
||||
clean:
|
||||
$(LATEXMK) -C -outdir=build main.tex
|
||||
$(LATEXMK) -c -outdir=figures figures/system_overview.tex
|
||||
$(LATEXMK) -c -outdir=figures figures/system_overview_onecolumn.tex
|
||||
$(LATEXMK) -c -outdir=figures figures/task_overview.tex
|
||||
|
||||
@@ -14,8 +14,21 @@ that is the strongest complete story supported by archived evidence:
|
||||
- a three-seed causal oracle shows that timely visible-participant activity is
|
||||
informative, while identity continuity is not currently necessary.
|
||||
|
||||
The draft also reflects two seed-0 exploratory screens reported in
|
||||
`../main/main.tex`: no-box full-frame F3 (Table 1) and a global-only
|
||||
chunk-construction comparison (Table 2, panel B). Neither supplies
|
||||
matched visual-motion evidence, endpoint gains, or the pending T/C factorial.
|
||||
The result directories cited by the main draft for F3 and its G+A recheck
|
||||
are absent from this checkout, so their underlying logs have not been
|
||||
independently audited here.
|
||||
|
||||
No semantic ENTRY/YIELD, addressee, online-detector, or real-time-throughput
|
||||
claim is made. The deployable matrix is explicitly single-seed and exploratory.
|
||||
The six-group L/T/C/B/H-Q/S table remains in `main.tex` as an
|
||||
**illustrative, unmeasured planning aid**, excluded from the page-limited
|
||||
PDF. Its directional statements are hypotheses and falsifiers, not EasyCom
|
||||
scores. Replace or remove every row after the corresponding experiment,
|
||||
including results that contradict the working story.
|
||||
|
||||
## Build
|
||||
|
||||
@@ -24,13 +37,25 @@ cd manuscript/targeted
|
||||
make
|
||||
```
|
||||
|
||||
The editable sources are `main.tex`, `references.bib`, and
|
||||
`figures/system_overview.tex`. Generated files are written to `build/`.
|
||||
The editable sources are `main.tex`, `references.bib`,
|
||||
`figures/task_overview.tex`, and `figures/system_overview_onecolumn.tex`.
|
||||
Generated LaTeX files are written to `build/`.
|
||||
|
||||
The build currently uses the repository's unmodified ICASSP 2026 style assets
|
||||
as a compatibility preview. Replace the `template_dir` in `latexmkrc` with the
|
||||
official ICASSP 2027 author kit before submission and repeat all page/layout
|
||||
checks.
|
||||
The preview is four technical pages plus a reference-only fifth page. Both
|
||||
the task diagram (Fig. 1) and the system diagram (Fig. 2) occupy one column.
|
||||
The current PDF reserves five numbered result tables: measured deployable
|
||||
and oracle screens share Table 1; L controls extend its first panel. The
|
||||
unmeasured T/C matrix and a separate measured chunk screen share Table 2;
|
||||
B, H, and S occupy Tables 3--5. Pending cells are explicitly unmeasured.
|
||||
Figure 3 is an H/Q case slot, not a selected or measured example. Its
|
||||
cutoff-marked timelines must be inserted only after the slice protocol and
|
||||
case-selection rule are frozen. The single-column system figure is generated
|
||||
from figures/system_overview_onecolumn.tex; the earlier full-width source
|
||||
is retained for comparison.
|
||||
|
||||
The build uses the repository's ICASSP 2027 LaTeX assets in
|
||||
`../ICASSP2027_Paper_Templates/`, selected by `latexmkrc`. The supplied
|
||||
`spconf.sty` and `IEEEbib.bst` match the 2026 copies.
|
||||
|
||||
## Evidence sources
|
||||
|
||||
|
||||
@@ -78,10 +78,23 @@ the science.
|
||||
causal active-speaker/floor estimator and require correct-time evidence to
|
||||
beat delayed, removed, and capacity-matched controls. Otherwise retain the
|
||||
honest audio-local paper and remove visual-method language.
|
||||
5. Rebuild with the official 2027 kit, keep technical content within four
|
||||
pages, put only references on an optional fifth page, and retain the IEEE
|
||||
AI-assisted drafting disclosure.
|
||||
5. Recheck the four-page technical limit after inserting results. The current
|
||||
build uses the 2027 kit and puts only references on page five; retain the
|
||||
IEEE AI-assisted drafting disclosure.
|
||||
|
||||
Passing gates 1--3 would support a solid ICASSP empirical paper. Passing gate
|
||||
4 with a meaningful matched-recall behavior gain would support the stronger
|
||||
method paper originally envisioned.
|
||||
|
||||
## Review 4: expanded five-page working preview
|
||||
|
||||
The layout now uses four pages for technical content and a fifth page for
|
||||
references only. The Introduction includes a causal task timeline; the
|
||||
method figure separates deployable inputs from privileged audit labels. The
|
||||
results presentation has measured main/oracle tables and a clearly labeled
|
||||
six-group **illustrative scenario** with explicit falsifiers. This improves
|
||||
readability and makes the intended experimental logic reviewable, but it does
|
||||
not raise the scientific verdict above borderline: L/T/C/B/H-Q/S values and
|
||||
full-pipeline efficiency remain unmeasured, and the scenario cannot be cited
|
||||
as evidence. Re-review after gates 1--4 and replacement of the scenario table
|
||||
with actual per-seed/per-session results.
|
||||
|
||||
Binary file not shown.
@@ -1,92 +1,71 @@
|
||||
\documentclass[tikz,border=2pt]{standalone}
|
||||
\usepackage{amsmath}
|
||||
\usetikzlibrary{arrows.meta,calc,positioning}
|
||||
\usetikzlibrary{arrows.meta,positioning,calc}
|
||||
\definecolor{navy}{HTML}{19324A}
|
||||
\definecolor{bluefill}{HTML}{EAF1F7}
|
||||
\definecolor{teal}{HTML}{0B7A75}
|
||||
\definecolor{tealfill}{HTML}{E5F4F2}
|
||||
\definecolor{orange}{HTML}{C96A12}
|
||||
\definecolor{teal}{HTML}{087F78}
|
||||
\definecolor{tealfill}{HTML}{E6F4F1}
|
||||
\definecolor{orange}{HTML}{B85D0F}
|
||||
\definecolor{orangefill}{HTML}{FFF0DF}
|
||||
\definecolor{purple}{HTML}{6E4B8B}
|
||||
\definecolor{purplefill}{HTML}{F1EBF6}
|
||||
\definecolor{grayline}{HTML}{697782}
|
||||
\definecolor{grayline}{HTML}{687782}
|
||||
\begin{document}
|
||||
\begin{tikzpicture}[
|
||||
font=\sffamily\scriptsize, >=Latex,
|
||||
flow/.style={-{Latex[length=1.7mm]},line width=0.65pt,draw=navy},
|
||||
font=\sffamily\scriptsize,>=Latex,
|
||||
flow/.style={-{Latex[length=1.6mm]},line width=.65pt,draw=navy},
|
||||
localflow/.style={flow,draw=teal},
|
||||
auditflow/.style={flow,draw=purple,dashed},
|
||||
box/.style={draw=navy,rounded corners=1.5pt,line width=0.65pt,
|
||||
fill=bluefill,minimum height=8mm,align=center,inner sep=2.2pt},
|
||||
localbox/.style={box,draw=teal,fill=tealfill},
|
||||
auditbox/.style={box,draw=purple,fill=purplefill,dashed},
|
||||
note/.style={font=\sffamily\tiny,text=grayline,align=center}
|
||||
]
|
||||
block/.style={draw=navy,fill=bluefill,rounded corners=1.6pt,
|
||||
line width=.6pt,minimum height=8.5mm,align=center,inner sep=2pt},
|
||||
local/.style={block,draw=teal,fill=tealfill},
|
||||
hint/.style={font=\sffamily\tiny,text=grayline,align=center}]
|
||||
\node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy]
|
||||
at (-0.1,3.2) {Prediction path: integrated history plus explicit recent evidence};
|
||||
\node[align=center,text=navy] (av) at (0.7,2.45)
|
||||
{observed full-scene\\audio--video prefix};
|
||||
\node[box] (omni) at (3.25,2.45) {frozen Qwen2.5-Omni\\8-s maximum context};
|
||||
\node[box] (global) at (5.95,2.45) {last-four-layer\\global state $G_n$};
|
||||
\node[align=center,text=teal] (audio) at (0.7,1.23)
|
||||
{most recent 1-s\\mixed audio};
|
||||
\node[localbox] (ae) at (3.25,1.23) {frozen audio encoder\\valid-token mean};
|
||||
\node[localbox] (ap) at (5.95,1.23) {local projection\\$\widetilde A_n$ + validity};
|
||||
\node[align=center,text=teal] (video) at (0.7,0.02)
|
||||
{recent causal\\person crops};
|
||||
\node[localbox,dashed] (ve) at (3.25,0.02)
|
||||
{frozen visual encoder\\crop / person pooling};
|
||||
\node[localbox,dashed] (vp) at (5.95,0.02)
|
||||
{optional projection\\$\widetilde V_n$ + validity};
|
||||
\node[box,minimum width=21mm] (fusion) at (8.65,1.23)
|
||||
{cutoff-aligned\\fusion head};
|
||||
\node[box,minimum width=24mm] (out) at (11.55,1.23)
|
||||
{$10\!\times\!3$ activity forecast\\0.2--2.0 s};
|
||||
\draw[flow] (av) -- (omni);
|
||||
\draw[flow] (omni) -- (global);
|
||||
at (0,3.12) {Deployable prediction path at the same causal cutoff $t_n$};
|
||||
\node[block,minimum width=25mm] (history) at (1.35,2.26)
|
||||
{observed AV history\\8-s maximum budget};
|
||||
\node[block,minimum width=25mm] (omni) at (4.15,2.26)
|
||||
{frozen Omni Thinker\\last-four-layer state};
|
||||
\node[local,minimum width=25mm] (audio) at (1.35,1.13)
|
||||
{last 1-s mixed audio\\cropped at $t_n$};
|
||||
\node[local,minimum width=25mm] (ae) at (4.15,1.13)
|
||||
{frozen audio encoder\\valid-token mean};
|
||||
\node[local,dashed,minimum width=25mm] (video) at (1.35,0)
|
||||
{optional recent frames\\provided person boxes};
|
||||
\node[local,dashed,minimum width=25mm] (ve) at (4.15,0)
|
||||
{frozen visual encoder\\pooled person tokens};
|
||||
\node[block,minimum width=20mm] (head) at (7.15,1.13)
|
||||
{normalized features\\+ validity $\to$ MLP};
|
||||
\node[block,minimum width=26mm] (out) at (10.4,1.13)
|
||||
{$10\times3$ forecast\\0.2--2.0-s horizons};
|
||||
\draw[flow] (history) -- (omni);
|
||||
\draw[localflow] (audio) -- (ae);
|
||||
\draw[localflow] (ae) -- (ap);
|
||||
\draw[localflow,dashed] (video) -- (ve);
|
||||
\draw[localflow,dashed] (ve) -- (vp);
|
||||
\draw[flow] (global.east) -- ++(0.30,0) |- (fusion.north);
|
||||
\draw[localflow] (ap) -- (fusion);
|
||||
\draw[localflow,dashed] (vp.east) -- ++(0.30,0) |- (fusion.south);
|
||||
\draw[flow] (fusion) -- (out);
|
||||
\node[note] at (11.45,0.18)
|
||||
{background / invisible wearer / aggregate others};
|
||||
|
||||
\draw[grayline,line width=0.45pt] (-0.1,-0.62) -- (12.9,-0.62);
|
||||
\draw[flow] (omni.east) -- ++(.38,0) |- (head.north);
|
||||
\draw[localflow] (ae) -- (head);
|
||||
\draw[localflow,dashed] (ve.east) -- ++(.38,0) |- (head.south);
|
||||
\draw[flow] (head) -- (out);
|
||||
\node[hint,text=navy] at (10.4,.17)
|
||||
{background / wearer / aggregate others};
|
||||
\draw[draw=grayline,line width=.45pt] (0,-.62)--(11.85,-.62);
|
||||
\node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy]
|
||||
at (-0.1,-0.94) {Strictly causal stream at cutoff $t_n$};
|
||||
\node[anchor=east,text=navy] at (2.05,-1.62) {Omni media};
|
||||
\foreach \a/\b in {2.3/3.85,3.85/5.4,5.4/6.95} {
|
||||
\draw[fill=bluefill,draw=navy,line width=0.55pt]
|
||||
(\a,-1.84) rectangle (\b,-1.40);
|
||||
\node[text=navy,font=\sffamily\tiny] at ({(\a+\b)/2},-1.62) {completed 2 s};
|
||||
at (0,-.94) {Bounded streaming state: no media after $t_n$};
|
||||
\node[anchor=east,text=navy] at (1.9,-1.62) {Omni blocks};
|
||||
\foreach \a/\b in {2.12/3.55,3.55/4.98,4.98/6.41} {
|
||||
\draw[draw=navy,fill=bluefill,line width=.55pt]
|
||||
(\a,-1.84) rectangle (\b,-1.39);
|
||||
\node[font=\sffamily\tiny,text=navy] at ({(\a+\b)/2},-1.615) {completed};
|
||||
}
|
||||
\draw[fill=orangefill,draw=orange,line width=0.65pt]
|
||||
(6.95,-1.84) rectangle (8.45,-1.40);
|
||||
\node[text=orange,font=\sffamily\tiny] at (7.70,-1.62) {observed tail};
|
||||
\draw[dashed,draw=orange,line width=0.75pt]
|
||||
(8.45,-2.95) -- (8.45,-1.10);
|
||||
\node[anchor=south,text=orange,font=\sffamily\tiny\bfseries]
|
||||
at (8.45,-1.10) {cutoff $t_n$};
|
||||
\node[note,anchor=west] at (8.85,-1.62)
|
||||
{replay completed payloads; recompute only\\the temporary tail; exclude future media};
|
||||
\node[anchor=east,text=teal] at (2.05,-2.31) {local evidence};
|
||||
\draw[draw=teal,line width=1.8pt] (7.65,-2.31) -- (8.45,-2.31);
|
||||
\node[note,anchor=west,text=teal] at (8.85,-2.31)
|
||||
{one-second causal window, refreshed every 200 ms};
|
||||
\node[anchor=east,text=navy] at (2.05,-2.86) {decision grid};
|
||||
\draw[draw=navy,line width=0.6pt] (2.3,-2.86) -- (8.45,-2.86);
|
||||
\foreach \x in {2.42,2.58,...,8.34} {
|
||||
\draw[draw=navy,line width=0.5pt] (\x,-2.95) -- (\x,-2.77);
|
||||
}
|
||||
|
||||
\node[auditbox,minimum width=42mm] (oracle) at (11.02,-2.78)
|
||||
{analysis only: past visible-person activity\\aggregate / per-person / delayed / shuffled};
|
||||
\draw[auditflow] (oracle.north) -- ++(0,0.32) -| (fusion.south);
|
||||
\node[note,text=purple] at (11.02,-3.35)
|
||||
{privileged information audit; never a deployable model input};
|
||||
\draw[draw=orange,fill=orangefill,line width=.65pt]
|
||||
(6.41,-1.84) rectangle (7.85,-1.39);
|
||||
\node[font=\sffamily\tiny,text=orange] at (7.13,-1.615) {observed tail};
|
||||
\draw[draw=orange,dashed,line width=.7pt] (7.85,-2.43)--(7.85,-1.12);
|
||||
\node[anchor=south,font=\sffamily\tiny\bfseries,text=orange]
|
||||
at (7.85,-1.12) {$t_n$};
|
||||
\node[hint,anchor=west] at (8.16,-1.62)
|
||||
{reuse completed payloads;\\recompute temporary tail only};
|
||||
\node[anchor=east,text=teal] at (1.9,-2.33) {local 1 s};
|
||||
\draw[draw=teal,line width=2pt] (7.14,-2.33)--(7.85,-2.33);
|
||||
\node[hint,anchor=west,text=teal] at (8.16,-2.33)
|
||||
{recent evidence refreshed every 200 ms};
|
||||
\node[hint,anchor=west] at (2.12,-2.74)
|
||||
{Privileged past-activity labels are evaluated separately as an information audit; they never enter this prediction path.};
|
||||
\end{tikzpicture}
|
||||
\end{document}
|
||||
|
||||
BIN
manuscript/targeted/figures/system_overview_onecolumn.pdf
Normal file
BIN
manuscript/targeted/figures/system_overview_onecolumn.pdf
Normal file
Binary file not shown.
50
manuscript/targeted/figures/system_overview_onecolumn.tex
Normal file
50
manuscript/targeted/figures/system_overview_onecolumn.tex
Normal file
@@ -0,0 +1,50 @@
|
||||
\documentclass[tikz,border=2pt]{standalone}
|
||||
\usetikzlibrary{arrows.meta,positioning}
|
||||
\definecolor{navy}{HTML}{19324A}
|
||||
\definecolor{bluefill}{HTML}{EAF1F7}
|
||||
\definecolor{teal}{HTML}{087F78}
|
||||
\definecolor{tealfill}{HTML}{E6F4F1}
|
||||
\definecolor{orange}{HTML}{B85D0F}
|
||||
\definecolor{orangefill}{HTML}{FFF0DF}
|
||||
\begin{document}
|
||||
\begin{tikzpicture}[
|
||||
font=\sffamily\small,
|
||||
>=Latex,
|
||||
box/.style={draw=navy,fill=bluefill,rounded corners=1.5pt,
|
||||
align=center,minimum height=7mm,inner sep=2pt},
|
||||
local/.style={box,draw=teal,fill=tealfill},
|
||||
flow/.style={-{Latex[length=1.6mm]},draw=navy,line width=.6pt}]
|
||||
\node[box,text width=20mm] (global) at (1.12,3.9)
|
||||
{Observed AV\\8-s maximum};
|
||||
\node[box,text width=22mm] (thinker) at (4.13,3.9)
|
||||
{Frozen Thinker\\readout $G$};
|
||||
\node[local,text width=20mm] (audio) at (1.12,2.9)
|
||||
{Recent 1-s\\mixed audio};
|
||||
\node[local,text width=22mm] (aenc) at (4.13,2.9)
|
||||
{Frozen encoder\\local $A$};
|
||||
\node[local,dashed,text width=20mm] (video) at (1.12,1.9)
|
||||
{Past person\\crops (optional)};
|
||||
\node[local,dashed,text width=22mm] (venc) at (4.13,1.9)
|
||||
{Frozen encoder\\pooled $V$};
|
||||
\node[box,text width=18mm] (head) at (7.05,2.9)
|
||||
{Validities +\\trained head};
|
||||
\draw[flow] (global) -- (thinker);
|
||||
\draw[flow,draw=teal] (audio) -- (aenc);
|
||||
\draw[flow,draw=teal,dashed] (video) -- (venc);
|
||||
\draw[flow] (thinker.east) -- ++(.18,0) |- (head.north);
|
||||
\draw[flow,draw=teal] (aenc) -- (head);
|
||||
\draw[flow,draw=teal,dashed] (venc.east) -- ++(.18,0) |- (head.south);
|
||||
\node[box,text width=32mm] (out) at (5.7,.88)
|
||||
{$10\times3$ future activity\\0.2--2.0-s horizons};
|
||||
\draw[flow] (head.south) -- (out.north);
|
||||
\draw[draw=navy,line width=.5pt] (0,.12) -- (8.25,.12);
|
||||
\node[anchor=west,text=navy] at (0,-.28) {Completed blocks};
|
||||
\foreach \x in {0,1.04,2.08} {
|
||||
\draw[draw=navy,fill=bluefill] (\x,-.93) rectangle ++(.96,.35);
|
||||
}
|
||||
\draw[draw=orange,fill=orangefill] (3.12,-.93) rectangle ++(.96,.35);
|
||||
\node[anchor=west,text=orange] at (4.25,-.76) {Observed tail};
|
||||
\node[anchor=west,text=navy] at (0,-1.33)
|
||||
{Reuse completed payloads; query a temporary tail at $t_n$.};
|
||||
\end{tikzpicture}
|
||||
\end{document}
|
||||
BIN
manuscript/targeted/figures/task_overview.pdf
Normal file
BIN
manuscript/targeted/figures/task_overview.pdf
Normal file
Binary file not shown.
39
manuscript/targeted/figures/task_overview.tex
Normal file
39
manuscript/targeted/figures/task_overview.tex
Normal file
@@ -0,0 +1,39 @@
|
||||
\documentclass[tikz,border=2pt]{standalone}
|
||||
\usepackage{amsmath}
|
||||
\usetikzlibrary{arrows.meta,calc}
|
||||
\definecolor{navy}{HTML}{19324A}
|
||||
\definecolor{teal}{HTML}{087F78}
|
||||
\definecolor{orange}{HTML}{B85D0F}
|
||||
\definecolor{light}{HTML}{EAF1F7}
|
||||
\definecolor{warm}{HTML}{FFF0DF}
|
||||
\begin{document}
|
||||
\begin{tikzpicture}[font=\sffamily\scriptsize,>=Latex]
|
||||
\node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy] at (0,2.48)
|
||||
{Predict the invisible wearer's future activity};
|
||||
\draw[draw=navy,fill=light,rounded corners=2pt] (0,1.25) rectangle (2.55,1.95);
|
||||
\node[align=center,text=navy] at (1.275,1.60)
|
||||
{camera + mixed audio\\visible people, unseen wearer};
|
||||
\draw[-{Latex[length=1.7mm]},draw=navy,line width=.7pt]
|
||||
(2.65,1.60)--(3.25,1.60);
|
||||
\draw[draw=navy,line width=.7pt] (3.32,1.28)--(3.32,1.94);
|
||||
\node[anchor=south,font=\sffamily\tiny\bfseries,text=navy] at (3.32,1.97)
|
||||
{decision $t_n$};
|
||||
\draw[draw=orange,fill=warm,rounded corners=2pt] (4.02,1.25) rectangle (7.6,1.95);
|
||||
\node[align=center,text=orange] at (5.81,1.60)
|
||||
{10 future states, every 0.2 s\\background / wearer / others};
|
||||
\draw[draw=navy,line width=.55pt] (.35,.66)--(7.55,.66);
|
||||
\foreach \x in {.35,1.08,1.81,2.54,3.27,4.00,4.73,5.46,6.19,6.92,7.55}
|
||||
{\draw[draw=navy,line width=.5pt] (\x,.57)--(\x,.75);}
|
||||
\draw[draw=teal,line width=2pt] (.35,.94)--(3.27,.94);
|
||||
\node[anchor=east,font=\sffamily\tiny,text=teal] at (.25,.94) {input};
|
||||
\draw[draw=orange,line width=2pt] (3.27,.94)--(7.55,.94);
|
||||
\node[anchor=west,font=\sffamily\tiny,text=orange] at (7.65,.94) {target};
|
||||
\node[font=\sffamily\tiny,text=navy] at (3.27,.36) {$t_n$};
|
||||
\node[font=\sffamily\tiny,text=navy] at (4.0,.36) {$+.2$ s};
|
||||
\node[font=\sffamily\tiny,text=navy] at (7.55,.36) {$+2.0$ s};
|
||||
\node[anchor=west,font=\sffamily\tiny,text=navy] at (.35,.02)
|
||||
{ENTRY-like: wearer silent $\to$ active};
|
||||
\node[anchor=west,font=\sffamily\tiny,text=navy] at (4.0,.02)
|
||||
{YIELD-like: wearer active $\to$ silent};
|
||||
\end{tikzpicture}
|
||||
\end{document}
|
||||
@@ -1,5 +1,5 @@
|
||||
my $path_separator = $^O eq 'MSWin32' ? ';' : ':';
|
||||
my $template_dir = '../ICASSP2026_Paper_Templates';
|
||||
my $template_dir = '../ICASSP2027_Paper_Templates';
|
||||
|
||||
$ENV{'TEXINPUTS'} =
|
||||
$template_dir . $path_separator . ($ENV{'TEXINPUTS'} // '');
|
||||
|
||||
@@ -1,42 +1,46 @@
|
||||
% Acceptance-target ICASSP 2027 draft grounded only in archived measurements.
|
||||
% The official 2027 author kit should replace the 2026-compatible style assets
|
||||
% before submission. Four technical pages plus an optional reference-only page.
|
||||
% Uses the repository's ICASSP 2027 author kit.
|
||||
% Four technical pages plus an optional reference-only page.
|
||||
\documentclass{article}
|
||||
\usepackage{spconf}
|
||||
\usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor}
|
||||
\usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor,float}
|
||||
\hypersetup{hidelinks}
|
||||
\newcommand{\vect}[1]{\boldsymbol{#1}}
|
||||
\newcommand{\G}{\mathrm{G}}
|
||||
\newcommand{\A}{\mathrm{A}}
|
||||
\newcommand{\V}{\mathrm{V}}
|
||||
\setlength{\emergencystretch}{2em}
|
||||
\setcounter{topnumber}{5}
|
||||
\setcounter{dbltopnumber}{5}
|
||||
\renewcommand{\topfraction}{0.95}
|
||||
\renewcommand{\dbltopfraction}{0.95}
|
||||
\renewcommand{\textfraction}{0.05}
|
||||
\renewcommand{\floatpagefraction}{0.75}
|
||||
\renewcommand{\dblfloatpagefraction}{0.75}
|
||||
\title{Does an Omni Model Know When the Wearer Will Speak?\\
|
||||
A Causal Global--Local Audit of Egocentric Speech Anticipation}
|
||||
\name{Authors withheld in this working draft}
|
||||
\address{}
|
||||
\name{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu,
|
||||
Xiaoju Dai, Li Zhang, and Guorun Chen}
|
||||
\address{School of Computer Science, Example University, Shanghai, China}
|
||||
|
||||
\begin{document}
|
||||
\ninept
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
An egocentric conversational agent must anticipate the future speech of an
|
||||
invisible camera wearer from a causal multiparty audio--video stream. We ask
|
||||
whether a frozen general-purpose Omni state exposes the sub-second evidence
|
||||
needed by this task. Our cutoff-aligned model combines an 8-s Qwen2.5-Omni
|
||||
context with the most recent 1-s audio and predicts three activity channels at
|
||||
ten horizons from 0.2 to 2.0~s. In a single-seed EasyCom screen, this model reaches
|
||||
81.72\% mAP at 0.2~s and 65.11\% averaged over horizons, compared with
|
||||
74.20\% and 61.19\% for released EgoSpeak. The gain is larger for wearer
|
||||
offset anticipation (+7.23 points) than onset anticipation (+1.47). Pooled
|
||||
visual features add only 0.29 mean-mAP points. A separate three-seed,
|
||||
strictly causal information audit explains this gap: privileged past activity
|
||||
of visible participants adds 4.83 points at 0.2~s and 1.77 mean-mAP points,
|
||||
whereas visibility, one-second-delayed activity, and identity shuffling show
|
||||
that timeliness matters more than identity continuity. The results isolate an
|
||||
evidence-access bottleneck: recent acoustic evidence is already useful, while
|
||||
current generic visual pooling does not recover the available visible-floor
|
||||
signal.
|
||||
We test whether a frozen Omni readout exposes sub-second evidence for
|
||||
forecasting an invisible camera wearer's speech from causal multiparty
|
||||
audio--video. Our 5-Hz model combines 8-s Qwen2.5-Omni context and recent
|
||||
1-s audio to predict three activity channels at ten 0.2--2.0-s horizons.
|
||||
On EasyCom, a seed-0 screen reaches 81.72\% mAP at 0.2~s and 65.11\%
|
||||
mean mAP, versus 74.20/61.19\% for released EgoSpeak. Wearer offset AP
|
||||
improves by 7.23 points, onset AP by 1.47. Pooled vision adds only 0.29
|
||||
mean-mAP points. A three-seed causal information audit finds that privileged
|
||||
past visible-person activity adds 4.83 points at 0.2~s and 1.77 mean-mAP
|
||||
points; visibility and delayed activity add little. A separate no-box
|
||||
full-frame screen reaches 82.38/65.54\%, but unmatched training and head
|
||||
capacity prevent visual attribution. Timely evidence access, not semantic
|
||||
floor understanding, remains the supported conclusion.
|
||||
\end{abstract}
|
||||
|
||||
\begin{keywords}
|
||||
@@ -51,8 +55,18 @@ Conversational agents must decide not only \emph{what} to say, but also
|
||||
\emph{when} to participate. This decision is particularly asymmetric in an
|
||||
egocentric stream: the camera wearer is normally invisible, yet the model must
|
||||
forecast that wearer's behavior from mixed audio and observations of other
|
||||
people. A useful predictor must update before a turn boundary is known and
|
||||
must not use media from after its decision cutoff.
|
||||
people. As Fig.~\ref{fig:task} illustrates, prediction precedes any known
|
||||
turn boundary and excludes media after the decision cutoff.
|
||||
|
||||
\begin{figure}[H]
|
||||
\centering
|
||||
\includegraphics[width=\columnwidth]{figures/task_overview.pdf}
|
||||
\caption{Task, not an observed event example. At each 5-Hz decision cutoff,
|
||||
only the past audio--video prefix is input; ten future activity states are
|
||||
targets. ENTRY-like and YIELD-like refer to wearer activity transitions,
|
||||
not semantic floor ownership.}
|
||||
\label{fig:task}
|
||||
\end{figure}
|
||||
|
||||
Voice Activity Projection (VAP) casts turn taking as continuous prediction of
|
||||
future activity \cite{ekstedt2022vap}. EgoSpeak brings this objective to
|
||||
@@ -74,28 +88,22 @@ state $\G$ from explicit recent evidence $\A$ and optional $\V$, and treat
|
||||
their value as an empirical question rather than assuming that a larger
|
||||
backbone solves temporal access.
|
||||
|
||||
This paper makes three contributions. First, we formulate a strictly causal
|
||||
global--local audit for invisible-wearer activity forecasting, with a common
|
||||
5-Hz cutoff and ten future horizons. Second, we realize the audit with a
|
||||
bounded Omni stream that combines completed history with a temporary observed
|
||||
tail and a separate one-second audio path. Third, we distinguish deployable
|
||||
feature gains from an analysis-only visible-activity oracle. This reveals a
|
||||
specific negative result---generic pooled or current-frame person features do
|
||||
not recover most of the available local-floor information---and identifies
|
||||
timely activity estimation, rather than identity continuity, as the next
|
||||
modeling target.
|
||||
We contribute a cutoff-aligned global--local audit, a bounded Omni stream
|
||||
with observed-tail refresh and recent audio, and a separation of deployable
|
||||
features from an analysis-only visible-activity oracle. Generic pooled or
|
||||
current-frame person features recover little of the available activity signal;
|
||||
timely activity estimation is the next modeling target.
|
||||
|
||||
\begin{figure*}[t]
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\includegraphics[width=0.97\textwidth]{figures/system_overview.pdf}
|
||||
\caption{Global--local causal audit. At each 200-ms cutoff, the prediction
|
||||
path fuses an integrated Omni state with explicit recent audio and optional
|
||||
person-crop vision. Completed 2-s blocks are reused while only the observed
|
||||
partial tail is temporarily recomputed. The purple branch is a privileged
|
||||
past-activity audit used to test information value; it is never a deployable
|
||||
input and contains no future activity.}
|
||||
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
|
||||
\caption{Deployable global--local path and bounded stream. At each 200-ms
|
||||
cutoff, the head fuses the frozen integrated Omni state with explicit recent
|
||||
audio and optional pooled person-crop vision. Completed 2-s payloads are
|
||||
reused; only the observed partial tail is temporarily recomputed. Privileged
|
||||
past-activity audit labels are kept outside this graph and are not inputs.}
|
||||
\label{fig:overview}
|
||||
\end{figure*}
|
||||
\end{figure}
|
||||
|
||||
\section{Method}
|
||||
\label{sec:method}
|
||||
@@ -137,16 +145,13 @@ is frozen-backbone adaptation rather than end-to-end training.
|
||||
|
||||
\subsection{Cutoff-Aligned Streaming}
|
||||
|
||||
The global stream uses source-aligned 2-s blocks and an 8-s maximum
|
||||
presentation budget. After warm-up, three completed blocks and the observed
|
||||
part of the current block are available; the exact span varies with block
|
||||
phase. Completed modality payloads are reused when the bounded language-model
|
||||
cache is rebuilt. At every cutoff, the incomplete current block is cropped at
|
||||
$t_n$, appended to a temporary cache, and queried for $\vect g_n$. Neither
|
||||
the temporary tail nor the query is committed. At a block boundary the
|
||||
completed payload is committed once and a new tail begins. Sessions and
|
||||
source discontinuities reset all caches. Thus the logical state is refreshed
|
||||
at 5~Hz without backfilling a completed block into earlier rows.
|
||||
Figure~\ref{fig:overview} uses source-aligned 2-s blocks under an 8-s
|
||||
presentation cap. After warm-up, three completed blocks and the observed
|
||||
current tail are available; span varies with block phase. Completed
|
||||
modality payloads are reused. At each cutoff, the tail is cropped at $t_n$
|
||||
and queried in a temporary cache, without committing it. Only completed
|
||||
blocks are committed; session or source discontinuities reset caches. This
|
||||
refreshes the logical state at 5~Hz without backfilling earlier rows.
|
||||
|
||||
\subsection{Visible-Activity Information Audit}
|
||||
|
||||
@@ -161,6 +166,148 @@ second, and (v) one OR-pooled aggregate-visible history. These conditions
|
||||
never expose future activity, but their speech labels make them privileged
|
||||
diagnostics, not deployable features or upper bounds.
|
||||
|
||||
\begin{table*}[t]
|
||||
\caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B:
|
||||
three-seed privileged past-activity audit, which is not deployable. Mean is
|
||||
ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events.
|
||||
F3 is a no-box, two-frame causal full-frame screen; it and the historical
|
||||
deployable rows are exploratory seed-0 runs with unmatched head/training
|
||||
details. F1 has three seeds. Dashes are unavailable, not zero.}
|
||||
\label{tab:main}
|
||||
\centering
|
||||
\normalsize
|
||||
\renewcommand{\arraystretch}{0.87}
|
||||
\setlength{\tabcolsep}{5.0pt}
|
||||
\begin{tabular}{@{}lccccc@{}}
|
||||
\toprule
|
||||
Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
|
||||
& Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
|
||||
\midrule
|
||||
EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
|
||||
Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
|
||||
Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
|
||||
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
|
||||
Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086
|
||||
& 65.400 & 26.019 & 45.371 \\\\
|
||||
No-box full-frame F3 & 82.380 & -- & 65.540 & -- & -- \\\\
|
||||
Current-frame person set F1 & $81.777{\pm}0.156$ & --
|
||||
& $64.951{\pm}0.095$ & -- & -- \\\\
|
||||
Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\
|
||||
Past-delayed $\A$ (L3) & -- & -- & -- & -- & -- \\\\
|
||||
Current-state persistence (L4) & -- & -- & -- & -- & -- \\\\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\par\smallskip
|
||||
\emph{Panel B: privileged past-activity audit (mean $\pm$ sample SD).}\par\smallskip
|
||||
\normalsize
|
||||
\setlength{\tabcolsep}{3.2pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
|
||||
\midrule
|
||||
None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
|
||||
Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
|
||||
Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
|
||||
Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
|
||||
Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
|
||||
Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
|
||||
& $\mathbf{66.951{\pm}.022}$ \\\\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table*}
|
||||
|
||||
\begin{table*}[t]
|
||||
\caption{Chunk and context audit. Panel A is an unmeasured matched T/C
|
||||
matrix: each row pairs $\G$ and $\G+\A$ on common valid cutoffs. Panel B is
|
||||
a separate exploratory seed-0 global-only screen (10 epochs, 8-s cap,
|
||||
2-FPS video); its rows are not entries in Panel A. Dashes are pending.}
|
||||
\label{tab:stream_context}
|
||||
\centering\normalsize
|
||||
\renewcommand{\arraystretch}{0.87}
|
||||
\setlength{\tabcolsep}{4pt}
|
||||
\begin{tabular}{@{}llccccc@{}}
|
||||
\toprule
|
||||
Group & Refresh / cap & $\G$ mAP & $\G+\A$ mAP & $\Delta$ mAP
|
||||
& $\Delta$ On & $\Delta$ Off \\
|
||||
\midrule
|
||||
T & Completed-only, 8 s & -- & -- & -- & -- & -- \\
|
||||
T & Observed-tail, 8 s & -- & -- & -- & -- & -- \\
|
||||
\midrule
|
||||
C & Tail-aware, 2 s & -- & -- & -- & -- & -- \\
|
||||
C & Tail-aware, 4 s & -- & -- & -- & -- & -- \\
|
||||
C & Tail-aware, 6 s & -- & -- & -- & -- & -- \\
|
||||
C & Tail-aware, 8 s & -- & -- & -- & -- & -- \\
|
||||
C & Tail-aware, 10 s & -- & -- & -- & -- & -- \\
|
||||
C & Tail-aware, 12 s & -- & -- & -- & -- & -- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\par\smallskip
|
||||
\emph{Panel B: reported 200-ms chunk screen; dense mAP (\%).}\par\smallskip
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Global feature construction & 0.2-s mAP & Mean mAP & Processor calls \\
|
||||
\midrule
|
||||
Native 2-s completed hold & 59.62 & 55.19 & 9,539 \\
|
||||
Non-native 200-ms commits & 67.05 & 57.84 & 95,453 \\
|
||||
Completed blocks + observed tail & 70.85 & 59.17 & -- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table*}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{B result slots: paired within-backbone local-audio gains (unmeasured).
|
||||
MiniCPM requires a matched readout/refresh/history audit before comparison.}
|
||||
\label{tab:backbone}
|
||||
\centering\normalsize
|
||||
\setlength{\tabcolsep}{3pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Backbone & $\G$ & $\G+\A$ & $\Delta$ \\
|
||||
\midrule
|
||||
Qwen Omni 3B & -- & -- & -- \\
|
||||
Qwen Omni 7B & -- & -- & -- \\
|
||||
MiniCPM-o 4.5 & -- & -- & -- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{H result slots: predeclared activity slices (unmeasured). Report
|
||||
positive/negative counts and per-session scores; AP is undefined for a
|
||||
pure-negative slice, which instead uses calibrated FPR.}
|
||||
\label{tab:hard_slices}
|
||||
\centering\normalsize
|
||||
\setlength{\tabcolsep}{2.5pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Slice & $N_+/N_-$ & AP & FPR \\
|
||||
\midrule
|
||||
Wearer onset & -- & -- & -- \\
|
||||
Wearer offset & -- & -- & -- \\
|
||||
Other pause (negative) & -- & n/a & -- \\
|
||||
Overlap (negative) & -- & n/a & -- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{S result slots: full-path cost on the same hardware (unmeasured).
|
||||
Include I/O and block transitions; logical 5 Hz is not measured throughput.}
|
||||
\label{tab:efficiency}
|
||||
\centering\normalsize
|
||||
\setlength{\tabcolsep}{3pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Stage & P50 ms & P95 ms & Peak GB \\
|
||||
\midrule
|
||||
Completed block & -- & -- & -- \\
|
||||
Observed tail & -- & -- & -- \\
|
||||
Local path + head & -- & -- & -- \\
|
||||
Full path & -- & -- & -- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Experiments}
|
||||
\label{sec:experiments}
|
||||
|
||||
@@ -191,36 +338,16 @@ already sufficient. Because that screen used Session~12 model selection
|
||||
whereas the historical pooled row did not, their difference is diagnostic,
|
||||
not a matched causal attribution.
|
||||
|
||||
\textbf{Six pre-specified control families.}
|
||||
We freeze six pending control families: (L) capacity and recency, (T) tail
|
||||
refresh crossed with local audio, (C) six context caps, (B) paired backbone
|
||||
transfer, (H/Q) error slices and timelines, and (S) full-path efficiency.
|
||||
They share the official split and physical cutoffs; the 8-s T/C row reuses
|
||||
one run. The source retains unmeasured hypotheses and falsifiers.
|
||||
|
||||
\section{Results}
|
||||
\label{sec:results}
|
||||
|
||||
\begin{table*}[t]
|
||||
\caption{EasyCom Sessions~1--3 results (\%). $\mathrm{mAP}_{.2}$ and
|
||||
$\mathrm{WAP}_{.2}$ are dense 0.2-s metrics; Mean averages dense mAP over ten
|
||||
horizons. On/Off are raw wearer endpoint AP, not semantic event scores.
|
||||
Deployable rows are archived single-seed screens; $\pm$ is shown only for the
|
||||
three-seed F1 screen. Dashes denote unavailable measurements.}
|
||||
\label{tab:main}
|
||||
\centering
|
||||
\scriptsize
|
||||
\setlength{\tabcolsep}{5.0pt}
|
||||
\begin{tabular}{@{}lccccc@{}}
|
||||
\toprule
|
||||
Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
|
||||
& Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
|
||||
\midrule
|
||||
EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
|
||||
Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
|
||||
Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
|
||||
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
|
||||
Omni $\G+\A+$pooled-$\V$ & \textbf{82.027} & \textbf{85.086}
|
||||
& \textbf{65.400} & \textbf{26.019} & 45.371 \\\\
|
||||
Current-frame person set F1 & $81.777{\pm}0.156$ & --
|
||||
& $64.951{\pm}0.095$ & -- & -- \\\\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table*}
|
||||
|
||||
\subsection{Global Context and Recent Audio Are Complementary}
|
||||
|
||||
Table~\ref{tab:main} shows that $\A$ is strong at 0.2~s, while $\G$ is
|
||||
@@ -238,6 +365,70 @@ the local-only head, so this comparison demonstrates predictive
|
||||
complementarity, not that semantic reasoning alone causes the gain. A wider
|
||||
history-matched control is required for a representation-level causal claim.
|
||||
|
||||
% Retained as a planning aid; illustrative, unmeasured patterns do not belong
|
||||
% in the page-limited submission preview.
|
||||
\iffalse
|
||||
\begin{table*}[t]
|
||||
\caption{Six-group \textbf{illustrative scenario, not measured results}.
|
||||
These are conditional, pre-run interpretations for the pending matrix, not
|
||||
EasyCom scores. $\Delta$ denotes paired mean-mAP change on common valid
|
||||
cutoffs; signs and shape must be replaced by measured values even if contrary
|
||||
to the story. L/T/C/B require seeds 0--2 and per-session reporting.}
|
||||
\label{tab:scenario}
|
||||
\centering
|
||||
\normalsize
|
||||
\renewcommand{\arraystretch}{1.12}
|
||||
\setlength{\tabcolsep}{4.2pt}
|
||||
\begin{tabular}{@{}p{0.095\textwidth}p{0.29\textwidth}p{0.29\textwidth}p{0.235\textwidth}@{}}
|
||||
\toprule
|
||||
Group & Frozen comparison & Illustrative pattern \emph{if observed} & Falsifier / interpretation limit \\
|
||||
\midrule
|
||||
L1--4 & $\A,\G,\G+\A,\G+\A+\V$; matched-width $\G$;
|
||||
past-delayed local cues; current-state persistence
|
||||
& $\Delta(\G+\A-\G)>0$ after width matching; delayed $\A$ loses
|
||||
near-horizon benefit; persistence does not recover the gain
|
||||
& Width-only or persistence matches fusion, or delay does not hurt:
|
||||
do not attribute gain to timely distinct evidence. \\
|
||||
\addlinespace[2.5pt]
|
||||
T1--2 & 2-s completed-only vs 200-ms observed-tail refresh,
|
||||
crossed with $\A$ on/off; causal replay and phase audit
|
||||
& Tail helps most without $\A$; local $\A$ still adds at each refresh;
|
||||
future-only media never changes prior readouts
|
||||
& No independent local gain under tail refresh implies redundancy;
|
||||
replay/leakage failure invalidates timing claims. \\
|
||||
\addlinespace[2.5pt]
|
||||
C1--2 & Genuine 2/4/6/8/10/12-s feature extraction; fixed 1-s $\A$;
|
||||
add $\G$-only at each cap for a window-stability claim
|
||||
& Fused performance plateaus around mid-range caps while
|
||||
$\G$-only remains more cap-sensitive
|
||||
& If long history substitutes for $\A$, local-evidence claim narrows;
|
||||
without C2, no robustness comparison is justified. \\
|
||||
\addlinespace[2.5pt]
|
||||
B0--2 & Paired $\G$ and $\G+\A$ on Qwen 3B/7B and, after causal
|
||||
interface audit, MiniCPM-o 4.5
|
||||
& Positive within-backbone paired $\Delta$ on all admissible rows;
|
||||
absolute backbone ranking is not inferred
|
||||
& Failed causal/history match is unsupported, not zero gain;
|
||||
one negative paired row defeats universal transfer. \\
|
||||
\addlinespace[2.5pt]
|
||||
H1/Q1 & Predefined onset/offset and other-pause/overlap slices;
|
||||
same-horizon, cutoff-marked case selection
|
||||
& Offset gains coexist with hard wearer-onset negatives; examples
|
||||
explain, rather than replace, counted slice statistics
|
||||
& Pure-negative slice AP is undefined; selected examples cannot
|
||||
establish reduced false-entry rate. \\
|
||||
\addlinespace[2.5pt]
|
||||
S1 & End-to-end p50/p95 latency and peak memory, including I/O,
|
||||
encoding, tail recomputation, local path, and boundaries
|
||||
& A feasible throughput--quality tradeoff can be plotted
|
||||
against 200-ms output cadence
|
||||
& If p95 exceeds cadence, describe logical 5-Hz prediction only,
|
||||
not real-time deployment. \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table*}
|
||||
\fi
|
||||
|
||||
\subsection{Generic Vision Does Not Recover the Available Signal}
|
||||
|
||||
Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
|
||||
@@ -245,32 +436,15 @@ Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
|
||||
averaged over horizons; onset/offset changes are +0.059/+0.120. A richer
|
||||
current-frame person-set model is stable across seeds but reaches lower mean
|
||||
mAP than the historical pooled row under a non-matched selection protocol.
|
||||
These results do not support a claim of learned speaker grounding.
|
||||
These results do not support a claim of learned speaker grounding. A separate
|
||||
seed-0 no-box full-frame F3 screen reaches 82.38\% at 0.2~s and 65.54\%
|
||||
mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same
|
||||
causal-valid dense protocol. F3 adds a two-frame 2-FPS token history and a
|
||||
shared GRU, while head capacity and training batch differ; raw boundary
|
||||
metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point
|
||||
difference does not isolate visual motion or floor behavior.
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Three-seed privileged-information audit (mean $\pm$ sample standard
|
||||
deviation, \%). Every row includes $\G+\A$; activity conditions use only
|
||||
past labels. Bold marks the best diagnostic, not a deployable model.}
|
||||
\label{tab:oracle}
|
||||
\centering
|
||||
\scriptsize
|
||||
\setlength{\tabcolsep}{3.2pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
|
||||
\midrule
|
||||
None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
|
||||
Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
|
||||
Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
|
||||
Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
|
||||
Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
|
||||
Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
|
||||
& $\mathbf{66.951{\pm}.022}$ \\\\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Table~\ref{tab:oracle} locates the missing information. Aggregate visible
|
||||
Table~\ref{tab:main}, panel B, locates the missing information. Aggregate visible
|
||||
activity improves over the audio baseline by 4.826 points at 0.2~s and 1.775
|
||||
mean-mAP points. Visibility alone changes mean mAP by only +0.040, and
|
||||
one-second-delayed activity by +0.164. Thus availability or extra parameters
|
||||
@@ -282,45 +456,73 @@ The majority of the oracle's 0.2-s macro gain comes from background and
|
||||
others; wearer AP rises by only 0.572. This is why neither the oracle nor a
|
||||
dense mAP gain establishes fewer false wearer entries.
|
||||
|
||||
\subsection{What the Audit Establishes}
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\fbox{\begin{minipage}[c][18mm][c]{0.94\columnwidth}
|
||||
\centering
|
||||
\textbf{Q1 case slots: examples pending}\\[2pt]
|
||||
Onset / other-pause hard negative \quad|\quad Offset / overlap\\
|
||||
Show source cutoff, observed activity and same-horizon wearer scores.
|
||||
\end{minipage}}
|
||||
\caption{Reserved H/Q case figure. Select examples by a frozen rule after
|
||||
the slice counts in Table~\ref{tab:hard_slices} are available. No case or
|
||||
behavioral improvement is asserted by this placeholder.}
|
||||
\label{fig:hq_cases}
|
||||
\end{figure}
|
||||
|
||||
Three conclusions survive the controls. First, a general-purpose global
|
||||
state and a one-second acoustic readout provide different useful timescales.
|
||||
Second, generic visual pooling has not converted visible people into reliable
|
||||
local floor evidence. Third, perfect timely aggregate activity contains
|
||||
recoverable information, whereas identity contributes little under the
|
||||
current target. A dedicated causal active-speaker front end---designed for
|
||||
mouth motion, blur, and short audio--visual synchrony as in egocentric ASD
|
||||
\cite{huh2025egoasd}---is consequently a better-motivated next step than a
|
||||
larger identity-aware fusion module.
|
||||
\subsection{Pending Controls and Limits}
|
||||
|
||||
The evidence remains bounded. EasyCom has only three official test sessions;
|
||||
the deployable matrix is single seed and has been examined repeatedly. The
|
||||
oracle uses labels and is not a performance upper bound. Provided boxes omit
|
||||
online detection cost, and logical 5-Hz output is not a measured throughput
|
||||
claim. Finally, aggregate others cannot supply addressee or semantic
|
||||
interruption ground truth. Confirmatory behavioral claims require a frozen
|
||||
training-side operating point, competing-risk event attribution, and
|
||||
multi-seed matched controls.
|
||||
Tables~\ref{tab:stream_context}--\ref{tab:efficiency} reserve result slots
|
||||
for T/C, B, H, and S; Fig.~\ref{fig:hq_cases} reserves Q cases. Panel B of
|
||||
Table~\ref{tab:stream_context} separately reports a global-only chunk screen:
|
||||
the 200-ms commits beat 2-s hold after startup exclusion but require about
|
||||
ten times as many processor calls across 12 sessions. Freshness and chunk
|
||||
construction both change, and the older tail bank is not extraction-matched;
|
||||
these values do not fill the pending T/C factorial or establish real-time
|
||||
throughput. L requires a
|
||||
trained width-matched global control and past-delayed features on identical
|
||||
valid pairs; T crosses tail refresh with local audio; C re-extracts each
|
||||
causal context cap and pairs it with a global-only run. B compares the
|
||||
within-backbone audio increment only after readout and history audits.
|
||||
H/Q reports slice counts and cutoff-marked examples; S measures full-path
|
||||
latency and memory. The oracle uses privileged labels, provided boxes omit
|
||||
online detection, and aggregate others cannot establish addressee or
|
||||
semantic floor ownership.
|
||||
|
||||
\subsection{Decision Rules and Reproducibility}
|
||||
|
||||
The unit of comparison is the same physical cutoff and future target time,
|
||||
not an arbitrary feature row. For every new condition we will archive
|
||||
input-source timestamps, common valid-pair counts, seeds, per-session AP,
|
||||
trainable parameter counts, processor/model revisions, checkpoint hashes,
|
||||
and prediction scores. Frames and horizons are correlated, not independent
|
||||
replicates.
|
||||
|
||||
We retain the fixed four-epoch recipe for the matched matrix. Any loss
|
||||
weight, threshold, context selection, or case-selection rule is frozen
|
||||
using training-side evidence before looking at Sessions~1--3. Because
|
||||
these test sessions have already informed this research program, even
|
||||
new seed runs remain exploratory rather than restoring an unseen test set.
|
||||
The raw endpoint AP in Table~\ref{tab:main} may motivate a behavioral
|
||||
hypothesis, but an ENTRY/YIELD error-rate claim additionally requires
|
||||
one-event-one-vote attribution with response windows censored at the next
|
||||
other onset and yield windows at the next other offset. Until that
|
||||
competing-risk evaluator passes real-timeline audit, we report only dense
|
||||
forecasting and endpoint anticipation.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
We presented a strictly causal global--local audit of egocentric wearer speech
|
||||
anticipation. Frozen Omni context and recent audio are strongly complementary,
|
||||
especially for future wearer cessation. In contrast, pooled or current-frame
|
||||
person features do not recover the local-floor signal revealed by a causal
|
||||
visible-activity oracle. The actionable conclusion is deliberately narrow:
|
||||
the next bottleneck is timely, learnable local activity evidence, not simply
|
||||
more multimodal capacity or persistent identity. This distinction provides a
|
||||
clearer target for future egocentric conversational agents and prevents dense
|
||||
forecasting gains from being mistaken for semantic floor control.
|
||||
In this strictly causal audit, frozen Omni context and recent audio improve
|
||||
wearer activity forecasting, especially future speech cessation. Generic
|
||||
pooled vision does not recover the timely visible-activity signal exposed by
|
||||
the analysis-only oracle. Matched controls and censored event evaluation
|
||||
remain necessary before claiming improved floor behavior.
|
||||
|
||||
\section{Acknowledgment}
|
||||
OpenAI Codex assisted with substantive drafting and figure layout in the
|
||||
abstract, introduction, and presentation of the archived analysis; the
|
||||
authors remain responsible for verification, experimental claims, and the
|
||||
final manuscript.
|
||||
OpenAI Codex assisted with drafting and figure layout; the authors remain
|
||||
responsible for all experimental claims and final content.
|
||||
|
||||
\clearpage
|
||||
\bibliographystyle{IEEEbib}
|
||||
\bibliography{references}
|
||||
\end{document}
|
||||
|
||||
Reference in New Issue
Block a user