From 80f270a30a319042992fa7ed7ebafaca8edfd3eb Mon Sep 17 00:00:00 2001 From: Xin Wang Date: Thu, 24 Sep 2026 18:38:51 +0800 Subject: [PATCH] Restructure targeted comparison tables and method names --- manuscript/targeted/main.tex | 68 +++++++++++++++----------- manuscript/targeted/tables_compact.tex | 45 +++++++++++------ 2 files changed, 71 insertions(+), 42 deletions(-) diff --git a/manuscript/targeted/main.tex b/manuscript/targeted/main.tex index 4371b42..248f39b 100644 --- a/manuscript/targeted/main.tex +++ b/manuscript/targeted/main.tex @@ -34,7 +34,7 @@ models and streaming Omni assistants address related problems, but the cutoff-level availability of local evidence in an integrated Omni state is underexplored for this wearer-centered task. We examine two interfaces: short-window encoder evidence (SWE) supplies recent audio and optional -person-crop vision directly to a frozen-Omni forecasting head; partial-tail +causal visual evidence directly to a frozen-Omni forecasting head; partial-tail refresh (PTR) queries only the observed prefix of the current audiovisual block at each 200-ms cutoff. On EasyCom, the exploratory global-plus-audio screen reaches 81.72\% mAP at 0.2~s and 65.11\% over ten horizons, versus @@ -123,7 +123,7 @@ evidence at each causal cutoff (Fig.~\ref{fig:overview}). EasyCom screens suggest that global context and local audio are complementary; gains from pooled vision and isolated PTR effects remain unproven. This study makes three contributions: (1) short-window encoder evidence (SWE) supplies recent -audio and optional person-crop vision to a frozen-Omni forecasting head; +audio and optional visual encoder evidence to a frozen-Omni forecasting head; (2) partial-tail refresh (PTR) queries the observed current-block prefix without committing it to persistent history; and (3) detailed EasyCom analyses accompany a specified Ego4D transfer protocol. Matched PTR, @@ -133,7 +133,7 @@ Ego4D scores, and event behavior remain pending. \centering \includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf} \caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context -with recent audio and optional pooled person-crop evidence in the forecasting +with recent audio and optional local visual evidence in the forecasting head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only the observed current-block tail at each 200-ms cutoff. The tail is not committed to persistent history.} @@ -212,30 +212,39 @@ pending, so PTR is a causal access rule rather than a demonstrated gain. \subsection{Short-Window Feature Extraction} We define every local observation by its prediction cutoff, before applying -any encoder-specific preprocessing. The default audio branch encodes the +any encoder-specific preprocessing. We denote the default Qwen-audio branch +by $\A$: it encodes the mixed waveform in $[t_n-1\,\mathrm{s},t_n)$ with the frozen Qwen2.5-Omni -audio encoder and averages valid output tokens. A wav2vec~2.0 convolutional -audio representation, as used by EgoSpeak \cite{kim2025egospeak}, is an -encoder-choice control; it must use the same one-second input interval and +audio encoder and averages valid output tokens. We denote by $\A_{\rm E}$ +the wav2vec~2.0 convolutional representation used by EgoSpeak +\cite{kim2025egospeak}. This encoder-choice control must use the same +one-second input interval and forecasting head before differences can be attributed to the encoder. It is not the source of the reported $\G+\A$ result. -For vision, a single-frame variant uses the latest frame strictly before -$t_n$; the pooled person-crop variant averages frozen Qwen vision tokens +We denote three visual branches by \emph{Pool}, \emph{SVA}, and \emph{TSVA}. +Pool averages frozen Qwen vision tokens within provided boxes, across visible people, and over valid 2-FPS frames in the preceding second. These pooled features summarize appearance but discard person identity and explicit motion. Boxes are dataset-provided, not predictions from an evaluated online detector. Two no-box full-frame -alternatives retain the post-merger spatial-token grid: F2 lets the -global--audio query attend to the latest causal frame; F3 first encodes up +alternatives retain the post-merger spatial-token grid. Spatial visual +attention (SVA) lets the global--audio query attend to the latest frame +strictly before $t_n$; temporal spatial visual attention (TSVA) first encodes up to two observations at each matching grid location in chronological order -with a shared GRU, then applies the same query to the resulting grid. F3 -links image coordinates, not people or tracks. Its difference from F2 +with a shared GRU, then applies the same query to the resulting grid. TSVA +links image coordinates, not people or tracks. Its difference from SVA confounds additional history and temporal capacity, so it cannot alone establish a motion or speaker-grounding mechanism. Empty or unavailable local evidence is indicated by the validity masks, never filled with a future observation. +The shorthand $\G$, $\G+\A$, and $\G+\A+$SVA identifies the three Omni +rows in the cross-dataset table; $\G+\A+$Pool, $\G+\A+$TSVA, and +$\G+\A_{\rm E}$ isolate visual or audio choices. Released EgoSpeak A+V +is an external baseline, not an encoder choice within our frozen-Omni head. +An unmeasured label specifies a comparison, not a result. + As a distinct, privileged short-window feature, we also derive visible non-wearer speech activity from transcript annotations \emph{before} each cutoff. Each visible participant contributes a two-second binary history @@ -253,7 +262,7 @@ or a performance upper bound. \caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B: three-seed privileged past-activity audit, which is not deployable. Mean is ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events. -F3 is a no-box, two-frame causal full-frame screen; it and the historical +TSVA is a no-box, two-frame causal full-frame screen; it and the historical deployable rows are exploratory seed-0 runs with unmatched head/training details. F1 has three seeds. Dashes are unavailable, not zero.} \label{tab:main} @@ -272,7 +281,7 @@ Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\ Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\ Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086 & 65.400 & 26.019 & 45.371 \\\\ -No-box full-frame F3 & 82.380 & -- & 65.540 & -- & -- \\\\ +No-box full-frame TSVA & 82.380 & -- & 65.540 & -- & -- \\\\ Current-frame person set F1 & $81.777{\pm}0.156$ & -- & $64.951{\pm}0.095$ & -- & -- \\\\ Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\ @@ -425,12 +434,13 @@ threshold-free quantities average ten horizons. They are not one-event-one- vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding. \textbf{Comparisons.} -We evaluate released EgoSpeak MiniROADA A+V; recent audio $\A$; global Omni -$\G$; $\G+\A$; and $\G+\A+$pooled-$\V$. A three-seed current-frame -frame-set attention screen tests whether a richer person aggregation is -already sufficient. Because that screen used Session~12 model selection -whereas the historical pooled row did not, their difference is diagnostic, -not a matched causal attribution. +The cross-dataset main matrix contains released EgoSpeak A+V and the three +Omni variants defined in Sec.~\ref{sec:method}. Missing SVA and Ego4D +cells remain blank until measured. EasyCom-only encoder choices serve as +diagnostic ablations rather than extra cross-dataset main rows. A three-seed +current-frame person-set attention +screen used Session~12 model selection whereas the historical pooled row did +not; their difference is not a matched causal attribution. \textbf{Pre-specified analyses.} We freeze six control families: (L) capacity and recency, (T) tail @@ -444,10 +454,11 @@ one run. The source retains unmeasured hypotheses and falsifiers. \subsection{EasyCom: Local Encoder Evidence} -Table~\ref{tab:main} shows that recent encoder-side audio $\A$ is strong at -0.2~s while integrated $\G$ is slightly stronger on the ten-horizon mean. -Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 over -$\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at +Table~\ref{tab:main} anchors the measured EasyCom comparison. In a separate +audio-only screen, recent encoder-side $\A$ reaches 79.69\% at 0.2~s and +60.51\% mean mAP; integrated $\G$ is slightly stronger on the ten-horizon +mean. Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 +over $\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at 0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23 points, onset AP by only 1.47: dense gains do not imply equal improvement at both activity boundaries. @@ -524,15 +535,16 @@ not real-time deployment. \\ \subsection{EasyCom: Visual Evidence and Ablations} -Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and +Table~\ref{tab:visual} separates visual screens from the cross-dataset main +matrix. Adding Pool to $\G+\A$ produces only +0.304 points at 0.2~s and +0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when averaged over horizons; onset/offset changes are +0.059/+0.120. A richer current-frame person-set model is stable across seeds but reaches lower mean mAP than the historical pooled row under a non-matched selection protocol. These results do not support a claim of learned speaker grounding. A separate -seed-0 no-box full-frame F3 screen reaches 82.38\% at 0.2~s and 65.54\% +seed-0 no-box full-frame TSVA screen reaches 82.38\% at 0.2~s and 65.54\% mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same -causal-valid dense protocol. F3 adds a two-frame 2-FPS token history and a +causal-valid dense protocol. TSVA adds a two-frame 2-FPS token history and a shared GRU, while head capacity and training batch differ; raw boundary metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point difference does not isolate visual motion or floor behavior. diff --git a/manuscript/targeted/tables_compact.tex b/manuscript/targeted/tables_compact.tex index 03f13f3..416a9fb 100644 --- a/manuscript/targeted/tables_compact.tex +++ b/manuscript/targeted/tables_compact.tex @@ -1,13 +1,11 @@ % Active result tables. A dash denotes an unreported or unmeasured value. \begin{table*}[t] -\caption{Dense future-activity mAP (\%) by prediction horizon. EasyCom -scores use Sessions~1--3; the complete ten-horizon curves are reported for -EgoSpeak, $\G+\A$, and F3. Other archived screens provide only 0.2-s and -ten-horizon mean scores, so intermediate cells are not interpolated. -Ego4D transfer is pending. Mean averages all ten horizons, including where -the individual values are unavailable. F3 and the fusion rows are exploratory -seed-0 runs; F1 reports a three-seed mean. These rows are not all -capacity- or training-matched.} +\caption{Dense future-activity mAP (\%) at ten horizons on EasyCom and +Ego4D. Each dataset has the same four-method comparison. EasyCom values +are measured on Sessions~1--3; $\G$ has only archived 0.2-s and mean +scores, and SVA awaits evaluation. Ego4D scores are pending. Dashes are unreported or unmeasured, +not zero; intermediate scores are never interpolated. The available Omni +rows are exploratory and not all training-matched.} \label{tab:main} \centering\ninept \renewcommand{\arraystretch}{0.91} @@ -19,20 +17,39 @@ Dataset / model & .2 & .4 & .6 & .8 & 1.0 & 1.2 & 1.4 & 1.6 & 1.8 & 2.0 & Mean \ \midrule \multicolumn{12}{@{}l}{\emph{EasyCom (measured)}} \\ EgoSpeak A+V & 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82 & 61.19 \\ -Recent $\A$ & 79.69 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 60.51 \\ Omni $\G$ & 73.45 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 60.85 \\ -$\G+\A$ & 81.72 & 75.92 & 70.38 & 66.46 & 63.81 & 61.37 & 60.00 & 58.32 & 56.95 & 56.16 & 65.11 \\ -$\G+\A+$pooled-$\V$ & 82.03 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 65.40 \\ -Full-frame F3 & 82.38 & 76.42 & 70.78 & 66.35 & 63.93 & 61.87 & 60.41 & 59.11 & 57.61 & 56.53 & 65.54 \\ -Person-set F1 & 81.78 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 64.95 \\ +Omni $\G+\A$ & 81.72 & 75.92 & 70.38 & 66.46 & 63.81 & 61.37 & 60.00 & 58.32 & 56.95 & 56.16 & 65.11 \\ +Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ \midrule \multicolumn{12}{@{}l}{\emph{Ego4D (transfer pending)}} \\ +EgoSpeak A+V & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ Omni $\G$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ -$\G+\A$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ +Omni $\G+\A$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ +Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ \bottomrule \end{tabular*} \end{table*} +\begin{table}[t] +\caption{EasyCom visual-encoding screens (\%). Branch names are defined +in Sec.~\ref{sec:method}. Available rows are exploratory and not +capacity- or training-matched.} +\label{tab:visual} +\centering\ninept +\renewcommand{\arraystretch}{0.92} +\setlength{\tabcolsep}{4pt} +\begin{tabular}{@{}lcc@{}} +\toprule +Visual branch on $\G+\A$ & mAP$_{.2}$ & Mean \\ +\midrule +None & 81.72 & 65.11 \\ +Pool & 82.03 & 65.40 \\ +SVA & -- & -- \\ +TSVA & 82.38 & 65.54 \\ +\bottomrule +\end{tabular} +\end{table} + \begin{table}[t] \caption{EasyCom privileged past-activity audit (\%). Three-seed mean $\pm$ sample SD. The extra activity cues are not deployable inputs; W is