From f14026e82ff31efd9709092596089291f7cc6711 Mon Sep 17 00:00:00 2001 From: Xin Wang Date: Mon, 14 Sep 2026 14:23:37 +0800 Subject: [PATCH] Update manuscript results section --- manuscript/main/main.tex | 154 +++++++++++++++++++++++++++++---------- 1 file changed, 115 insertions(+), 39 deletions(-) diff --git a/manuscript/main/main.tex b/manuscript/main/main.tex index 0480a6e..cc75d25 100644 --- a/manuscript/main/main.tex +++ b/manuscript/main/main.tex @@ -223,7 +223,7 @@ its label/input boundary is fully reported.} We evaluate on EasyCom following the EgoSpeak split, using Sessions 4--12 for training and development and Sessions 1--3 for final evaluation \cite{donley2021easycom}. All hyperparameters, event windows, operating thresholds, and checkpoints are selected without access to the test sessions. The input stream is converted to strictly causal examples at 5~Hz. Each decision point is paired with ten future 0.2-s activity bins for background, camera wearer, and aggregate non-wearer speech. Provided person boxes are used only to construct the visible-person inputs; participant identity, close-mic audio, transcripts, and future observations are not available to the model. \todo{Report dataset duration, number of training/test examples and events, and the frozen development/calibration split.} -\subsection{Model Configuration and Training} +\subsection{Model Configuration and Training} Our main backbone is Qwen2.5-Omni-3B. We compare a frozen global Omni readout, matched PEFT/LoRA adaptation, global features with recent local audio, a parameter- and history-matched temporal local branch, and the proposed causal local audio--visual adapter. Qwen2.5-Omni-7B is used only as a capacity-scaling control. All learned variants use the same causal context, prediction head, training examples, and class-balanced binary cross-entropy objective in Eq.~\eqref{eq:loss}. Unless otherwise stated, model selection is performed on the development data and final numbers are averaged over three random seeds. \todo{Freeze context length, local window/rates, optimizer, epochs, LoRA configuration, trainable parameter counts, and inference latency.} @@ -256,26 +256,29 @@ are reported as complementary measures. \section{Results and Analysis} \label{sec:results} -\subsection{Baselines} -The confirmatory comparison contains: an EgoSpeak reproduction; the frozen -Qwen2.5-Omni-3B global state $G$; matched LoRA/PEFT; $G$ plus recent local -audio; a parameter- and history-matched temporal local branch; and the proposed -audio--visible-person adapter. Qwen2.5-Omni-7B is a capacity-scaling control, -not a second paper storyline. \todo{Freeze all baseline implementations and -ensure the EgoSpeak comparison uses identical targets, split, causal validity, -and evaluation.} +\subsection{Main Comparison} +Table~\ref{tab:main} compares the proposed causal local audiovisual adapter +with an EgoSpeak reproduction, the frozen Qwen2.5-Omni-3B global state $G$, +matched PEFT, $G$ plus recent local audio, and a parameter- and history-matched +temporal local branch. Qwen2.5-Omni-7B is included only as a capacity-scaling +control. All methods use the same split, causal inputs, targets, and evaluation +protocol. Event thresholds are selected on training-side calibration data and +frozen before test evaluation. \todo{Freeze all baseline implementations and +verify identical targets, causal validity, and evaluation.} \begin{table}[t] -\caption{Confirmatory EasyCom results. Lower multi-other false entries are -better; all event rows use a training-side selected operating point.} +\caption{Confirmatory EasyCom results. Higher mAP and yield recall are better; +lower false entries and onset latency are better. FP/min is measured at +matched valid-onset recall $R=0.8$.} \label{tab:main} \centering \small \resizebox{\columnwidth}{!}{% \begin{tabular}{lccccc} \hline -Method & 0.2-s mAP & Mean mAP & FP/min@$R=.8$ & Onset lat. & Yield R\\ +Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$ & FP/min $\downarrow$ +& Onset lat. $\downarrow$ & Yield R $\uparrow$\\ \hline EgoSpeak reproduction & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ @@ -288,33 +291,75 @@ Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \t \end{tabular}} \end{table} -\textbf{Information opportunity.} -On development Session~12, an analysis-only seed-0 oracle supplied perfectly -timed visible-other speech activity through an explicit non-deployable path. -Aggregate visible activity improved 0.2-s mAP by 6.97 points and ten-horizon -mean mAP by 2.91 points over matched global-plus-local-audio. Visibility-only -and one-second-delayed activity remained near audio, while per-person histories -did not outperform identity-shuffled histories. This motivates timely local -floor evidence, not participant identity. \todo{After seeds 1--2, report the -interval or remove the exact oracle numbers from the four-page paper.} +\textbf{Dense forecasting.} +The proposed local AV adapter improves 0.2-s mAP from +\todo{$X$ to $Y$} over the strongest matched baseline and improves +mean mAP by \todo{$Z$ points}. +The gain is largest at \todo{short prediction horizons / transition-adjacent +frames / specify actual observation}, indicating that local audiovisual +evidence primarily improves \todo{the interpretation supported by the data}. -\textbf{Main result.} -Table~\ref{tab:main} should establish one result: \todo{the selected local -adapter reduces multi-other false entry at matched valid-onset recall, retains -onset latency/yield behavior, improves dense forecasting, and recovers a -meaningful fraction of the oracle opportunity}. The 7B row should determine -whether \todo{capacity alone closes the gap or the adaptation structure remains -useful at larger scale}. No conclusion is drawn across all Omni families from -Qwen-only evidence. +\textbf{Floor-entry behavior.} +At matched valid-onset recall $R=0.8$, the proposed adapter reduces +multi-other false entries from \todo{$X$} to \todo{$Y$} FP/min +(\todo{$Z\%$ relative reduction}). +This reduction is obtained while \todo{maintaining / improving} onset latency +and yield recall. +Thus, the improvement cannot be explained solely by a more conservative +decision threshold. + +\textbf{Comparison with adaptation baselines.} +Matched PEFT yields \todo{result}, whereas adding recent local audio alone +yields \todo{result}. +The parameter- and history-matched temporal branch achieves +\todo{result}. +The remaining gap between this control and the proposed adapter therefore +suggests that \todo{correctly timed audiovisual local evidence contributes +beyond additional temporal capacity}. +\todo{If this difference is not statistically supported, state instead that +the local temporal branch accounts for most of the gain.} + +\subsection{Information Opportunity} + +Before designing the deployable adapter, we measured whether local floor +information was useful at all using an analysis-only oracle. +The oracle receives perfectly timed visible-other speech activity through a +non-deployable pathway and is never used in the main comparison. + +On development Session~12, aggregate visible activity improved 0.2-s mAP by +6.97 points and ten-horizon mean mAP by 2.91 points over the matched +global-plus-local-audio baseline. +In contrast, visibility alone and one-second-delayed activity remained close +to the audio baseline, while person-specific histories did not outperform +identity-shuffled histories. + +These controls indicate that the useful signal is primarily +\emph{timely local floor evidence}, rather than participant identity or +static visibility. +The proposed adapter is therefore designed to approximate this information +using only causally available audiovisual observations. + +\todo{After completing all seeds, report mean $\pm$ confidence interval here. +If the oracle effect is unstable, describe it qualitatively and omit the exact +numbers from the four-page version.} + +\subsection{Mechanism Analysis} + +To determine whether the gain arises from correctly aligned visual floor +evidence rather than additional parameters or generic visual context, we +evaluate the controls in Table~\ref{tab:ablation}. \begin{table}[t] -\caption{Mechanism controls for the selected Omni-3B local adapter.} +\caption{Mechanism controls for the selected Omni-3B local adapter. All +variants use the same prediction head and comparable temporal capacity.} \label{tab:ablation} \centering \small +\resizebox{\columnwidth}{!}{% \begin{tabular}{lccc} \hline -Local condition & Mean mAP & FP/min@$R=.8$ & Yield R\\ +Local condition & Mean mAP $\uparrow$ & FP/min@$R=.8$ $\downarrow$ +& Yield R $\uparrow$\\ \hline Correct causal local AV & \todo{--} & \todo{--} & \todo{--}\\ Remove local vision & \todo{--} & \todo{--} & \todo{--}\\ @@ -322,17 +367,48 @@ Time-shift local vision & \todo{--} & \todo{--} & \todo{--}\\ Global/full-frame visual pool & \todo{--} & \todo{--} & \todo{--}\\ Same-rate/capacity control & \todo{--} & \todo{--} & \todo{--}\\ \hline -\end{tabular} +\end{tabular}} \end{table} -\textbf{Mechanism and limitations.} -Table~\ref{tab:ablation} should test whether gains require correctly timed local -visual evidence rather than extra parameters, regularization, or generic scene -context. \todo{Report paired intervals and the supported interpretation. If -correct and shifted/removed vision do not separate, drop the audiovisual +Removing local vision changes \todo{metric} by \todo{$X$}, while temporally +shifting the visual stream by \todo{$\Delta t$} changes it by \todo{$Y$}. +Replacing person-local visual evidence with a global/full-frame visual pool +produces \todo{result}. +Together, these comparisons \todo{support / do not support} the hypothesis +that the benefit depends specifically on correctly timed local visual +evidence. + +The same-rate/capacity control achieves \todo{result}, showing that +\todo{the gain cannot / can largely} be attributed to additional temporal +parameters or higher-rate processing alone. + +\todo{Report paired intervals and the supported interpretation. If correct +and shifted/removed vision do not separate, drop the audiovisual evidence-routing claim. If dense mAP improves but event behavior does not, do -not claim improved floor control. If legitimate onset/yield degrades, stop the -adapter rather than describing lower false entry as success.} +not claim improved floor control. If legitimate onset or yield degrades, do +not describe lower false entry as success.} + +\subsection{Scaling Control} + +Increasing the frozen Omni backbone from 3B to 7B yields +\todo{$X$ points} in 0.2-s mAP and \todo{$Y$} FP/min at matched recall. +The proposed 3B adapter \todo{remains better than / approaches / is surpassed +by} the 7B scaling control. + +This result suggests that \todo{architectural access to local causal evidence +provides a complementary benefit to backbone scaling}. +Because both backbones belong to the Qwen2.5-Omni family, we do not claim that +this conclusion generalizes to all omni-modal models. + +\subsection{Limitations and Qualitative Analysis} + +The results support \todo{precise supported claim}, but several limitations +remain. +First, \todo{EasyCom size/session diversity limitation}. +Second, the adapter relies on \todo{state the actual visible-person box or +tracking assumption}. +Third, \todo{failure condition: occlusion, overlapping speakers, rapid floor +exchange, etc.} remains challenging. \todo{Add one compact, pre-selected qualitative timeline only if space permits: one multi-other false-entry improvement paired with one preserved real wearer