Add competing-risk evaluation and align ICASSP story
This commit is contained in:
Xin Wang
@@ -1,7 +1,6 @@
|
||||
% Internal ICASSP 2027 draft using the official ICASSP 2026 template bundled
|
||||
% in this repository. Replace the class/style files when the 2027 author kit
|
||||
% is released, and remove every visible \todo{} marker before submission.
|
||||
% Compile from this directory with: make
|
||||
% ICASSP 2027 content skeleton. The final paper must use the official 2027
|
||||
% author kit and contain no visible \todo{} markers. Planned experiments are
|
||||
% intentionally marked rather than presented as completed work.
|
||||
\documentclass{article}
|
||||
\usepackage{spconf}
|
||||
\usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}
|
||||
@@ -10,7 +9,8 @@
|
||||
\newcommand{\vect}[1]{\boldsymbol{#1}}
|
||||
\setlength{\emergencystretch}{2em}
|
||||
|
||||
\title{Adapting Omni Models for Causal Egocentric Multiparty Floor-Entry Forecasting}
|
||||
\title{Is It My Turn? Causal Local Evidence Adaptation of Omni Models for
|
||||
Egocentric Floor Forecasting}
|
||||
|
||||
% TODO: Insert the final author list and affiliations after the ICASSP 2027
|
||||
% review policy and author order are finalized.
|
||||
@@ -22,603 +22,340 @@
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
In multiparty conversation, a pause need not invite an egocentric participant
|
||||
to speak: it may instead mark a handover between two other people. We study
|
||||
causal egocentric multiparty floor-entry forecasting, where the target is the
|
||||
future speech participation of the first-person camera wearer---the one
|
||||
participant who is necessarily absent from the visible speaker set. This
|
||||
wearer-relative target distinguishes our setting from recent multimodal VAP
|
||||
models that forecast visible floor holders or observe human--human turn
|
||||
dynamics. Following EgoSpeak, we use causal first-person audio--video and the
|
||||
human wearer's future speaking behavior as an imitation target, and ask how a
|
||||
jointly pretrained audio--visual Omni model should be adapted for predictions
|
||||
every 200~ms. Our global--local formulation combines a causal Omni readout with
|
||||
a lightweight pathway for recent mixed audio and visible-person observations.
|
||||
On EasyCom development data, a single-seed oracle analysis finds that perfectly
|
||||
aligned aggregate activity among visible non-wearers improves 0.2-s mAP by
|
||||
6.97 points and ten-horizon mean mAP by 2.91 points over global Omni plus recent
|
||||
audio, whereas person-resolved histories do not outperform an
|
||||
identity-shuffled control. This motivates timely local floor evidence without
|
||||
assuming that person continuity is useful. The full study compares frozen,
|
||||
parameter-efficient, and learned local adaptation and evaluates false wearer
|
||||
entry at matched wearer-onset recall together with yield behavior.
|
||||
\todo{After the train-side study is locked, replace the final two sentences
|
||||
with the selected learned local variant and insert: its oracle-gap recovery,
|
||||
multi-other false-entry reduction at 80\% onset recall, onset/yield
|
||||
non-degradation, and 95\% intervals. Mention track continuity only if its
|
||||
controls separate.}
|
||||
Real-time audiovisual agents must decide not only what to say, but whether an
|
||||
emerging floor opportunity belongs to them. This is difficult in multiparty
|
||||
egocentric interaction: the target wearer is behind the camera, while the
|
||||
visible people provide only indirect evidence about whether the surrounding
|
||||
floor remains occupied. We study strictly causal forecasting of the wearer's
|
||||
future speech activity every 200~ms. Although a general-purpose Omni model
|
||||
provides rich long-context audio--visual representations, its global readout is
|
||||
not explicitly trained to expose sub-second, wearer-relative floor evidence.
|
||||
We therefore study a candidate causal local floor adapter that combines the global
|
||||
Omni state with short-window mixed audio and an anonymous set of visible-person
|
||||
observations, without assuming persistent speaker identity. On EasyCom,
|
||||
\todo{report the final three-seed dense and competing-risk-censored event
|
||||
results: false-entry reduction at matched wearer-onset recall, onset latency,
|
||||
yield behavior, confidence intervals, and oracle-gap recovery}. Controlled
|
||||
\todo{time-shift/remove-vision/same-rate or capacity-matched} interventions
|
||||
show that \todo{state the supported mechanism, without claiming addressee or
|
||||
speaker identity}. These results demonstrate \todo{the narrow conclusion
|
||||
actually supported by the final experiments}.
|
||||
\end{abstract}
|
||||
|
||||
\begin{keywords}
|
||||
conversational agents, turn-taking, speech initiation, egocentric video,
|
||||
omni models
|
||||
conversational agents, turn-taking, egocentric video, multimodal adaptation,
|
||||
voice activity projection
|
||||
\end{keywords}
|
||||
|
||||
\section{Introduction}
|
||||
\label{sec:intro}
|
||||
|
||||
Natural turn-taking requires a conversational system to anticipate not only
|
||||
when speech will stop, but whether an emerging floor opportunity actually
|
||||
belongs to it. This distinction is particularly important in multiparty
|
||||
interaction. In a dyadic conversation, a pause after one interlocutor finishes
|
||||
often provides an opportunity for the other to respond; in a group
|
||||
conversation, the same pause may simply mark a handover between two other
|
||||
participants. An egocentric conversational agent that reacts only to silence
|
||||
or aggregate speech activity may therefore enter the floor while the
|
||||
surrounding conversation should continue without it. We study this problem as
|
||||
\emph{egocentric multiparty floor-entry forecasting}: continuously predicting
|
||||
the near-future speech participation of the first-person participant from
|
||||
causal audio--visual observations.
|
||||
when speech will stop, but whether a floor opportunity actually belongs to it.
|
||||
In a dyadic exchange, a pause after one interlocutor finishes often invites the
|
||||
other to respond. In a group conversation, the same pause may instead mark a
|
||||
handover between two other participants. An egocentric agent that reacts only
|
||||
to silence or aggregate speech may therefore interrupt a conversation that
|
||||
should continue without it. We study \emph{egocentric floor forecasting}:
|
||||
continuously predicting the near-future speech participation of the
|
||||
first-person wearer from strictly causal audio--visual observations.
|
||||
|
||||
Voice Activity Projection (VAP) provides a predictive formulation of
|
||||
turn-taking by continuously projecting the joint future voice activity of
|
||||
conversational participants, from which turn shifts, holds, and backchannels
|
||||
can be inferred \cite{ekstedt2022vap}. EgoSpeak extends predictive speech
|
||||
initiation to the first-person setting, using causal, untrimmed egocentric
|
||||
audio--visual observations and the future speaking behavior of the camera
|
||||
wearer as supervision \cite{kim2025egospeak}. This behavior is a useful
|
||||
imitation target for the perceptual decision faced by an egocentric agent:
|
||||
given everything observed so far, is the first-person participant likely to
|
||||
enter the floor next? EgoSpeak establishes that this prediction can be learned
|
||||
from real-world egocentric streams, but leaves open how recent jointly
|
||||
pretrained audio--visual Omni models should be used for such fine-grained
|
||||
causal forecasting.
|
||||
Voice Activity Projection (VAP) introduced continuous prediction of future
|
||||
voice activity for turn-taking \cite{ekstedt2022vap}. EgoSpeak brought this
|
||||
formulation to causal, untrimmed first-person streams, using future camera-
|
||||
wearer activity as an imitation target \cite{kim2025egospeak}. Complementary
|
||||
work adds visual gaze, head pose, and facial behavior to predictive turn-taking
|
||||
\cite{russell2025mmvap}, while MuVAP combines global acoustic dynamics with
|
||||
visible face tracks to forecast visible floor holders in multiparty scenes
|
||||
\cite{qi2026muvap}. Our target has a different role structure: the participant
|
||||
whose future action is predicted is the one participant absent from the
|
||||
visible set. The visible people are contextual evidence for the unseen
|
||||
wearer's decision, not candidate output identities.
|
||||
|
||||
In parallel, recent work has expanded predictive turn-taking along multimodal
|
||||
and multiparty dimensions. MM-VAP incorporates facial expression, head pose,
|
||||
and gaze into predictive turn-taking for two-party visual interaction
|
||||
\cite{russell2025mmvap}. A recent social-robot MM-VAP framework adapts
|
||||
voice-activity-related audio--visual encoders for anticipating human--human
|
||||
turn dynamics in observer or mediation settings \cite{cano2026mmvap}. MuVAP
|
||||
further extends VAP to multiparty interaction using a monaural mixed-audio
|
||||
stream and a single camera, grounding acoustic dynamics in visible face tracks
|
||||
and predicting current and future visible floor holders \cite{qi2026muvap}.
|
||||
These formulations are complementary to our egocentric setting. In MuVAP, the
|
||||
candidate future speakers belong to the set visible to the camera; in
|
||||
mediation-oriented MM-VAP, the system observes turn dynamics between human
|
||||
participants. Here, by contrast, the participant whose future floor entry must
|
||||
be predicted is located \emph{behind the camera} and is therefore absent from
|
||||
the visible participant set.
|
||||
This role asymmetry exposes an adaptation problem for general-purpose Omni
|
||||
models. Omni pretraining favors globally integrated representations useful for
|
||||
multimodal understanding and generation. Egocentric floor control instead
|
||||
depends on timely prosodic and visible-person activity at sub-second scale,
|
||||
including whether other participants continue to occupy the floor. Increasing
|
||||
model capacity or applying a generic parameter-efficient update does not
|
||||
explicitly preserve or route this evidence to a dense wearer-relative
|
||||
prediction head. The open question is therefore not merely whether an Omni
|
||||
backbone can be applied to EgoSpeak, but what task structure is still required
|
||||
to turn general audiovisual understanding into reliable floor control.
|
||||
|
||||
This leaves an underexplored intersection: continuous causal prediction of an
|
||||
egocentric participant's own future floor participation in a changing
|
||||
multiparty scene using a general-purpose audio--visual Omni model. The defining
|
||||
difference is its viewpoint and prediction target: the model must project the
|
||||
activity of the unseen first-person participant from the causal stream used to
|
||||
observe everyone else. We investigate how pretrained Omni representations can
|
||||
be adapted under the EgoSpeak wearer-relative forecasting objective. Current
|
||||
Qwen2.5-Omni results motivate this study, but do not establish a limitation
|
||||
shared by Omni models in general.
|
||||
Our candidate decomposition separates long-context understanding from
|
||||
short-window floor evidence. A
|
||||
causal global Omni state represents the broader scene and conversation. In
|
||||
parallel, a lightweight local path encodes recent mixed audio and a changing,
|
||||
anonymous set of visible-person observations. A residual adapter makes this
|
||||
local evidence accessible to the common ten-horizon activity head. The primary
|
||||
variant deliberately avoids cross-time identity: the current development
|
||||
oracle supports timely aggregate visible-other activity but not person
|
||||
continuity. Track-aware structure is considered only if it defeats no-ID,
|
||||
identity-corrupted, and capacity-matched controls.
|
||||
|
||||
Our working hypothesis separates broad context from timely floor evidence. A
|
||||
causal global Omni state can represent the scene and conversational history,
|
||||
while a lightweight local pathway preserves recent mixed audio and a changing
|
||||
set of visible-person observations. The latter may be aggregated per frame or
|
||||
organized into anonymous tracks; track continuity becomes a contribution only
|
||||
if it outperforms aggregate, no-ID, shuffled, and capacity-matched controls. An
|
||||
analysis-only oracle screen on held-out EasyCom Session~12 supports the weaker
|
||||
hypothesis: perfectly timed aggregate visible-other activity improves a
|
||||
matched global-plus-audio model by 6.97 points at 0.2~s and 2.91 points in
|
||||
ten-horizon mean mAP, whereas person-resolved histories do not outperform an
|
||||
identity-shuffled control. Thus the current evidence motivates timely local
|
||||
floor information, not speaker identity or addressee understanding.
|
||||
|
||||
We retain EgoSpeak-compatible dense future-activity prediction for
|
||||
comparability and add a behavior-aligned event analysis. In particular,
|
||||
per-frame average precision can hide the failure that motivates this work:
|
||||
entering during an exchange among other participants. We therefore measure
|
||||
false wearer entries during multi-other opportunities at matched wearer-onset
|
||||
recall, together with response latency and wearer-yield behavior. These events
|
||||
are observable timing-derived interaction proxies, not semantic EOT, INT, or
|
||||
addressee labels; the final protocol uses competing-risk censoring so an event
|
||||
cannot claim a later action across an incompatible floor change. This follows
|
||||
the broader event-oriented emphasis on recall, false alarms, and latency
|
||||
\cite{jiang2026turnbench}.
|
||||
|
||||
Our contributions are: (1) a controlled study of frozen and
|
||||
parameter-efficient Omni adaptation for EgoSpeak-style causal forecasting;
|
||||
(2) an oracle information audit that distinguishes timely aggregate local
|
||||
floor evidence from person continuity; (3) a global--local adaptation selected
|
||||
through causal, alignment, and capacity-matched controls; and (4) an event
|
||||
protocol that tests whether dense gains reduce false first-person floor entry
|
||||
without suppressing valid entry or yield. EasyCom supplies synchronized
|
||||
egocentric audio--video, per-person tracks, and voice activity
|
||||
\cite{donley2021easycom}; target-of-speech annotations, when used, serve only
|
||||
as analysis strata and never as model input or required output.
|
||||
The final paper will claim the following three contributions only if the
|
||||
corresponding TODO gates pass:
|
||||
\begin{itemize}
|
||||
\item We identify and test a temporal--role mismatch between a global Omni
|
||||
readout and causal egocentric floor forecasting for an invisible wearer.
|
||||
\item We evaluate a wearer-relative causal local floor adapter that combines
|
||||
long-context Omni information with short-window audio and anonymous
|
||||
visible-person evidence.
|
||||
\item We evaluate whether the adapter reduces false wearer entries during
|
||||
multi-other exchanges at matched valid-onset recall, using causal alignment
|
||||
interventions and competing-risk-censored entry/yield analysis.
|
||||
\end{itemize}
|
||||
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
|
||||
\todo{After model selection, draw the final system figure. Show the shared
|
||||
interface: raw causal AV to a global Omni readout; a parallel recent-audio and
|
||||
visible-person path; aggregate/no-ID/track-organized variants; residual fusion;
|
||||
and the common $10\times3$ head. Draw the transcript-derived oracle only as a
|
||||
dashed, analysis-only upper-bound path. Mark causal cutoffs, provided versus
|
||||
online tracks, trainable modules, NULL/OFFSCREEN handling, and resets.}}}
|
||||
\caption{Global context and causal local evidence for egocentric floor action.
|
||||
Track organization is a controlled candidate, not a presupposed source of
|
||||
speaker identity.}
|
||||
\fbox{\parbox[c][1.00in][c]{0.94\columnwidth}{\centering
|
||||
\todo{Draw only the selected method: causal AV to the long-context Omni path;
|
||||
the short-window mixed-audio and anonymous person-set path; residual fusion;
|
||||
the common $10\times3$ head; and the wearer-silent/active event interpretation.
|
||||
Mark physical cutoffs, trainable modules, person masks, and NULL/OFFSCREEN.
|
||||
Do not draw oracle labels, rejected track variants, or multiple backbones.}}}
|
||||
\caption{Long-context Omni understanding and short-window local floor evidence
|
||||
for wearer-relative future-activity prediction.}
|
||||
\label{fig:method}
|
||||
\end{figure}
|
||||
|
||||
\section{Methods}
|
||||
\section{Method}
|
||||
\label{sec:method}
|
||||
|
||||
\subsection{Wearer-relative activity projection}
|
||||
\subsection{Causal wearer-relative projection}
|
||||
|
||||
At decision time $t$, the model observes only the causal stream
|
||||
$\mathcal{X}_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s.
|
||||
For bin $k$, the target
|
||||
$\vect{y}_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
|
||||
At decision time $t$, the model observes only the stream
|
||||
$\mathcal X_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s. For
|
||||
bin $k$, the multilabel target
|
||||
$\vect y_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
|
||||
background, camera-wearer activity, and activity by one or more other people.
|
||||
Wearer--other overlap is allowed; hence this is not a three-way softmax. The
|
||||
output is
|
||||
Wearer--other overlap is allowed. The model outputs
|
||||
\begin{equation}
|
||||
\hat{\vect{Y}}_t=f_\theta(\mathcal{X}_{\leq t})\in[0,1]^{K\times3},
|
||||
\qquad q_t=\hat y^{\mathrm{w}}_{t,1},
|
||||
\label{eq:task}
|
||||
\hat{\vect Y}_t=f_\theta(\mathcal X_{\leq t})
|
||||
\in[0,1]^{K\times3}.
|
||||
\label{eq:task}
|
||||
\end{equation}
|
||||
where $q_t$ is the immediate wearer-entry score used by the event protocol.
|
||||
It is interpreted as an \emph{entry} score only in wearer-silent opportunity
|
||||
windows; elsewhere it remains a future-activity probability. This retains the
|
||||
EgoSpeak target and evaluation while exposing a deployable 5-Hz decision
|
||||
signal.
|
||||
The same wearer-activity forecast has two controller interpretations. In a
|
||||
wearer-silent opportunity, $\hat y^{\mathrm w}_{t,1}$ is an entry score. While
|
||||
the wearer is active, $1-\hat y^{\mathrm w}_{t,1}$ is a near-term yield score.
|
||||
The current self-state is used to define these observable event strata, not as
|
||||
an unreported oracle model input. \todo{If the final adapter explicitly uses
|
||||
self-state, define its strictly causal and deployment-available source and add
|
||||
a matched no-state ablation; otherwise keep self-state evaluation-only.}
|
||||
|
||||
\subsection{Omni backbones and representation protocols}
|
||||
\subsection{Global Omni baseline}
|
||||
|
||||
Let $m$ index an omni backbone and let $\vect{g}^{(m)}_t$ be a
|
||||
backbone-specific causal global readout at the same physical cutoff $t$.
|
||||
Qwen2.5-Omni \cite{xu2025qwenomni} is the currently implemented family;
|
||||
Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o~4.5
|
||||
\cite{cui2026minicpmo45} form the planned initial comparison. These models have
|
||||
different native streaming APIs, temporal tokenization, hidden dimensions, and
|
||||
adaptation interfaces. We therefore emphasize within-backbone deltas rather
|
||||
than treating raw cross-model scores as a pure architecture ranking.
|
||||
Let $\vect g_t\in\mathbb R^{d_g}$ be the causal global readout of
|
||||
Qwen2.5-Omni at the physical cutoff $t$ \cite{xu2025qwenomni}. The frozen
|
||||
baseline trains only a dimension-matched prediction head. A matched PEFT/LoRA
|
||||
baseline updates a pre-registered subset of the Omni model under the same
|
||||
examples, context, optimizer budget, head, and evaluator. Its purpose is to
|
||||
test whether generic parameter adaptation is sufficient; one negative recipe
|
||||
is not interpreted as a general limitation of LoRA or all Omni models.
|
||||
|
||||
Every supported backbone is evaluated under three representation protocols:
|
||||
(i) a frozen global readout with a dimension-matched prediction head;
|
||||
(ii) a pre-registered parameter-efficient adaptation budget; and (iii) the
|
||||
same global readout augmented by causal local evidence. Input cutoffs, EgoSpeak
|
||||
targets, context duration, training examples, head capacity, and evaluation
|
||||
are held fixed whenever the backbone permits it. A failure of one LoRA recipe
|
||||
is reported as that recipe's result, not as proof that the backbone cannot
|
||||
adapt.
|
||||
\todo{Freeze the exact Qwen2.5-Omni-3B revision, processor, input rates,
|
||||
streaming/cache semantics, readout layer, context, dtype, LoRA targets/rank,
|
||||
trainable parameter budget, and cutoff audit. Add Qwen2.5-Omni-7B only as a
|
||||
capacity-scaling control. Do not add another backbone family unless it is
|
||||
needed to support and experimentally earns a cross-family claim.}
|
||||
|
||||
\todo{Implement and audit one adapter for each planned backbone. For every
|
||||
model, document model revision, processor, native audio/video rates, exact
|
||||
hidden-state readout, causal/streaming semantics, context limit, dtype,
|
||||
trainable LoRA targets/rank, parameter count, peak memory, and P50/P95 latency.
|
||||
Verify no future frames or samples enter $\vect g^{(m)}_t$. If a model cannot
|
||||
expose a comparable causal state, report it as unsupported rather than inventing
|
||||
a non-equivalent readout. Add future backbones only before the experiment
|
||||
protocol is locked.}
|
||||
\subsection{Causal local floor adapter}
|
||||
|
||||
\subsection{Global context and causal local evidence}
|
||||
|
||||
The global state $\vect g^{(m)}_t$ is responsible for broad scene and
|
||||
conversational context, but is not assumed to retain every localized cue needed
|
||||
for floor control. From the most
|
||||
recent $L_a$ seconds, a causal local audio encoder produces $\vect a_t$. At
|
||||
time $t$, provided boxes during method isolation---and an online detector and
|
||||
tracker in the deployable condition---produce a variable-size anonymous set
|
||||
$\mathcal P_t=\{(\vect v_{t,i},\vect b_{t,i},\vect r_{t,i})\}_{i=1}^{N_t}$,
|
||||
where $\vect v_{t,i}$ is a crop feature, $\vect b_{t,i}$ is box geometry, and
|
||||
$\vect r_{t,i}$ contains only causal reliability variables such as visibility,
|
||||
track age, and time since observation. A shared causal track encoder updates
|
||||
The candidate local path operates on the most recent $L_\ell$ seconds. At local instant
|
||||
$\tau\leq t$, a causal audio encoder produces $\vect a_\tau$ from mixed audio.
|
||||
For the anonymous visible-person set
|
||||
$\mathcal P_\tau=\{(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau}$,
|
||||
a shared visual encoder and masked set operator produce
|
||||
\begin{equation}
|
||||
\vect h_{t,i}=F_{\mathrm{trk}}\!\left(\vect h^-_{t,i},
|
||||
W_v\vect v_{t,i}+W_b\phi(\vect b_{t,i})+W_r\vect r_{t,i}\right).
|
||||
\label{eq:track}
|
||||
\vect v_\tau=\mathrm{SetPool}
|
||||
\left(\{E_v(\vect x_{\tau,i},\vect b_{\tau,i})}_{i=1}^{N_\tau}
|
||||
\cup\{\vect v_\varnothing\}\right),
|
||||
\label{eq:set}
|
||||
\end{equation}
|
||||
All tracks share $F_{\mathrm{trk}}$; the temporary ID only retrieves
|
||||
$\vect h^-_{t,i}$ and is never embedded. The global-audio query then reads the
|
||||
current person set by masked cross-attention,
|
||||
\begin{align}
|
||||
\vect u_t &= \mathrm{MHA}\!\left(
|
||||
W_q[\vect g^{(m)}_t;\vect a_t],
|
||||
[\vect h_{\varnothing};\vect h_{t,1};\ldots;\vect h_{t,N_t}]
|
||||
\right), \\
|
||||
\vect z^{(m)}_t &= \vect g^{(m)}_t+W_o\vect u_t,
|
||||
where $\vect v_\varnothing$ handles an empty or offscreen visible set. The
|
||||
operator is permutation invariant and uses no participant-ID embedding or
|
||||
cross-time identity. A causal short-window encoder summarizes local dynamics,
|
||||
\begin{equation}
|
||||
\vect \ell_t=F_\ell
|
||||
\left(\{[\vect a_\tau;\vect v_\tau]\}_{\tau\in(t-L_\ell,t]}\right).
|
||||
\label{eq:local}
|
||||
\end{equation}
|
||||
The adapter preserves the global initialization through a zero-initialized
|
||||
residual output,
|
||||
\begin{equation}
|
||||
\vect z_t=\vect g_t+W_o\,
|
||||
\sigma\!\left(W_g\vect g_t+W_\ell\vect\ell_t\right),
|
||||
\qquad W_o\vert_{\mathrm{init}}=0,
|
||||
\label{eq:adapter}
|
||||
\end{align}
|
||||
where $\vect h_{\varnothing}$ is a learned NULL/OFFSCREEN token. Padding is
|
||||
masked, so the same network handles $N_t=0$ and changing group size. For a
|
||||
bounded implementation, $N_t>N_{\max}$ is resolved by a deterministic ranking
|
||||
using detector confidence and track age available at $t$, and truncation is
|
||||
reported. The operator is permutation equivariant
|
||||
\cite{lee2019settransformer}; temporary tracker IDs provide continuity, not
|
||||
identity semantics.
|
||||
\label{eq:fusion}
|
||||
\end{equation}
|
||||
and the common head maps $\vect z_t$ to Eq.~\eqref{eq:task}. This implements an
|
||||
evidence-access hypothesis rather than adding an unconstrained second model:
|
||||
the global path retains broad context, while the local path preserves timely
|
||||
floor evidence.
|
||||
|
||||
Equation~\eqref{eq:track} is a candidate organization of local evidence, not a
|
||||
claim that recurrent identity is necessary. Simpler alternatives replace each
|
||||
$\vect h_{t,i}$ with a current-frame observation or pool the set into one
|
||||
aggregate vector. This distinction is important because the current oracle
|
||||
screen favors aggregate activity and does not separate correct from shuffled
|
||||
identity histories.
|
||||
\todo{Freeze $L_\ell$, local audio/video rates, crop source and resolution,
|
||||
set pooling, causal temporal encoder, masks, reset rules, parameter count, and
|
||||
latency. If the local path runs faster than the 5-Hz global/output grid, compare
|
||||
it with a same-parameter 5-Hz variant before using the term ``dual-rate.'' Use
|
||||
provided causal boxes only as a clearly labeled method-isolation condition;
|
||||
online detection/tracking is not a paper contribution unless actually run.}
|
||||
|
||||
This decomposition is informed by MuVAP: its global audio representation
|
||||
queries same-frame causal ASD track representations, followed by temporal
|
||||
modeling and gated global-to-speaker fusion \cite{qi2026muvap}. We retain the
|
||||
useful separation between global floor dynamics and person-organized evidence,
|
||||
but not its role-relative output: our fixed target is the invisible wearer's
|
||||
entry/yield action, not the next visible speaker. The zero-initialized residual
|
||||
output follows the conservative adapter principle used to inject visual
|
||||
evidence without perturbing the base prediction at initialization
|
||||
\cite{rouditchenko2024whisperflamingo}. Unlike a zero scalar gate, it permits
|
||||
the residual output layer to receive a gradient immediately. Local evidence
|
||||
is derived from the same causal raw media, not close microphones, transcripts,
|
||||
speaker labels, global identity anchors, or future frames.
|
||||
\subsection{Training}
|
||||
|
||||
We evaluate candidates in increasing structural complexity: pooled recent
|
||||
audio/vision; a capacity-matched temporal branch; a per-frame person set with
|
||||
no cross-time identity; a shared track-wise temporal encoder; and person--audio
|
||||
cross-attention as in Eqs.~\eqref{eq:track}--\eqref{eq:adapter}. Only a
|
||||
candidate that improves the pre-registered event metric over simpler,
|
||||
aggregate, identity-shuffled, and capacity-matched controls becomes the final
|
||||
method. If person continuity does not separate from those controls, the final
|
||||
model remains a global-plus-local aggregate floor-evidence model. A limited latent
|
||||
bottleneck is tested only if large visible sets make full person attention a
|
||||
runtime bottleneck; it is not assumed to improve accuracy.
|
||||
|
||||
\todo{Run the candidate ladder before fixing $H_\phi$. Match trainable
|
||||
parameters and effective local history. Compare a shared local encoder across
|
||||
backbones (cross-model control) with each backbone's native local encoder
|
||||
(within-model control); do not mix these protocols in one claimed delta.
|
||||
For track candidates, specify visibility/age masks, gap/source resets,
|
||||
OFFSCREEN handling, and global-rename invariance. Describe only the selected
|
||||
architecture as the final method; move rejected candidates to ablations.}
|
||||
|
||||
\subsection{Training objectives}
|
||||
|
||||
The common main loss is class-balanced binary cross entropy over all horizons
|
||||
and activity channels. This dense EgoSpeak target is the only supervision
|
||||
required by the task. If a person-aware candidate survives the activity-only
|
||||
comparison, an optional auxiliary head predicts current active-speaker state:
|
||||
The primary objective is class-balanced binary cross entropy over valid
|
||||
horizons and activity channels,
|
||||
\begin{equation}
|
||||
\mathcal{L}=\mathcal{L}_{\mathrm{act}}+
|
||||
\lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}.
|
||||
\mathcal L_{\mathrm{act}}=
|
||||
\sum_{t,k,c}m_{t,k}w_c\,
|
||||
\mathrm{BCE}(\hat y^c_{t,k},y^c_{t,k}).
|
||||
\label{eq:loss}
|
||||
\end{equation}
|
||||
Participant ID, target-of-speech, transcript, close-microphone audio, and
|
||||
future frames are never model inputs. Target-of-speech may stratify qualitative
|
||||
analysis but is not a predicted variable or a requirement for the primary
|
||||
multi-other event definition.
|
||||
|
||||
\todo{First compare all representation/adaptation protocols with activity loss
|
||||
only. Then ablate +ASD for the selected person-aware candidate. Add an
|
||||
event-specific wearer onset/yield loss only as a separately reported extension.
|
||||
Select all coefficients and early stopping on a training-side calibration
|
||||
fold; do not inspect Sessions 1--3 during selection.}
|
||||
|
||||
\subsection{Track source and deployment contract}
|
||||
|
||||
The intended inference pipeline runs a face/head detector on each incoming
|
||||
frame and associates detections with an online multi-object tracker using only
|
||||
observations at or before $t$. Its temporary IDs route the shared recurrent
|
||||
state in Eq.~\eqref{eq:track}; they do not denote names, fixed seats, or
|
||||
cross-session identities. New tracks initialize from a learned state, short
|
||||
misses retain a causally aged state, and expired tracks are removed. With no
|
||||
visible face, the global/audio path and NULL/OFFSCREEN token remain valid. No
|
||||
future-frame smoothing, full-video identity anchor, or cross-session
|
||||
recognition is permitted. EasyCom state is continued only across genuinely
|
||||
adjacent source segments and reset across redacted gaps; Ego4D state is local
|
||||
to a clip.
|
||||
|
||||
During method isolation, dataset-provided boxes and anonymous IDs form a
|
||||
separate \emph{provided-track} condition. This answers whether person
|
||||
structure can help under accurate localization, but is not reported as online
|
||||
detector performance. The deployable \emph{online-track} condition runs the
|
||||
detector/tracker above and includes its errors and latency. This separation is
|
||||
important because MuVAP used random full-video identity anchors to build its
|
||||
dataset tracks; that preprocessing is useful for annotation but is not a
|
||||
strictly online association algorithm \cite{qi2026muvap}.
|
||||
|
||||
\todo{Select and freeze the face/head detector, causal tracker, thresholds,
|
||||
maximum age, and $N_{\max}$ on training-side data. Report detector recall,
|
||||
provided/online-track coverage, mean/maximum people, truncations, ID switches,
|
||||
fragments, occlusion gaps, source resets, and P50/P95 detector--tracker latency
|
||||
for both datasets. Compare provided tracks, online tracks, and controlled box
|
||||
jitter/dropout/fragmentation. Until the online row is run, limit deployment
|
||||
claims to the intended pipeline and label all numbers ``provided-track.''}
|
||||
No transcript, target-of-speech label, close microphone, future frame, or
|
||||
per-person speech label enters the deployable model. \todo{Add an active-speaker
|
||||
or event auxiliary loss only if it improves the frozen primary protocol and
|
||||
its label/input boundary is fully reported.}
|
||||
|
||||
\section{Experiments}
|
||||
\label{sec:setup}
|
||||
\label{sec:experiments}
|
||||
|
||||
\textbf{Data and splits.}
|
||||
We use the EgoSpeak EasyCom protocol: Sessions 4--12 for model development and
|
||||
Sessions 1--3 for final reporting. All architecture, checkpoint, threshold,
|
||||
and hyperparameter choices use training-side folds only. Inputs contain the
|
||||
camera stream and shared array audio available to the wearer; close-talking
|
||||
microphones, per-person voice labels, and target-of-speech labels are not model
|
||||
inputs. We also report aggregate activity forecasting on the EgoSpeak Ego4D
|
||||
split \cite{grauman2022ego4d}. Ego4D Audio-Visual Diarization supplies
|
||||
clip-local anonymous participant tracks; its per-person voice coverage on the
|
||||
433-clip EgoSpeak split must be audited before reporting the same multi-other
|
||||
event analysis.
|
||||
\textbf{Data and split.}
|
||||
We follow the EgoSpeak EasyCom split: Sessions 4--12 for training-side model
|
||||
development and Sessions 1--3 for final reporting \cite{donley2021easycom}.
|
||||
Thresholds, windows, checkpoints, loss weights, and early stopping are selected
|
||||
without using the official test sessions. Already observed test results are
|
||||
treated as exploratory internal evidence and are not included in the paper.
|
||||
\todo{Report duration, frame/event counts, class prevalence, visibility and
|
||||
box coverage, source discontinuities, and the frozen calibration fold.}
|
||||
|
||||
\textbf{Oracle information audit.}
|
||||
Before selecting a learned local module, we measure what information could in
|
||||
principle complement the global state. Analysis-only sidecars expose
|
||||
transcript-derived speech activity no later than the current cutoff under six
|
||||
conditions: local audio, visibility only, one-second-delayed activity, timely
|
||||
aggregate visible activity, per-person histories, and histories with their
|
||||
cross-time identity assignments shuffled. These labels never enter a
|
||||
deployable model. The audit separates three hypotheses that ordinary feature
|
||||
ablation conflates: the value of seeing people, the value of knowing current
|
||||
visible-other activity, and the incremental value of preserving who produced
|
||||
that activity.
|
||||
\textbf{Baselines.}
|
||||
The confirmatory comparison contains: an EgoSpeak reproduction; the frozen
|
||||
Qwen2.5-Omni-3B global state $G$; matched LoRA/PEFT; $G$ plus recent local
|
||||
audio; a parameter- and history-matched temporal local branch; and the proposed
|
||||
audio--visible-person adapter. Qwen2.5-Omni-7B is a capacity-scaling control,
|
||||
not a second paper storyline. \todo{Freeze all baseline implementations and
|
||||
ensure the EgoSpeak comparison uses identical targets, split, causal validity,
|
||||
and evaluation.}
|
||||
|
||||
\textbf{Backbone comparison.}
|
||||
The initial study covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and
|
||||
MiniCPM-o~4.5. For each backbone $m$, we first compare its frozen causal state
|
||||
$G_m$, a parameter-efficiently adapted state $P_m$, and local-evidence
|
||||
augmentation $G_m+L$. The primary diagnostic is the within-backbone change;
|
||||
cross-backbone absolute scores are secondary because native tokenization,
|
||||
pretraining, streaming interfaces, and hidden-state semantics differ. Common
|
||||
local encoders test whether adaptation transfers across global backbones,
|
||||
whereas native local encoders are a separate, model-specific protocol. All
|
||||
rows share physical input cutoffs, targets, data splits, and evaluation.
|
||||
Unsupported causal readouts are disclosed rather than replaced by offline or
|
||||
future-conditioned features.
|
||||
\textbf{Event protocol.}
|
||||
Dense ten-horizon AP/mAP is retained for EgoSpeak comparability. The implemented
|
||||
competing-risk evaluator counts each eligible aggregate-other boundary once,
|
||||
ends a response window before the next other onset, and ends a yield window
|
||||
before the next other offset. Pending the remaining P0 real-data audit and
|
||||
scenario freeze, the intended primary behavior metric is false wearer entries per
|
||||
minute of multi-other opportunity at a training-side selected wearer-onset
|
||||
recall, with $R=.8$ as the headline operating point. The intended analysis also
|
||||
reports onset latency and wearer-yield recall/latency, following the
|
||||
event-oriented emphasis on recall, false alarms, and latency
|
||||
\cite{jiang2026turnbench}. The protocol will censor a response window after an
|
||||
other offset at the next other onset, and a yield window after an other onset
|
||||
at the next other offset.
|
||||
|
||||
\todo{Implement the three backbone adapters and validate state timestamps with
|
||||
an impulse/cutoff test. Pre-register one matched PEFT budget and one common
|
||||
local branch. Report native-vs-common local encoders in separate table panels.
|
||||
Keep future model additions outside the confirmatory comparison once this
|
||||
matrix is locked.}
|
||||
\todo{P0 blocker: run the tested competing-risk evaluator on real predictions,
|
||||
manually audit sampled timelines, and freeze pairing before treating any event
|
||||
score as a paper result. Specify overlap,
|
||||
minimum duration, timeouts, duplicate detections, source boundaries, offscreen
|
||||
cases, and event counts. Until this is complete, call existing values
|
||||
timing-derived proxies rather than final event results.}
|
||||
|
||||
\textbf{Event construction.}
|
||||
We derive \emph{wearer onset} events directly from the wearer activity target.
|
||||
A \emph{multi-other exchange} is a wearer-silent opportunity interval in which
|
||||
at least two distinct non-wearer participant IDs contribute temporally ordered
|
||||
speech activity. It is an observable interaction proxy, not a claim about the
|
||||
speakers' mental intent or addressee. Threshold $\tau_r$ is selected on the
|
||||
development split to obtain wearer-onset recall
|
||||
$r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary metric is false wearer
|
||||
entries per minute of multi-other opportunity at matched onset recall; the
|
||||
headline operating point is $r=0.8$. We also report signed onset latency,
|
||||
wearer-yield recall/latency, and the original ten-horizon per-frame mAP.
|
||||
|
||||
\todo{Freeze the event pairing and competing-risk rules, including the
|
||||
multi-other alternation window, minimum speech/silence, overlap, maximum onset
|
||||
delay, duplicate detections, one detection per event, track visibility, source
|
||||
boundaries, and group/offscreen cases. Publish train/test onset counts,
|
||||
multi-other events/minutes, number of distinct visible speakers, and exclusion
|
||||
counts. Manually audit at least 100 events. Treat all previously viewed
|
||||
Sessions 1--3 scores as exploratory; lock the protocol and model before the
|
||||
confirmatory three-seed run.}
|
||||
|
||||
\textbf{Baselines and controls.}
|
||||
For each backbone we compare: frozen global state $G_m$; PEFT $P_m$; $G_m$
|
||||
plus recent local audio; and $G_m$ plus pooled local audio--visual evidence.
|
||||
This compact matrix tests the representation hypothesis before investing in a
|
||||
complex fusion module. The first learned target after the oracle audit is a
|
||||
causal estimator of timely aggregate visible-person floor evidence. Only if
|
||||
person-organized features survive three seeds, visual time/person corruptions,
|
||||
and the event gate do we compare a parameter-matched temporal branch without
|
||||
person continuity, a separately trained per-frame/no-ID set branch, and a
|
||||
track-consistent masked cross-attention branch. Provided-track and online-track
|
||||
inputs are reported separately. On a frozen track-aware checkpoint, a clip-consistent
|
||||
global renaming must leave outputs unchanged, whereas \emph{broken
|
||||
continuity} periodically remaps IDs while preserving every crop, timestamp,
|
||||
audio sample, and target. Time-shifted and removed vision provide additional
|
||||
alignment controls.
|
||||
|
||||
\todo{Define the broken-continuity corruption before evaluation: remap only at
|
||||
pre-registered 1-s boundaries or report a fixed ID-switch-rate curve. Do not
|
||||
call a clip-consistent slot permutation ``broken ID.'' Verify unchanged crops,
|
||||
frame counts, audio, labels, and visibility masks. Train the no-ID/reset
|
||||
baseline from scratch with matched parameters; evaluate all test-time
|
||||
counterfactuals using one frozen checkpoint and one frozen threshold.}
|
||||
|
||||
\textbf{Implementation and statistics.}
|
||||
All prediction heads run at 5~Hz and use an 8-s causal observation window;
|
||||
model-specific deviations required by a native streaming interface are stated
|
||||
explicitly. In the frozen protocol the omni backbone does not update. In PEFT
|
||||
and local protocols, trainable parameters and optimizer exposure are reported
|
||||
separately. Every temporal state resets at non-contiguous source boundaries.
|
||||
We report three seeds and mean$\pm$standard deviation. Confidence intervals for
|
||||
event metrics use paired cluster bootstrap over sessions/conversation blocks
|
||||
rather than treating adjacent frames as independent. Runtime includes P50/P95
|
||||
decision latency, throughput, peak memory, and trainable parameter count.
|
||||
|
||||
\todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions,
|
||||
$L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter
|
||||
count, tracker timeout/ranking, hardware, and end-to-end real-time latency
|
||||
after the architecture is frozen.}
|
||||
\textbf{Controls and statistics.}
|
||||
The main intervention compares correctly aligned local evidence with
|
||||
time-shifted evidence, removed vision, global-frame visual pooling, and the
|
||||
capacity-matched temporal branch. A same-rate control is required for a
|
||||
dual-rate claim. Track/no-ID/broken-continuity controls are added only if a
|
||||
track-aware variant first wins on training-side data. We report three seeds,
|
||||
mean$\pm$sample standard deviation, and paired cluster-bootstrap intervals over
|
||||
sessions or conversation blocks. \todo{Freeze seeds, bootstrap unit/count,
|
||||
optimizer, epochs, hardware, throughput, P50/P95 latency, and trainable
|
||||
parameters.}
|
||||
|
||||
\section{Results and Analysis}
|
||||
\label{sec:results}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Causal oracle-information screen on EasyCom development Session~12
|
||||
(seed 0, four epochs). Oracle speech activity is analysis-only. Values are
|
||||
percentages under the causal-valid protocol.}
|
||||
\label{tab:oracle}
|
||||
\centering
|
||||
\small
|
||||
\begin{tabular}{lcc}
|
||||
\hline
|
||||
Input condition & 0.2-s mAP & Mean mAP\\
|
||||
\hline
|
||||
Global + local audio & 72.58 & 58.77\\
|
||||
Visibility only & 71.95 & 58.86\\
|
||||
Activity delayed by 1 s & 72.02 & 58.90\\
|
||||
Aggregate visible activity & \textbf{79.55} & \textbf{61.68}\\
|
||||
Per-person activity histories & 78.80 & 61.11\\
|
||||
Identity-shuffled histories & 78.86 & 61.19\\
|
||||
\hline
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Table~\ref{tab:oracle} establishes an information opportunity, not a deployable
|
||||
result. Timely aggregate activity among visible non-wearers provides a large
|
||||
gain, while visibility alone and stale activity do not. However, preserving
|
||||
person-resolved history gives no advantage over destroying its cross-time
|
||||
identity assignment, and both trail the aggregate representation. The learned
|
||||
method must therefore first recover useful local floor evidence; person-track
|
||||
continuity is retained only if subsequent no-ID and broken-track controls
|
||||
separate.
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Exploratory Qwen2.5-Omni-3B local-branch evidence on EasyCom test
|
||||
Sessions 1--3. Values are percentages from one seed, not a cross-backbone or
|
||||
confirmatory result.}
|
||||
\label{tab:preliminary}
|
||||
\centering
|
||||
\small
|
||||
\resizebox{\columnwidth}{!}{%
|
||||
\begin{tabular}{lccc}
|
||||
\hline
|
||||
Model & 0.2-s mAP & 0.2-s wearer AP & Mean mAP\\
|
||||
\hline
|
||||
Qwen-3B $G$+local A & 81.72 & 85.07 & 65.11\\
|
||||
Qwen-3B $G$+local AV-pool & 82.03 & 85.09 & 65.40\\
|
||||
\hline
|
||||
$\Delta$ visual & +0.30 & +0.02 & +0.29\\
|
||||
\hline
|
||||
\end{tabular}}
|
||||
\end{table}
|
||||
|
||||
Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
|
||||
Pooled vision gives a small gain, while immediate wearer AP is nearly
|
||||
unchanged. This is compatible with either useful generic local context or
|
||||
regularization; it is not evidence of person grounding. Only matched-recall
|
||||
event evaluation and visual-alignment controls can distinguish these cases.
|
||||
A separate Qwen-3B dense-cache diagnostic gave 58.16 mean mAP for its frozen
|
||||
readout and 56.20 for one LoRA recipe. Because that diagnostic used a different
|
||||
feature protocol, it is not combined numerically with
|
||||
Table~\ref{tab:preliminary}; it only shows why a fair, protocol-matched PEFT
|
||||
study remains necessary.
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Planned cross-backbone EasyCom event evaluation. Lower
|
||||
$\mathrm{FP}_{\mathrm{mo}}$ is better; all rows use the same
|
||||
development-selected operating protocol. ``Local'' denotes the train-side
|
||||
selected common local-evidence candidate.}
|
||||
\caption{Confirmatory EasyCom results. Lower multi-other false entries are
|
||||
better; all event rows use a training-side selected operating point.}
|
||||
\label{tab:main}
|
||||
\centering
|
||||
\small
|
||||
\resizebox{\columnwidth}{!}{%
|
||||
\begin{tabular}{lccccc}
|
||||
\hline
|
||||
Backbone/protocol & $\mathrm{FP}_{\mathrm{mo}}$/min@$R=.8$ & Onset R &
|
||||
Onset latency & Yield R & Mean mAP\\
|
||||
Method & 0.2-s mAP & Mean mAP & FP/min@$R=.8$ & Onset lat. & Yield R\\
|
||||
\hline
|
||||
Qwen-3B frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Qwen-3B PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Qwen-3B + local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Qwen-7B frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Qwen-7B PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
MiniCPM-o frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
MiniCPM-o PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
EgoSpeak reproduction & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
$G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
$G+$matched temporal local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
$G+$causal local AV (ours) & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||
\hline
|
||||
\end{tabular}}
|
||||
\end{table}
|
||||
|
||||
\todo{Report Table~\ref{tab:main} as mean$\pm$standard deviation over three
|
||||
seeds, plus 95\% paired cluster-bootstrap confidence intervals for frozen vs
|
||||
PEFT and frozen vs local within each backbone. A general omni-representation
|
||||
claim requires a consistent pattern across at least two backbone families; an
|
||||
effect confined to Qwen must be stated as Qwen-specific. The behavior claim is
|
||||
supported only if multi-other FP/min decreases at matched wearer-onset recall
|
||||
without worse onset latency or wearer-yield recall. If the selected local
|
||||
candidate uses tracks, also report its paired intervals against no-ID and
|
||||
temporal-capacity controls.}
|
||||
\textbf{Information opportunity.}
|
||||
On development Session~12, an analysis-only seed-0 oracle supplied perfectly
|
||||
timed visible-other speech activity through an explicit non-deployable path.
|
||||
Aggregate visible activity improved 0.2-s mAP by 6.97 points and ten-horizon
|
||||
mean mAP by 2.91 points over matched global-plus-local-audio. Visibility-only
|
||||
and one-second-delayed activity remained near audio, while per-person histories
|
||||
did not outperform identity-shuffled histories. This motivates timely local
|
||||
floor evidence, not participant identity. \todo{After seeds 1--2, report the
|
||||
interval or remove the exact oracle numbers from the four-page paper.}
|
||||
|
||||
\textbf{Main result.}
|
||||
Table~\ref{tab:main} should establish one result: \todo{the selected local
|
||||
adapter reduces multi-other false entry at matched valid-onset recall, retains
|
||||
onset latency/yield behavior, improves dense forecasting, and recovers a
|
||||
meaningful fraction of the oracle opportunity}. The 7B row should determine
|
||||
whether \todo{capacity alone closes the gap or the adaptation structure remains
|
||||
useful at larger scale}. No conclusion is drawn across all Omni families from
|
||||
Qwen-only evidence.
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Conditional track-continuity counterfactual on the best track-aware
|
||||
model. This table is retained only if train-side selection chooses such a
|
||||
candidate; all rows reuse one frozen checkpoint.}
|
||||
\label{tab:counterfactual}
|
||||
\caption{Mechanism controls for the selected Omni-3B local adapter.}
|
||||
\label{tab:ablation}
|
||||
\centering
|
||||
\small
|
||||
\resizebox{\columnwidth}{!}{%
|
||||
\begin{tabular}{lccc}
|
||||
\hline
|
||||
Track/visual evidence & $\mathrm{FP}_{\mathrm{mo}}$/min & Onset recall & 0.2-s mAP\\
|
||||
Local condition & Mean mAP & FP/min@$R=.8$ & Yield R\\
|
||||
\hline
|
||||
Correct tracks & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Global ID rename & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Broken continuity & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Time shifted & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Removed & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Correct causal local AV & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Remove local vision & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Time-shift local vision & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Global/full-frame visual pool & \todo{--} & \todo{--} & \todo{--}\\
|
||||
Same-rate/capacity control & \todo{--} & \todo{--} & \todo{--}\\
|
||||
\hline
|
||||
\end{tabular}}
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
If a track-aware candidate is selected, Table~\ref{tab:counterfactual} is the
|
||||
continuity test. Global renaming must be
|
||||
numerically invariant because IDs carry no semantics. If the model exploits
|
||||
person history, correct tracks should outperform broken continuity even though
|
||||
both contain exactly the same per-frame people. If broken, time-shifted, and
|
||||
removed variants perform similarly to correct tracks, the defensible
|
||||
conclusion is additional capacity or generic visual context, not useful
|
||||
person-continuous reasoning.
|
||||
\textbf{Mechanism and limitations.}
|
||||
Table~\ref{tab:ablation} should test whether gains require correctly timed local
|
||||
visual evidence rather than extra parameters, regularization, or generic scene
|
||||
context. \todo{Report paired intervals and the supported interpretation. If
|
||||
correct and shifted/removed vision do not separate, drop the audiovisual
|
||||
evidence-routing claim. If dense mAP improves but event behavior does not, do
|
||||
not claim improved floor control. If legitimate onset/yield degrades, stop the
|
||||
adapter rather than describing lower false entry as success.}
|
||||
|
||||
\todo{Add one compact error analysis split by model family, track visibility,
|
||||
number of visible people, overlap, track age/fragmentation, and
|
||||
backchannel/interruption proxy. Include event counts with every rate. Move
|
||||
rejected PEFT recipes and fusion candidates to supplementary material unless
|
||||
they directly explain the selected method.}
|
||||
|
||||
\textbf{Qualitative evidence.}
|
||||
For a multi-other exchange, visualize annotation-derived per-person activity
|
||||
as an analysis reference, provided track IDs when applicable, wearer ground
|
||||
truth, the frozen-global and selected-local wearer-entry scores, and the single
|
||||
frozen threshold on one physical time axis. If the selected candidate is
|
||||
track-aware, also show its broken-continuity score. A useful
|
||||
example shows the frozen baseline rising at an other-speaker boundary while
|
||||
the selected adapter remains below threshold; a companion positive example
|
||||
must show that a real wearer onset is not suppressed.
|
||||
|
||||
\todo{Do not hand-pick only successful clips. Before viewing the final test
|
||||
plots, freeze a deterministic selection rule and show: largest paired
|
||||
improvement, near-median improvement, one failure, and one correctly retained
|
||||
wearer onset. Include full local timelines and source frame times, not cropped
|
||||
screenshots without context. State that examples illustrate the aggregate
|
||||
event result and cannot establish it alone.}
|
||||
\todo{Add one compact, pre-selected qualitative timeline only if space permits:
|
||||
one multi-other false-entry improvement paired with one preserved real wearer
|
||||
onset or one failure. Examples illustrate aggregate rates and never establish
|
||||
the claim by themselves.}
|
||||
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
We study when an egocentric agent should enter or yield a multiparty floor, not
|
||||
merely whether someone will speak. The working decomposition is global Omni
|
||||
context plus causal local audiovisual floor evidence. The current
|
||||
Qwen2.5-Omni-3B oracle audit supports the second term: timely aggregate
|
||||
visible-other activity is informative beyond the global state and recent
|
||||
mixed audio. It does not support the stronger claim that participant identity
|
||||
or track continuity adds value, because per-person histories do not beat an
|
||||
identity-shuffled control. Dense activity forecasting supplies a comparable
|
||||
target, while matched-recall event evaluation must determine whether a learned
|
||||
local estimator actually prevents intrusions during multi-other exchanges
|
||||
without suppressing valid wearer entry or yield. \todo{After the controlled
|
||||
study, state which learned local variant was selected, how much of the oracle
|
||||
gap it recovered, whether the pattern generalized beyond Qwen, the
|
||||
quantitative false-entry and onset/yield results, and the main deployment
|
||||
limitation. If no learned variant passes these tests, conclude with the
|
||||
information audit rather than claiming a track-based solution.}
|
||||
This study asks whether a general-purpose Omni representation can be adapted
|
||||
for causal floor forecasting of an invisible egocentric wearer. The candidate
|
||||
method separates long-context Omni understanding from short-window local floor
|
||||
evidence without assuming persistent speaker identity. \todo{State the final
|
||||
learned-method improvement, matched-recall false-entry result, onset/yield
|
||||
non-degradation, causal-control evidence, and main limitation. If no learned
|
||||
variant passes these tests, reframe the submission as an information audit or
|
||||
do not make the method claim.}
|
||||
|
||||
\bibliographystyle{IEEEbib}
|
||||
\bibliography{references}
|
||||
|
||||
Reference in New Issue
Block a user