wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add competing-risk evaluation and align ICASSP story

Browse Source
This commit is contained in:
Xin Wang
2026-09-10 11:12:02 +08:00
parent 0abefd6f77
commit 4b7d2c0c20
30 changed files with 3420 additions and 724 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
manuscript/main/README.md
View File

@@ -25,32 +25,28 @@ main/
build/ # generated, ignored
```
The author/affiliation, final results, Method, Experimental Setup, Results and
Conclusion are still explicit TODOs. Do not turn planned experiments into
reported findings.
The manuscript is intentionally a content skeleton with visible TODOs. Do not
turn planned implementations, controls, or results into completed claims.
The current paper thesis is model-agnostic and deliberately narrower than
addressee prediction. It asks whether general-purpose omni representations
expose the fine-grained evidence needed for 200 ms wearer floor-action
projection, whether PEFT can recover it, and whether causal local audio/person
evidence provides a reliable complement. The initial planned backbone set is:
The frozen paper story is documented in
[`../../docs/icassp2027_acceptance_storyline_2026-09-10.md`](../../docs/icassp2027_acceptance_storyline_2026-09-10.md).
The paper asks how long-context Omni understanding can be adapted into
sub-second, wearer-relative floor control for an invisible egocentric wearer.
Its single candidate contribution is a causal local floor-evidence adapter:
- Qwen2.5-Omni-3B;
- Qwen2.5-Omni-7B;
- MiniCPM-o 4.5.
- Qwen2.5-Omni-3B is the primary global backbone;
- matched LoRA/PEFT is a baseline, not a separate contribution;
- Qwen2.5-Omni-7B is an optional capacity-scaling control;
- the primary local method uses an anonymous, no-ID visible-person set;
- track continuity is conditional and must defeat no-ID/shuffled/capacity
controls before entering the paper;
- MiniCPM-o and a cross-family sweep are removed from the core plan;
- the oracle is analysis-only motivation, not a deployable method or headline;
- the publication gate is lower competing-risk-censored multi-other false
entry at matched wearer-onset recall without worse valid onset or yield.
The backbone and final fusion method are **not frozen yet**. Current Qwen-3B
numbers are exploratory motivation, not evidence that every omni model has the
same limitation. Compare frozen, PEFT, and local-evidence protocols within each
backbone; report common local encoders separately from model-native local
encoders. Raw cross-model scores are secondary because the models expose
different streaming/readout interfaces.
Track-consistent person fusion remains one candidate after pooled and
capacity-matched controls. If selected, dataset participant IDs route
short-term causal state only; they are not semantic inputs. The publication
gate is an event-level reduction in multi-other false entries at matched
wearer-onset recall without degrading onset latency or wearer-yield behavior.
A general omni claim requires a consistent result across at least two backbone
families; otherwise the paper must state a model-specific finding. Selected
examples are explanatory evidence only.
Competing-risk censoring and training-side threshold calibration are now
implemented and unit-tested. The remaining P0 blocker is the real-data event
audit and protocol/scenario freeze; until then, event values are timing-derived
proxies. Already viewed Sessions 1--3 numbers are exploratory internal evidence
and must not appear in the confirmatory paper table.

110
manuscript/main/REVIEWER_CHECKLIST.md
View File

@@ -2,13 +2,11 @@
This is an internal execution checklist. It is not part of the submitted PDF.
## Reviewer verdict on the current draft
## Reviewer verdict and frozen paper shape
The potentially publishable question is now model-agnostic: **do
general-purpose omni representations expose the fine-grained evidence needed
for 200 ms egocentric floor action, and can causal local evidence reduce false
wearer entries at matched onset recall when frozen readout or PEFT is
insufficient?**
The potentially publishable question is: **can a general-purpose Omni readout
be adapted into 200 ms, wearer-relative floor control by explicitly preserving
short-window causal audio and anonymous visible-person evidence?**
The contribution is not “the first audio-visual multiparty turn model.” VAP,
EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible
@@ -17,17 +15,25 @@ wearer, combined with a controlled diagnosis of global omni states and a
behavior-aligned multi-other event protocol.
Current Qwen2.5-Omni-3B results are exploratory motivation only. They neither
prove a limitation shared by omni models nor establish track-consistent fusion
as the final method. The planned initial backbone set is Qwen2.5-Omni-3B,
Qwen2.5-Omni-7B, and MiniCPM-o 4.5.
prove a limitation shared by Omni models nor establish track-consistent fusion
as the final method. Qwen2.5-Omni-3B is primary; Qwen2.5-Omni-7B is an optional
capacity control. MiniCPM-o and a cross-family sweep are outside the core paper.
## P0 — Freeze the task before the backbone sweep
The four-page paper has only three contributions: temporal--role adaptation
mismatch, one causal local floor adapter, and causal behavioral evidence.
LoRA, the oracle, and the event protocol are respectively a baseline,
motivation, and validation mechanism.
## P0 — Freeze the task before confirmatory adaptation runs
- [ ] Define the exact 5 Hz causal cutoff and physical target interval for each
of the ten 0.2 s horizons.
- [ ] Freeze wearer-onset/yield and multi-other exchange definitions.
- [ ] Add competing-risk rules: a candidate event ends at the first matched
wearer action, incompatible floor change, timeout, or invalid source boundary.
- [x] Implement and unit-test competing-risk rules: a response candidate ends
before the next other onset and a yield candidate before the next other
offset; right-censored and invalid windows are counted explicitly.
- [ ] Audit the competing-risk implementation on real timelines and freeze the
timeout/source-boundary convention before confirmatory reporting.
- [ ] Specify overlap, minimum silence, maximum response delay, duplicate
detections, source discontinuities, track visibility, offscreen speakers,
and whether target-of-speech strata are diagnostic only.
@@ -41,9 +47,9 @@ Qwen2.5-Omni-7B, and MiniCPM-o 4.5.
least 100 randomly selected events. Publish the number of unique other IDs and
multi-other opportunity minutes before the main sweep.
## P1 — Audit each backbone interface
## P1 — Audit the primary backbone interface
For Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o 4.5, record:
For Qwen2.5-Omni-3B, and for 7B only if used as the scaling control, record:
- [ ] exact model/code revision, processor, dtype, and hardware;
- [ ] native audio/video sampling and streaming/chunk semantics;
@@ -57,9 +63,9 @@ a model cannot expose a comparable causal state, report that protocol as
unsupported. Raw scores are not a pure architecture ranking; emphasize
within-backbone deltas.
## P2 — Establish the compact representation matrix
## P2 — Establish the compact adaptation matrix
Run three seeds per supported backbone with the same data, physical cutoffs,
Run three seeds for the primary backbone with the same data, physical cutoffs,
target, prediction-head capacity, optimizer exposure, and evaluator:
1. Frozen global causal state.
@@ -67,41 +73,39 @@ target, prediction-head capacity, optimizer exposure, and evaluator:
3. Global state plus recent local audio.
4. Global state plus pooled local audio and vision.
Use a common local encoder as the primary cross-backbone control. If native
local encoders are tested, report them in a separate within-model panel. Do not
mix common-encoder and native-encoder deltas in one claim.
Add a parameter- and history-matched temporal local branch before attributing
gain to audiovisual organization. Use the same local adapter for the optional
7B scaling row.
**Gate:** baseline standard deviations are known; the evaluator reproduces
identical scores from saved predictions; parameter counts and training budgets
are reported. A single failed LoRA recipe is evidence about that recipe, not
proof that the model cannot adapt.
## P3 — Select local fusion only after P2
## P3 — Train the no-ID local floor adapter
If local evidence gives a reliable event-level gain, compare candidates in
increasing complexity:
The primary method is a short-window causal encoder over recent mixed audio and
per-frame anonymous visible-person sets. It uses no participant-ID embedding or
cross-time track state. Compare it with:
1. Pooled local audio/vision.
2. Parameter-matched temporal branch.
3. Per-frame person set with no cross-time identity.
4. Shared track-wise temporal encoder.
5. Person–audio cross-attention.
6. Optional per-person active-speaker auxiliary loss, only after selecting a
person-aware activity-only model.
1. Recent local audio only.
2. Global/full-frame visual pooling.
3. A parameter- and history-matched temporal branch.
4. Time-shifted local vision.
5. Removed local vision.
6. A same-rate control if the local path runs faster than 5 Hz.
Keep the dense EgoSpeak wearer/others target as the primary task. Target of
speech is analysis-only and never model input. Describe only the train-side
winner as the final method; move rejected candidates to ablations.
speech is analysis-only and never model input. Use ``dual-rate'' only if the
local input/update rate differs explicitly and defeats the same-rate control.
For a track-aware candidate, temporary IDs route hidden state only: no ID
embedding, cross-recording identity, or fixed seating slot. Unit-test variable
person count, empty sets, padding, slot permutation, global ID renaming,
Unit-test variable person count, empty sets, padding, person-set permutation,
source/gap resets, and strict causal cutoffs.
## P4 — Run visual counterfactuals conditionally
## P4 — Consider track continuity only after the no-ID method wins
Only if a track-aware candidate is selected, evaluate one frozen checkpoint
with:
Only if a track-aware candidate then beats the no-ID method on training-side
data, evaluate one frozen checkpoint with:
- correct person tracks;
- clip-consistent global renaming, which must be invariant;
@@ -121,11 +125,14 @@ retrain test-time counterfactuals.
- [ ] Operating points at wearer-onset recall 0.7, 0.8, and 0.9; headline 0.8.
- [ ] False wearer entries per multi-other opportunity minute, onset latency,
wearer-yield recall/latency, and original ten-horizon AP/mAP.
- [ ] Error strata: backbone, visibility, number of visible people, overlap,
offscreen activity, track age/fragmentation, and transition proxy.
- [ ] Error strata: visibility, number of visible people, overlap, offscreen
activity, and transition proxy. Add track age/fragmentation only if a
track-aware method is selected.
- [ ] Ego4D AVD coverage audit on the exact EgoSpeak split before claiming the
same multi-other analysis.
- [ ] Provided-track and online-tracker conditions named separately.
- [ ] Name the causal box source and its limitations. Add separate
provided-track/online-tracker rows only if tracking enters the selected
method.
Qualitative cases follow a frozen selection rule and include the largest paired
improvement, a median case, a failure, and one preserved true wearer onset.
@@ -134,23 +141,20 @@ Examples explain rates; they never replace them.
## Paper assembly order
1. Freeze and audit the event protocol.
2. Implement and causally audit all backbone readouts.
3. Complete the compact frozen/PEFT/local matrix.
4. Decide whether the paper can support a cross-family or only model-specific
representation claim.
5. Select a local fusion candidate using training-side event metrics.
6. Run conditional visual/track counterfactuals.
7. Produce the cross-backbone event table and paired intervals.
8. Draw one compact final system figure; remove rejected methods from the main
2. Audit the Qwen2.5-Omni-3B readout and matched PEFT baseline.
3. Complete the frozen/PEFT/audio/matched-temporal/local-AV matrix.
4. Run correct/time-shift/remove-vision controls and the optional 7B scaling row.
5. Consider track continuity only if the no-ID method already passes.
6. Produce one confirmatory event table and one mechanism-ablation table.
7. Draw one compact final system figure; remove rejected methods from the main
narrative.
9. Replace every visible TODO and verify the ICASSP 2027 kit/page limit.
8. Replace every visible TODO and verify the ICASSP 2027 kit/page limit.
## Stop/go decisions
- **General claim:** at least two backbone families show the same frozen/PEFT
weakness and a reliable local-evidence improvement.
- **Model-specific claim:** the effect appears only within Qwen; say so and do
not generalize to all omni models.
- **Claim boundary:** with Qwen-only evidence, describe a Qwen2.5-Omni case
study and the tested adaptation mechanism; do not generalize to all Omni
models.
- **Audio-local reframe:** local audio improves but all visual variants fail;
claim a fine-grained acoustic readout bottleneck, not social visual reasoning.
- **Track-aware method:** correct continuity beats pooled vision, temporal

787
manuscript/main/main.tex
View File

@@ -1,7 +1,6 @@
% Internal ICASSP 2027 draft using the official ICASSP 2026 template bundled
% in this repository. Replace the class/style files when the 2027 author kit
% is released, and remove every visible \todo{} marker before submission.
% Compile from this directory with: make
% ICASSP 2027 content skeleton. The final paper must use the official 2027
% author kit and contain no visible \todo{} markers. Planned experiments are
% intentionally marked rather than presented as completed work.
\documentclass{article}
\usepackage{spconf}
\usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}
@@ -10,7 +9,8 @@
\newcommand{\vect}[1]{\boldsymbol{#1}}
\setlength{\emergencystretch}{2em}
\title{Adapting Omni Models for Causal Egocentric Multiparty Floor-Entry Forecasting}
\title{Is It My Turn? Causal Local Evidence Adaptation of Omni Models for
Egocentric Floor Forecasting}
% TODO: Insert the final author list and affiliations after the ICASSP 2027
% review policy and author order are finalized.
@@ -22,603 +22,340 @@
\maketitle
\begin{abstract}
In multiparty conversation, a pause need not invite an egocentric participant
to speak: it may instead mark a handover between two other people. We study
causal egocentric multiparty floor-entry forecasting, where the target is the
future speech participation of the first-person camera wearer---the one
participant who is necessarily absent from the visible speaker set. This
wearer-relative target distinguishes our setting from recent multimodal VAP
models that forecast visible floor holders or observe human--human turn
dynamics. Following EgoSpeak, we use causal first-person audio--video and the
human wearer's future speaking behavior as an imitation target, and ask how a
jointly pretrained audio--visual Omni model should be adapted for predictions
every 200~ms. Our global--local formulation combines a causal Omni readout with
a lightweight pathway for recent mixed audio and visible-person observations.
On EasyCom development data, a single-seed oracle analysis finds that perfectly
aligned aggregate activity among visible non-wearers improves 0.2-s mAP by
6.97 points and ten-horizon mean mAP by 2.91 points over global Omni plus recent
audio, whereas person-resolved histories do not outperform an
identity-shuffled control. This motivates timely local floor evidence without
assuming that person continuity is useful. The full study compares frozen,
parameter-efficient, and learned local adaptation and evaluates false wearer
entry at matched wearer-onset recall together with yield behavior.
\todo{After the train-side study is locked, replace the final two sentences
with the selected learned local variant and insert: its oracle-gap recovery,
multi-other false-entry reduction at 80\% onset recall, onset/yield
non-degradation, and 95\% intervals. Mention track continuity only if its
controls separate.}
Real-time audiovisual agents must decide not only what to say, but whether an
emerging floor opportunity belongs to them. This is difficult in multiparty
egocentric interaction: the target wearer is behind the camera, while the
visible people provide only indirect evidence about whether the surrounding
floor remains occupied. We study strictly causal forecasting of the wearer's
future speech activity every 200~ms. Although a general-purpose Omni model
provides rich long-context audio--visual representations, its global readout is
not explicitly trained to expose sub-second, wearer-relative floor evidence.
We therefore study a candidate causal local floor adapter that combines the global
Omni state with short-window mixed audio and an anonymous set of visible-person
observations, without assuming persistent speaker identity. On EasyCom,
\todo{report the final three-seed dense and competing-risk-censored event
results: false-entry reduction at matched wearer-onset recall, onset latency,
yield behavior, confidence intervals, and oracle-gap recovery}. Controlled
\todo{time-shift/remove-vision/same-rate or capacity-matched} interventions
show that \todo{state the supported mechanism, without claiming addressee or
speaker identity}. These results demonstrate \todo{the narrow conclusion
actually supported by the final experiments}.
\end{abstract}
\begin{keywords}
conversational agents, turn-taking, speech initiation, egocentric video,
omni models
conversational agents, turn-taking, egocentric video, multimodal adaptation,
voice activity projection
\end{keywords}
\section{Introduction}
\label{sec:intro}
Natural turn-taking requires a conversational system to anticipate not only
when speech will stop, but whether an emerging floor opportunity actually
belongs to it. This distinction is particularly important in multiparty
interaction. In a dyadic conversation, a pause after one interlocutor finishes
often provides an opportunity for the other to respond; in a group
conversation, the same pause may simply mark a handover between two other
participants. An egocentric conversational agent that reacts only to silence
or aggregate speech activity may therefore enter the floor while the
surrounding conversation should continue without it. We study this problem as
\emph{egocentric multiparty floor-entry forecasting}: continuously predicting
the near-future speech participation of the first-person participant from
causal audio--visual observations.
when speech will stop, but whether a floor opportunity actually belongs to it.
In a dyadic exchange, a pause after one interlocutor finishes often invites the
other to respond. In a group conversation, the same pause may instead mark a
handover between two other participants. An egocentric agent that reacts only
to silence or aggregate speech may therefore interrupt a conversation that
should continue without it. We study \emph{egocentric floor forecasting}:
continuously predicting the near-future speech participation of the
first-person wearer from strictly causal audio--visual observations.
Voice Activity Projection (VAP) provides a predictive formulation of
turn-taking by continuously projecting the joint future voice activity of
conversational participants, from which turn shifts, holds, and backchannels
can be inferred \cite{ekstedt2022vap}. EgoSpeak extends predictive speech
initiation to the first-person setting, using causal, untrimmed egocentric
audio--visual observations and the future speaking behavior of the camera
wearer as supervision \cite{kim2025egospeak}. This behavior is a useful
imitation target for the perceptual decision faced by an egocentric agent:
given everything observed so far, is the first-person participant likely to
enter the floor next? EgoSpeak establishes that this prediction can be learned
from real-world egocentric streams, but leaves open how recent jointly
pretrained audio--visual Omni models should be used for such fine-grained
causal forecasting.
Voice Activity Projection (VAP) introduced continuous prediction of future
voice activity for turn-taking \cite{ekstedt2022vap}. EgoSpeak brought this
formulation to causal, untrimmed first-person streams, using future camera-
wearer activity as an imitation target \cite{kim2025egospeak}. Complementary
work adds visual gaze, head pose, and facial behavior to predictive turn-taking
\cite{russell2025mmvap}, while MuVAP combines global acoustic dynamics with
visible face tracks to forecast visible floor holders in multiparty scenes
\cite{qi2026muvap}. Our target has a different role structure: the participant
whose future action is predicted is the one participant absent from the
visible set. The visible people are contextual evidence for the unseen
wearer's decision, not candidate output identities.
In parallel, recent work has expanded predictive turn-taking along multimodal
and multiparty dimensions. MM-VAP incorporates facial expression, head pose,
and gaze into predictive turn-taking for two-party visual interaction
\cite{russell2025mmvap}. A recent social-robot MM-VAP framework adapts
voice-activity-related audio--visual encoders for anticipating human--human
turn dynamics in observer or mediation settings \cite{cano2026mmvap}. MuVAP
further extends VAP to multiparty interaction using a monaural mixed-audio
stream and a single camera, grounding acoustic dynamics in visible face tracks
and predicting current and future visible floor holders \cite{qi2026muvap}.
These formulations are complementary to our egocentric setting. In MuVAP, the
candidate future speakers belong to the set visible to the camera; in
mediation-oriented MM-VAP, the system observes turn dynamics between human
participants. Here, by contrast, the participant whose future floor entry must
be predicted is located \emph{behind the camera} and is therefore absent from
the visible participant set.
This role asymmetry exposes an adaptation problem for general-purpose Omni
models. Omni pretraining favors globally integrated representations useful for
multimodal understanding and generation. Egocentric floor control instead
depends on timely prosodic and visible-person activity at sub-second scale,
including whether other participants continue to occupy the floor. Increasing
model capacity or applying a generic parameter-efficient update does not
explicitly preserve or route this evidence to a dense wearer-relative
prediction head. The open question is therefore not merely whether an Omni
backbone can be applied to EgoSpeak, but what task structure is still required
to turn general audiovisual understanding into reliable floor control.
This leaves an underexplored intersection: continuous causal prediction of an
egocentric participant's own future floor participation in a changing
multiparty scene using a general-purpose audio--visual Omni model. The defining
difference is its viewpoint and prediction target: the model must project the
activity of the unseen first-person participant from the causal stream used to
observe everyone else. We investigate how pretrained Omni representations can
be adapted under the EgoSpeak wearer-relative forecasting objective. Current
Qwen2.5-Omni results motivate this study, but do not establish a limitation
shared by Omni models in general.
Our candidate decomposition separates long-context understanding from
short-window floor evidence. A
causal global Omni state represents the broader scene and conversation. In
parallel, a lightweight local path encodes recent mixed audio and a changing,
anonymous set of visible-person observations. A residual adapter makes this
local evidence accessible to the common ten-horizon activity head. The primary
variant deliberately avoids cross-time identity: the current development
oracle supports timely aggregate visible-other activity but not person
continuity. Track-aware structure is considered only if it defeats no-ID,
identity-corrupted, and capacity-matched controls.
Our working hypothesis separates broad context from timely floor evidence. A
causal global Omni state can represent the scene and conversational history,
while a lightweight local pathway preserves recent mixed audio and a changing
set of visible-person observations. The latter may be aggregated per frame or
organized into anonymous tracks; track continuity becomes a contribution only
if it outperforms aggregate, no-ID, shuffled, and capacity-matched controls. An
analysis-only oracle screen on held-out EasyCom Session~12 supports the weaker
hypothesis: perfectly timed aggregate visible-other activity improves a
matched global-plus-audio model by 6.97 points at 0.2~s and 2.91 points in
ten-horizon mean mAP, whereas person-resolved histories do not outperform an
identity-shuffled control. Thus the current evidence motivates timely local
floor information, not speaker identity or addressee understanding.
We retain EgoSpeak-compatible dense future-activity prediction for
comparability and add a behavior-aligned event analysis. In particular,
per-frame average precision can hide the failure that motivates this work:
entering during an exchange among other participants. We therefore measure
false wearer entries during multi-other opportunities at matched wearer-onset
recall, together with response latency and wearer-yield behavior. These events
are observable timing-derived interaction proxies, not semantic EOT, INT, or
addressee labels; the final protocol uses competing-risk censoring so an event
cannot claim a later action across an incompatible floor change. This follows
the broader event-oriented emphasis on recall, false alarms, and latency
\cite{jiang2026turnbench}.
Our contributions are: (1) a controlled study of frozen and
parameter-efficient Omni adaptation for EgoSpeak-style causal forecasting;
(2) an oracle information audit that distinguishes timely aggregate local
floor evidence from person continuity; (3) a global--local adaptation selected
through causal, alignment, and capacity-matched controls; and (4) an event
protocol that tests whether dense gains reduce false first-person floor entry
without suppressing valid entry or yield. EasyCom supplies synchronized
egocentric audio--video, per-person tracks, and voice activity
\cite{donley2021easycom}; target-of-speech annotations, when used, serve only
as analysis strata and never as model input or required output.
The final paper will claim the following three contributions only if the
corresponding TODO gates pass:
\begin{itemize}
\item We identify and test a temporal--role mismatch between a global Omni
readout and causal egocentric floor forecasting for an invisible wearer.
\item We evaluate a wearer-relative causal local floor adapter that combines
long-context Omni information with short-window audio and anonymous
visible-person evidence.
\item We evaluate whether the adapter reduces false wearer entries during
multi-other exchanges at matched valid-onset recall, using causal alignment
interventions and competing-risk-censored entry/yield analysis.
\end{itemize}
\begin{figure}[t]
\centering
\fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
\todo{After model selection, draw the final system figure. Show the shared
interface: raw causal AV to a global Omni readout; a parallel recent-audio and
visible-person path; aggregate/no-ID/track-organized variants; residual fusion;
and the common $10\times3$ head. Draw the transcript-derived oracle only as a
dashed, analysis-only upper-bound path. Mark causal cutoffs, provided versus
online tracks, trainable modules, NULL/OFFSCREEN handling, and resets.}}}
\caption{Global context and causal local evidence for egocentric floor action.
Track organization is a controlled candidate, not a presupposed source of
speaker identity.}
\fbox{\parbox[c][1.00in][c]{0.94\columnwidth}{\centering
\todo{Draw only the selected method: causal AV to the long-context Omni path;
the short-window mixed-audio and anonymous person-set path; residual fusion;
the common $10\times3$ head; and the wearer-silent/active event interpretation.
Mark physical cutoffs, trainable modules, person masks, and NULL/OFFSCREEN.
Do not draw oracle labels, rejected track variants, or multiple backbones.}}}
\caption{Long-context Omni understanding and short-window local floor evidence
for wearer-relative future-activity prediction.}
\label{fig:method}
\end{figure}
\section{Methods}
\section{Method}
\label{sec:method}
\subsection{Wearer-relative activity projection}
\subsection{Causal wearer-relative projection}
At decision time $t$, the model observes only the causal stream
$\mathcal{X}_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s.
For bin $k$, the target
$\vect{y}_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
At decision time $t$, the model observes only the stream
$\mathcal X_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s. For
bin $k$, the multilabel target
$\vect y_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
background, camera-wearer activity, and activity by one or more other people.
Wearer--other overlap is allowed; hence this is not a three-way softmax. The
output is
Wearer--other overlap is allowed. The model outputs
\begin{equation}
\hat{\vect{Y}}_t=f_\theta(\mathcal{X}_{\leq t})\in[0,1]^{K\times3},
\qquad q_t=\hat y^{\mathrm{w}}_{t,1},
\label{eq:task}
\hat{\vect Y}_t=f_\theta(\mathcal X_{\leq t})
\in[0,1]^{K\times3}.
\label{eq:task}
\end{equation}
where $q_t$ is the immediate wearer-entry score used by the event protocol.
It is interpreted as an \emph{entry} score only in wearer-silent opportunity
windows; elsewhere it remains a future-activity probability. This retains the
EgoSpeak target and evaluation while exposing a deployable 5-Hz decision
signal.
The same wearer-activity forecast has two controller interpretations. In a
wearer-silent opportunity, $\hat y^{\mathrm w}_{t,1}$ is an entry score. While
the wearer is active, $1-\hat y^{\mathrm w}_{t,1}$ is a near-term yield score.
The current self-state is used to define these observable event strata, not as
an unreported oracle model input. \todo{If the final adapter explicitly uses
self-state, define its strictly causal and deployment-available source and add
a matched no-state ablation; otherwise keep self-state evaluation-only.}
\subsection{Omni backbones and representation protocols}
\subsection{Global Omni baseline}
Let $m$ index an omni backbone and let $\vect{g}^{(m)}_t$ be a
backbone-specific causal global readout at the same physical cutoff $t$.
Qwen2.5-Omni \cite{xu2025qwenomni} is the currently implemented family;
Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o~4.5
\cite{cui2026minicpmo45} form the planned initial comparison. These models have
different native streaming APIs, temporal tokenization, hidden dimensions, and
adaptation interfaces. We therefore emphasize within-backbone deltas rather
than treating raw cross-model scores as a pure architecture ranking.
Let $\vect g_t\in\mathbb R^{d_g}$ be the causal global readout of
Qwen2.5-Omni at the physical cutoff $t$ \cite{xu2025qwenomni}. The frozen
baseline trains only a dimension-matched prediction head. A matched PEFT/LoRA
baseline updates a pre-registered subset of the Omni model under the same
examples, context, optimizer budget, head, and evaluator. Its purpose is to
test whether generic parameter adaptation is sufficient; one negative recipe
is not interpreted as a general limitation of LoRA or all Omni models.
Every supported backbone is evaluated under three representation protocols:
(i) a frozen global readout with a dimension-matched prediction head;
(ii) a pre-registered parameter-efficient adaptation budget; and (iii) the
same global readout augmented by causal local evidence. Input cutoffs, EgoSpeak
targets, context duration, training examples, head capacity, and evaluation
are held fixed whenever the backbone permits it. A failure of one LoRA recipe
is reported as that recipe's result, not as proof that the backbone cannot
adapt.
\todo{Freeze the exact Qwen2.5-Omni-3B revision, processor, input rates,
streaming/cache semantics, readout layer, context, dtype, LoRA targets/rank,
trainable parameter budget, and cutoff audit. Add Qwen2.5-Omni-7B only as a
capacity-scaling control. Do not add another backbone family unless it is
needed to support and experimentally earns a cross-family claim.}
\todo{Implement and audit one adapter for each planned backbone. For every
model, document model revision, processor, native audio/video rates, exact
hidden-state readout, causal/streaming semantics, context limit, dtype,
trainable LoRA targets/rank, parameter count, peak memory, and P50/P95 latency.
Verify no future frames or samples enter $\vect g^{(m)}_t$. If a model cannot
expose a comparable causal state, report it as unsupported rather than inventing
a non-equivalent readout. Add future backbones only before the experiment
protocol is locked.}
\subsection{Causal local floor adapter}
\subsection{Global context and causal local evidence}
The global state $\vect g^{(m)}_t$ is responsible for broad scene and
conversational context, but is not assumed to retain every localized cue needed
for floor control. From the most
recent $L_a$ seconds, a causal local audio encoder produces $\vect a_t$. At
time $t$, provided boxes during method isolation---and an online detector and
tracker in the deployable condition---produce a variable-size anonymous set
$\mathcal P_t=\{(\vect v_{t,i},\vect b_{t,i},\vect r_{t,i})\}_{i=1}^{N_t}$,
where $\vect v_{t,i}$ is a crop feature, $\vect b_{t,i}$ is box geometry, and
$\vect r_{t,i}$ contains only causal reliability variables such as visibility,
track age, and time since observation. A shared causal track encoder updates
The candidate local path operates on the most recent $L_\ell$ seconds. At local instant
$\tau\leq t$, a causal audio encoder produces $\vect a_\tau$ from mixed audio.
For the anonymous visible-person set
$\mathcal P_\tau=\{(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau}$,
a shared visual encoder and masked set operator produce
\begin{equation}
\vect h_{t,i}=F_{\mathrm{trk}}\!\left(\vect h^-_{t,i},
W_v\vect v_{t,i}+W_b\phi(\vect b_{t,i})+W_r\vect r_{t,i}\right).
\label{eq:track}
\vect v_\tau=\mathrm{SetPool}
\left(\{E_v(\vect x_{\tau,i},\vect b_{\tau,i})}_{i=1}^{N_\tau}
\cup\{\vect v_\varnothing\}\right),
\label{eq:set}
\end{equation}
All tracks share $F_{\mathrm{trk}}$; the temporary ID only retrieves
$\vect h^-_{t,i}$ and is never embedded. The global-audio query then reads the
current person set by masked cross-attention,
\begin{align}
\vect u_t &= \mathrm{MHA}\!\left(
W_q[\vect g^{(m)}_t;\vect a_t],
[\vect h_{\varnothing};\vect h_{t,1};\ldots;\vect h_{t,N_t}]
\right), \\
\vect z^{(m)}_t &= \vect g^{(m)}_t+W_o\vect u_t,
where $\vect v_\varnothing$ handles an empty or offscreen visible set. The
operator is permutation invariant and uses no participant-ID embedding or
cross-time identity. A causal short-window encoder summarizes local dynamics,
\begin{equation}
\vect \ell_t=F_\ell
\left(\{[\vect a_\tau;\vect v_\tau]\}_{\tau\in(t-L_\ell,t]}\right).
\label{eq:local}
\end{equation}
The adapter preserves the global initialization through a zero-initialized
residual output,
\begin{equation}
\vect z_t=\vect g_t+W_o\,
\sigma\!\left(W_g\vect g_t+W_\ell\vect\ell_t\right),
\qquad W_o\vert_{\mathrm{init}}=0,
\label{eq:adapter}
\end{align}
where $\vect h_{\varnothing}$ is a learned NULL/OFFSCREEN token. Padding is
masked, so the same network handles $N_t=0$ and changing group size. For a
bounded implementation, $N_t>N_{\max}$ is resolved by a deterministic ranking
using detector confidence and track age available at $t$, and truncation is
reported. The operator is permutation equivariant
\cite{lee2019settransformer}; temporary tracker IDs provide continuity, not
identity semantics.
\label{eq:fusion}
\end{equation}
and the common head maps $\vect z_t$ to Eq.~\eqref{eq:task}. This implements an
evidence-access hypothesis rather than adding an unconstrained second model:
the global path retains broad context, while the local path preserves timely
floor evidence.
Equation~\eqref{eq:track} is a candidate organization of local evidence, not a
claim that recurrent identity is necessary. Simpler alternatives replace each
$\vect h_{t,i}$ with a current-frame observation or pool the set into one
aggregate vector. This distinction is important because the current oracle
screen favors aggregate activity and does not separate correct from shuffled
identity histories.
\todo{Freeze $L_\ell$, local audio/video rates, crop source and resolution,
set pooling, causal temporal encoder, masks, reset rules, parameter count, and
latency. If the local path runs faster than the 5-Hz global/output grid, compare
it with a same-parameter 5-Hz variant before using the term ``dual-rate.'' Use
provided causal boxes only as a clearly labeled method-isolation condition;
online detection/tracking is not a paper contribution unless actually run.}
This decomposition is informed by MuVAP: its global audio representation
queries same-frame causal ASD track representations, followed by temporal
modeling and gated global-to-speaker fusion \cite{qi2026muvap}. We retain the
useful separation between global floor dynamics and person-organized evidence,
but not its role-relative output: our fixed target is the invisible wearer's
entry/yield action, not the next visible speaker. The zero-initialized residual
output follows the conservative adapter principle used to inject visual
evidence without perturbing the base prediction at initialization
\cite{rouditchenko2024whisperflamingo}. Unlike a zero scalar gate, it permits
the residual output layer to receive a gradient immediately. Local evidence
is derived from the same causal raw media, not close microphones, transcripts,
speaker labels, global identity anchors, or future frames.
\subsection{Training}
We evaluate candidates in increasing structural complexity: pooled recent
audio/vision; a capacity-matched temporal branch; a per-frame person set with
no cross-time identity; a shared track-wise temporal encoder; and person--audio
cross-attention as in Eqs.~\eqref{eq:track}--\eqref{eq:adapter}. Only a
candidate that improves the pre-registered event metric over simpler,
aggregate, identity-shuffled, and capacity-matched controls becomes the final
method. If person continuity does not separate from those controls, the final
model remains a global-plus-local aggregate floor-evidence model. A limited latent
bottleneck is tested only if large visible sets make full person attention a
runtime bottleneck; it is not assumed to improve accuracy.
\todo{Run the candidate ladder before fixing $H_\phi$. Match trainable
parameters and effective local history. Compare a shared local encoder across
backbones (cross-model control) with each backbone's native local encoder
(within-model control); do not mix these protocols in one claimed delta.
For track candidates, specify visibility/age masks, gap/source resets,
OFFSCREEN handling, and global-rename invariance. Describe only the selected
architecture as the final method; move rejected candidates to ablations.}
\subsection{Training objectives}
The common main loss is class-balanced binary cross entropy over all horizons
and activity channels. This dense EgoSpeak target is the only supervision
required by the task. If a person-aware candidate survives the activity-only
comparison, an optional auxiliary head predicts current active-speaker state:
The primary objective is class-balanced binary cross entropy over valid
horizons and activity channels,
\begin{equation}
\mathcal{L}=\mathcal{L}_{\mathrm{act}}+
\lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}.
\mathcal L_{\mathrm{act}}=
\sum_{t,k,c}m_{t,k}w_c\,
\mathrm{BCE}(\hat y^c_{t,k},y^c_{t,k}).
\label{eq:loss}
\end{equation}
Participant ID, target-of-speech, transcript, close-microphone audio, and
future frames are never model inputs. Target-of-speech may stratify qualitative
analysis but is not a predicted variable or a requirement for the primary
multi-other event definition.
\todo{First compare all representation/adaptation protocols with activity loss
only. Then ablate +ASD for the selected person-aware candidate. Add an
event-specific wearer onset/yield loss only as a separately reported extension.
Select all coefficients and early stopping on a training-side calibration
fold; do not inspect Sessions 1--3 during selection.}
\subsection{Track source and deployment contract}
The intended inference pipeline runs a face/head detector on each incoming
frame and associates detections with an online multi-object tracker using only
observations at or before $t$. Its temporary IDs route the shared recurrent
state in Eq.~\eqref{eq:track}; they do not denote names, fixed seats, or
cross-session identities. New tracks initialize from a learned state, short
misses retain a causally aged state, and expired tracks are removed. With no
visible face, the global/audio path and NULL/OFFSCREEN token remain valid. No
future-frame smoothing, full-video identity anchor, or cross-session
recognition is permitted. EasyCom state is continued only across genuinely
adjacent source segments and reset across redacted gaps; Ego4D state is local
to a clip.
During method isolation, dataset-provided boxes and anonymous IDs form a
separate \emph{provided-track} condition. This answers whether person
structure can help under accurate localization, but is not reported as online
detector performance. The deployable \emph{online-track} condition runs the
detector/tracker above and includes its errors and latency. This separation is
important because MuVAP used random full-video identity anchors to build its
dataset tracks; that preprocessing is useful for annotation but is not a
strictly online association algorithm \cite{qi2026muvap}.
\todo{Select and freeze the face/head detector, causal tracker, thresholds,
maximum age, and $N_{\max}$ on training-side data. Report detector recall,
provided/online-track coverage, mean/maximum people, truncations, ID switches,
fragments, occlusion gaps, source resets, and P50/P95 detector--tracker latency
for both datasets. Compare provided tracks, online tracks, and controlled box
jitter/dropout/fragmentation. Until the online row is run, limit deployment
claims to the intended pipeline and label all numbers ``provided-track.''}
No transcript, target-of-speech label, close microphone, future frame, or
per-person speech label enters the deployable model. \todo{Add an active-speaker
or event auxiliary loss only if it improves the frozen primary protocol and
its label/input boundary is fully reported.}
\section{Experiments}
\label{sec:setup}
\label{sec:experiments}
\textbf{Data and splits.}
We use the EgoSpeak EasyCom protocol: Sessions 4--12 for model development and
Sessions 1--3 for final reporting. All architecture, checkpoint, threshold,
and hyperparameter choices use training-side folds only. Inputs contain the
camera stream and shared array audio available to the wearer; close-talking
microphones, per-person voice labels, and target-of-speech labels are not model
inputs. We also report aggregate activity forecasting on the EgoSpeak Ego4D
split \cite{grauman2022ego4d}. Ego4D Audio-Visual Diarization supplies
clip-local anonymous participant tracks; its per-person voice coverage on the
433-clip EgoSpeak split must be audited before reporting the same multi-other
event analysis.
\textbf{Data and split.}
We follow the EgoSpeak EasyCom split: Sessions 4--12 for training-side model
development and Sessions 1--3 for final reporting \cite{donley2021easycom}.
Thresholds, windows, checkpoints, loss weights, and early stopping are selected
without using the official test sessions. Already observed test results are
treated as exploratory internal evidence and are not included in the paper.
\todo{Report duration, frame/event counts, class prevalence, visibility and
box coverage, source discontinuities, and the frozen calibration fold.}
\textbf{Oracle information audit.}
Before selecting a learned local module, we measure what information could in
principle complement the global state. Analysis-only sidecars expose
transcript-derived speech activity no later than the current cutoff under six
conditions: local audio, visibility only, one-second-delayed activity, timely
aggregate visible activity, per-person histories, and histories with their
cross-time identity assignments shuffled. These labels never enter a
deployable model. The audit separates three hypotheses that ordinary feature
ablation conflates: the value of seeing people, the value of knowing current
visible-other activity, and the incremental value of preserving who produced
that activity.
\textbf{Baselines.}
The confirmatory comparison contains: an EgoSpeak reproduction; the frozen
Qwen2.5-Omni-3B global state $G$; matched LoRA/PEFT; $G$ plus recent local
audio; a parameter- and history-matched temporal local branch; and the proposed
audio--visible-person adapter. Qwen2.5-Omni-7B is a capacity-scaling control,
not a second paper storyline. \todo{Freeze all baseline implementations and
ensure the EgoSpeak comparison uses identical targets, split, causal validity,
and evaluation.}
\textbf{Backbone comparison.}
The initial study covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and
MiniCPM-o~4.5. For each backbone $m$, we first compare its frozen causal state
$G_m$, a parameter-efficiently adapted state $P_m$, and local-evidence
augmentation $G_m+L$. The primary diagnostic is the within-backbone change;
cross-backbone absolute scores are secondary because native tokenization,
pretraining, streaming interfaces, and hidden-state semantics differ. Common
local encoders test whether adaptation transfers across global backbones,
whereas native local encoders are a separate, model-specific protocol. All
rows share physical input cutoffs, targets, data splits, and evaluation.
Unsupported causal readouts are disclosed rather than replaced by offline or
future-conditioned features.
\textbf{Event protocol.}
Dense ten-horizon AP/mAP is retained for EgoSpeak comparability. The implemented
competing-risk evaluator counts each eligible aggregate-other boundary once,
ends a response window before the next other onset, and ends a yield window
before the next other offset. Pending the remaining P0 real-data audit and
scenario freeze, the intended primary behavior metric is false wearer entries per
minute of multi-other opportunity at a training-side selected wearer-onset
recall, with $R=.8$ as the headline operating point. The intended analysis also
reports onset latency and wearer-yield recall/latency, following the
event-oriented emphasis on recall, false alarms, and latency
\cite{jiang2026turnbench}. The protocol will censor a response window after an
other offset at the next other onset, and a yield window after an other onset
at the next other offset.
\todo{Implement the three backbone adapters and validate state timestamps with
an impulse/cutoff test. Pre-register one matched PEFT budget and one common
local branch. Report native-vs-common local encoders in separate table panels.
Keep future model additions outside the confirmatory comparison once this
matrix is locked.}
\todo{P0 blocker: run the tested competing-risk evaluator on real predictions,
manually audit sampled timelines, and freeze pairing before treating any event
score as a paper result. Specify overlap,
minimum duration, timeouts, duplicate detections, source boundaries, offscreen
cases, and event counts. Until this is complete, call existing values
timing-derived proxies rather than final event results.}
\textbf{Event construction.}
We derive \emph{wearer onset} events directly from the wearer activity target.
A \emph{multi-other exchange} is a wearer-silent opportunity interval in which
at least two distinct non-wearer participant IDs contribute temporally ordered
speech activity. It is an observable interaction proxy, not a claim about the
speakers' mental intent or addressee. Threshold $\tau_r$ is selected on the
development split to obtain wearer-onset recall
$r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary metric is false wearer
entries per minute of multi-other opportunity at matched onset recall; the
headline operating point is $r=0.8$. We also report signed onset latency,
wearer-yield recall/latency, and the original ten-horizon per-frame mAP.
\todo{Freeze the event pairing and competing-risk rules, including the
multi-other alternation window, minimum speech/silence, overlap, maximum onset
delay, duplicate detections, one detection per event, track visibility, source
boundaries, and group/offscreen cases. Publish train/test onset counts,
multi-other events/minutes, number of distinct visible speakers, and exclusion
counts. Manually audit at least 100 events. Treat all previously viewed
Sessions 1--3 scores as exploratory; lock the protocol and model before the
confirmatory three-seed run.}
\textbf{Baselines and controls.}
For each backbone we compare: frozen global state $G_m$; PEFT $P_m$; $G_m$
plus recent local audio; and $G_m$ plus pooled local audio--visual evidence.
This compact matrix tests the representation hypothesis before investing in a
complex fusion module. The first learned target after the oracle audit is a
causal estimator of timely aggregate visible-person floor evidence. Only if
person-organized features survive three seeds, visual time/person corruptions,
and the event gate do we compare a parameter-matched temporal branch without
person continuity, a separately trained per-frame/no-ID set branch, and a
track-consistent masked cross-attention branch. Provided-track and online-track
inputs are reported separately. On a frozen track-aware checkpoint, a clip-consistent
global renaming must leave outputs unchanged, whereas \emph{broken
continuity} periodically remaps IDs while preserving every crop, timestamp,
audio sample, and target. Time-shifted and removed vision provide additional
alignment controls.
\todo{Define the broken-continuity corruption before evaluation: remap only at
pre-registered 1-s boundaries or report a fixed ID-switch-rate curve. Do not
call a clip-consistent slot permutation ``broken ID.'' Verify unchanged crops,
frame counts, audio, labels, and visibility masks. Train the no-ID/reset
baseline from scratch with matched parameters; evaluate all test-time
counterfactuals using one frozen checkpoint and one frozen threshold.}
\textbf{Implementation and statistics.}
All prediction heads run at 5~Hz and use an 8-s causal observation window;
model-specific deviations required by a native streaming interface are stated
explicitly. In the frozen protocol the omni backbone does not update. In PEFT
and local protocols, trainable parameters and optimizer exposure are reported
separately. Every temporal state resets at non-contiguous source boundaries.
We report three seeds and mean$\pm$standard deviation. Confidence intervals for
event metrics use paired cluster bootstrap over sessions/conversation blocks
rather than treating adjacent frames as independent. Runtime includes P50/P95
decision latency, throughput, peak memory, and trainable parameter count.
\todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions,
$L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter
count, tracker timeout/ranking, hardware, and end-to-end real-time latency
after the architecture is frozen.}
\textbf{Controls and statistics.}
The main intervention compares correctly aligned local evidence with
time-shifted evidence, removed vision, global-frame visual pooling, and the
capacity-matched temporal branch. A same-rate control is required for a
dual-rate claim. Track/no-ID/broken-continuity controls are added only if a
track-aware variant first wins on training-side data. We report three seeds,
mean$\pm$sample standard deviation, and paired cluster-bootstrap intervals over
sessions or conversation blocks. \todo{Freeze seeds, bootstrap unit/count,
optimizer, epochs, hardware, throughput, P50/P95 latency, and trainable
parameters.}
\section{Results and Analysis}
\label{sec:results}
\begin{table}[t]
\caption{Causal oracle-information screen on EasyCom development Session~12
(seed 0, four epochs). Oracle speech activity is analysis-only. Values are
percentages under the causal-valid protocol.}
\label{tab:oracle}
\centering
\small
\begin{tabular}{lcc}
\hline
Input condition & 0.2-s mAP & Mean mAP\\
\hline
Global + local audio & 72.58 & 58.77\\
Visibility only & 71.95 & 58.86\\
Activity delayed by 1 s & 72.02 & 58.90\\
Aggregate visible activity & \textbf{79.55} & \textbf{61.68}\\
Per-person activity histories & 78.80 & 61.11\\
Identity-shuffled histories & 78.86 & 61.19\\
\hline
\end{tabular}
\end{table}
Table~\ref{tab:oracle} establishes an information opportunity, not a deployable
result. Timely aggregate activity among visible non-wearers provides a large
gain, while visibility alone and stale activity do not. However, preserving
person-resolved history gives no advantage over destroying its cross-time
identity assignment, and both trail the aggregate representation. The learned
method must therefore first recover useful local floor evidence; person-track
continuity is retained only if subsequent no-ID and broken-track controls
separate.
\begin{table}[t]
\caption{Exploratory Qwen2.5-Omni-3B local-branch evidence on EasyCom test
Sessions 1--3. Values are percentages from one seed, not a cross-backbone or
confirmatory result.}
\label{tab:preliminary}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccc}
\hline
Model & 0.2-s mAP & 0.2-s wearer AP & Mean mAP\\
\hline
Qwen-3B $G$+local A & 81.72 & 85.07 & 65.11\\
Qwen-3B $G$+local AV-pool & 82.03 & 85.09 & 65.40\\
\hline
$\Delta$ visual & +0.30 & +0.02 & +0.29\\
\hline
\end{tabular}}
\end{table}
Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
Pooled vision gives a small gain, while immediate wearer AP is nearly
unchanged. This is compatible with either useful generic local context or
regularization; it is not evidence of person grounding. Only matched-recall
event evaluation and visual-alignment controls can distinguish these cases.
A separate Qwen-3B dense-cache diagnostic gave 58.16 mean mAP for its frozen
readout and 56.20 for one LoRA recipe. Because that diagnostic used a different
feature protocol, it is not combined numerically with
Table~\ref{tab:preliminary}; it only shows why a fair, protocol-matched PEFT
study remains necessary.
\begin{table}[t]
\caption{Planned cross-backbone EasyCom event evaluation. Lower
$\mathrm{FP}_{\mathrm{mo}}$ is better; all rows use the same
development-selected operating protocol. ``Local'' denotes the train-side
selected common local-evidence candidate.}
\caption{Confirmatory EasyCom results. Lower multi-other false entries are
better; all event rows use a training-side selected operating point.}
\label{tab:main}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccccc}
\hline
Backbone/protocol & $\mathrm{FP}_{\mathrm{mo}}$/min@$R=.8$ & Onset R &
Onset latency & Yield R & Mean mAP\\
Method & 0.2-s mAP & Mean mAP & FP/min@$R=.8$ & Onset lat. & Yield R\\
\hline
Qwen-3B frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-3B PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-3B + local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-7B frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-7B PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
MiniCPM-o frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
MiniCPM-o PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
EgoSpeak reproduction & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
$G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
$G+$matched temporal local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
$G+$causal local AV (ours) & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}}
\end{table}
\todo{Report Table~\ref{tab:main} as mean$\pm$standard deviation over three
seeds, plus 95\% paired cluster-bootstrap confidence intervals for frozen vs
PEFT and frozen vs local within each backbone. A general omni-representation
claim requires a consistent pattern across at least two backbone families; an
effect confined to Qwen must be stated as Qwen-specific. The behavior claim is
supported only if multi-other FP/min decreases at matched wearer-onset recall
without worse onset latency or wearer-yield recall. If the selected local
candidate uses tracks, also report its paired intervals against no-ID and
temporal-capacity controls.}
\textbf{Information opportunity.}
On development Session~12, an analysis-only seed-0 oracle supplied perfectly
timed visible-other speech activity through an explicit non-deployable path.
Aggregate visible activity improved 0.2-s mAP by 6.97 points and ten-horizon
mean mAP by 2.91 points over matched global-plus-local-audio. Visibility-only
and one-second-delayed activity remained near audio, while per-person histories
did not outperform identity-shuffled histories. This motivates timely local
floor evidence, not participant identity. \todo{After seeds 1--2, report the
interval or remove the exact oracle numbers from the four-page paper.}
\textbf{Main result.}
Table~\ref{tab:main} should establish one result: \todo{the selected local
adapter reduces multi-other false entry at matched valid-onset recall, retains
onset latency/yield behavior, improves dense forecasting, and recovers a
meaningful fraction of the oracle opportunity}. The 7B row should determine
whether \todo{capacity alone closes the gap or the adaptation structure remains
useful at larger scale}. No conclusion is drawn across all Omni families from
Qwen-only evidence.
\begin{table}[t]
\caption{Conditional track-continuity counterfactual on the best track-aware
model. This table is retained only if train-side selection chooses such a
candidate; all rows reuse one frozen checkpoint.}
\label{tab:counterfactual}
\caption{Mechanism controls for the selected Omni-3B local adapter.}
\label{tab:ablation}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccc}
\hline
Track/visual evidence & $\mathrm{FP}_{\mathrm{mo}}$/min & Onset recall & 0.2-s mAP\\
Local condition & Mean mAP & FP/min@$R=.8$ & Yield R\\
\hline
Correct tracks & \todo{--} & \todo{--} & \todo{--}\\
Global ID rename & \todo{--} & \todo{--} & \todo{--}\\
Broken continuity & \todo{--} & \todo{--} & \todo{--}\\
Time shifted & \todo{--} & \todo{--} & \todo{--}\\
Removed & \todo{--} & \todo{--} & \todo{--}\\
Correct causal local AV & \todo{--} & \todo{--} & \todo{--}\\
Remove local vision & \todo{--} & \todo{--} & \todo{--}\\
Time-shift local vision & \todo{--} & \todo{--} & \todo{--}\\
Global/full-frame visual pool & \todo{--} & \todo{--} & \todo{--}\\
Same-rate/capacity control & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}}
\end{tabular}
\end{table}
If a track-aware candidate is selected, Table~\ref{tab:counterfactual} is the
continuity test. Global renaming must be
numerically invariant because IDs carry no semantics. If the model exploits
person history, correct tracks should outperform broken continuity even though
both contain exactly the same per-frame people. If broken, time-shifted, and
removed variants perform similarly to correct tracks, the defensible
conclusion is additional capacity or generic visual context, not useful
person-continuous reasoning.
\textbf{Mechanism and limitations.}
Table~\ref{tab:ablation} should test whether gains require correctly timed local
visual evidence rather than extra parameters, regularization, or generic scene
context. \todo{Report paired intervals and the supported interpretation. If
correct and shifted/removed vision do not separate, drop the audiovisual
evidence-routing claim. If dense mAP improves but event behavior does not, do
not claim improved floor control. If legitimate onset/yield degrades, stop the
adapter rather than describing lower false entry as success.}
\todo{Add one compact error analysis split by model family, track visibility,
number of visible people, overlap, track age/fragmentation, and
backchannel/interruption proxy. Include event counts with every rate. Move
rejected PEFT recipes and fusion candidates to supplementary material unless
they directly explain the selected method.}
\textbf{Qualitative evidence.}
For a multi-other exchange, visualize annotation-derived per-person activity
as an analysis reference, provided track IDs when applicable, wearer ground
truth, the frozen-global and selected-local wearer-entry scores, and the single
frozen threshold on one physical time axis. If the selected candidate is
track-aware, also show its broken-continuity score. A useful
example shows the frozen baseline rising at an other-speaker boundary while
the selected adapter remains below threshold; a companion positive example
must show that a real wearer onset is not suppressed.
\todo{Do not hand-pick only successful clips. Before viewing the final test
plots, freeze a deterministic selection rule and show: largest paired
improvement, near-median improvement, one failure, and one correctly retained
wearer onset. Include full local timelines and source frame times, not cropped
screenshots without context. State that examples illustrate the aggregate
event result and cannot establish it alone.}
\todo{Add one compact, pre-selected qualitative timeline only if space permits:
one multi-other false-entry improvement paired with one preserved real wearer
onset or one failure. Examples illustrate aggregate rates and never establish
the claim by themselves.}
\section{Conclusion}
\label{sec:conclusion}
We study when an egocentric agent should enter or yield a multiparty floor, not
merely whether someone will speak. The working decomposition is global Omni
context plus causal local audiovisual floor evidence. The current
Qwen2.5-Omni-3B oracle audit supports the second term: timely aggregate
visible-other activity is informative beyond the global state and recent
mixed audio. It does not support the stronger claim that participant identity
or track continuity adds value, because per-person histories do not beat an
identity-shuffled control. Dense activity forecasting supplies a comparable
target, while matched-recall event evaluation must determine whether a learned
local estimator actually prevents intrusions during multi-other exchanges
without suppressing valid wearer entry or yield. \todo{After the controlled
study, state which learned local variant was selected, how much of the oracle
gap it recovered, whether the pattern generalized beyond Qwen, the
quantitative false-entry and onset/yield results, and the main deployment
limitation. If no learned variant passes these tests, conclude with the
information audit rather than claiming a track-based solution.}
This study asks whether a general-purpose Omni representation can be adapted
for causal floor forecasting of an invisible egocentric wearer. The candidate
method separates long-context Omni understanding from short-window local floor
evidence without assuming persistent speaker identity. \todo{State the final
learned-method improvement, matched-recall false-entry result, onset/yield
non-degradation, causal-control evidence, and main limitation. If no learned
variant passes these tests, reframe the submission as an information audit or
do not make the method claim.}
\bibliographystyle{IEEEbib}
\bibliography{references}