diff --git a/manuscript/main/main.tex b/manuscript/main/main.tex index 3f6a550..c793d9c 100644 --- a/manuscript/main/main.tex +++ b/manuscript/main/main.tex @@ -10,8 +10,8 @@ \newcommand{\vect}[1]{\boldsymbol{#1}} \setlength{\emergencystretch}{2em} -\title{WHEN SHOULD AN EGOCENTRIC AGENT SPEAK? DIAGNOSING AND ADAPTING -OMNI REPRESENTATIONS FOR MULTIPARTY FLOOR ACTION} +\title{WHEN SHOULD AN EGOCENTRIC CONVERSATIONAL AGENT SPEAK? PROBING AND +ADAPTING OMNI MODELS FOR MULTIPARTY FLOOR ACTION} % TODO: Insert the final author list and affiliations after the ICASSP 2027 % review policy and author order are finalized. @@ -23,59 +23,87 @@ OMNI REPRESENTATIONS FOR MULTIPARTY FLOOR ACTION} \maketitle \begin{abstract} -Before generating a response, an egocentric conversational agent must decide -whether to speak or keep listening in a scene that may contain several other -people. General-purpose omni models offer broad streaming audio--visual -context, but it is unclear whether their global representations expose the -fine-grained local cues required for a decision every 200~ms. We study this -question across representative omni backbones and adaptation regimes: frozen -global readout, parameter-efficient tuning, and causal augmentation with -recent audio and visible-person histories. The human camera wearer serves as a -behavioral proxy for the agent, while the dense EgoSpeak target remains the -common output. An EasyCom event protocol reports false wearer entries during -multi-other exchanges at matched wearer-onset recall, and track/alignment -controls test whether visual gains follow person continuity rather than added -capacity. Exploratory Qwen2.5-Omni-3B results motivate the study but do not yet -establish a limitation shared by omni models or determine the final adapter. +An egocentric conversational agent operating among several people must decide +whether to speak or keep listening before it generates a response. EgoSpeak +operationalizes this problem using recordings from human camera wearers: +causal first-person audio--video forms the observation, and the wearer's future +speaking behavior forms the action target. The wearer is thus a data-collection +surrogate rather than the deployed system. EgoSpeak evaluates +separately extracted audio--visual features with Transformer, GRU, and Mamba +temporal models; whether end-to-end pretrained omni models that jointly process +raw audio and video expose the +fine-grained cues required for a decision every 200~ms remains unclear. We +probe representative omni backbones using frozen readout, +parameter-efficient tuning, and causal side branches that preserve recent +audio and visible-person evidence. The study tests whether a global omni state +underrepresents other-person activity and whether local evidence compensates +without making the agent passive. An EasyCom event protocol measures false +agent entries during multi-other exchanges at matched recall of wearer-derived +onset targets. Exploratory Qwen2.5-Omni-3B results motivate this hypothesis but +do not yet establish a cross-model limitation or determine the final adapter. \todo{After the train-side study is locked, name the selected backbones and adapter and insert: frozen/PEFT/local deltas, multi-other false-entry reduction at 80\% onset recall, onset/yield non-degradation, and 95\% intervals.} \end{abstract} \begin{keywords} -turn-taking, speech initiation, egocentric video, omni models, -audio-visual adaptation +conversational agents, turn-taking, speech initiation, egocentric video, +omni models \end{keywords} \section{Introduction} \label{sec:intro} -Consider an embodied conversational agent that observes the scene through an -egocentric camera and hears mixed environmental audio. Before response -generation and speech synthesis, a causal floor controller must decide every -200~ms whether the agent should continue listening, begin speaking, or yield. -In a dyadic exchange, a pause or an apparent end of turn may be an invitation -to respond. This assumption fails in multiparty interaction: one participant -may hand the floor to another visible participant, and an agent that reacts -only to acoustic availability may interrupt an exchange that does not involve -it. We study this floor-action controller; response content generation and -closed-loop conversational effects are outside our scope. The observed human -camera wearer is a behavioral proxy rather than an equivalent autonomous -agent: its activity supplies human demonstrations of floor timing under the -same first-person observation geometry. +We consider a conversational agent equipped with an egocentric camera and +microphone and deployed in a scene where several people may speak. Before +response generation and speech synthesis, its causal floor controller must +decide every 200~ms whether to continue listening, begin speaking, or yield. A +pause in dyadic dialogue may invite a response; in multiparty interaction it +may instead mark a handover between two other people. An agent that reacts only +to acoustic availability can therefore interrupt a conversation that it +should continue observing. -Voice Activity Projection (VAP) models future joint speech activity and -derives turn-taking events without manually labeling each event -\cite{ekstedt2022vap}. EgoSpeak extends future activity prediction to causal, -untrimmed egocentric video, forecasting wearer and aggregate non-wearer -activity over the next two seconds \cite{kim2025egospeak}. Its role-relative -output is attractive for a wearable agent, but an ``others'' channel discards -whether the same or different visible people produced the recent activity and -how their behavior evolved across time. Conventional per-frame average -precision can consequently be dominated by ordinary activity while hiding -precisely the false-entry behavior that person-consistent visual cues may -prevent. Event-oriented evaluation likewise emphasizes recall, false alarms, -and latency rather than a single frame-level score \cite{jiang2026turnbench}. +Existing datasets do not contain autonomous agents making these decisions at +scale. Following EgoSpeak \cite{kim2025egospeak}, we use human egocentric +recordings to construct supervision: the camera wearer defines the first-person +sensor origin, and the wearer's observed speaking behavior defines the desired +floor-action target. This is an imitation-learning proxy for a future agent +under comparable observations. Throughout the paper, \emph{wearer} names this +dataset role and target channel; it is neither the deployed agent nor a +semantic model input. Response content and closed-loop effects of deploying an +agent remain outside our scope. + +EgoSpeak already establishes real-time speech-initiation prediction in causal, +untrimmed egocentric video. It combines separately pretrained ResNet-50 and +wav2vec2 feature extractors with Transformer/LSTR, single-layer GRU, and Mamba +temporal models, forecasting the camera wearer and aggregate other-person +activity over the next two seconds \cite{kim2025egospeak}. Thus our novelty is +not the task definition or the use of first-person data. The unresolved model +question is whether an end-to-end pretrained audio--visual omni backbone, +operating on causal raw media, exposes an adequate state for this dense +floor-action controller. To our knowledge, such omni representations have not been +systematically evaluated or adapted for the EgoSpeak task. + +Our working hypothesis is specific. An omni model may retain broad scene and +speech semantics while its global causal state makes recent prosody, activity +boundaries, and individual visible-speaker motion difficult for a shallow +5-Hz head to recover. This would appear as inaccurate prediction of +other-person activity and poorly timed agent entries, especially in +multiparty exchanges. A causal local side branch may compensate by preserving +recent audio and anonymous person-level evidence before fusion with the global +state. This is a hypothesis to test, not an assumed omni-model deficiency: +frozen, parameter-efficiently tuned, and locally augmented variants must be +compared within each backbone. + +Voice Activity Projection (VAP) provides the broader future-activity view of +turn taking \cite{ekstedt2022vap}. EgoSpeak's role-relative output preserves +compatibility with that view, but its aggregate ``others'' channel discards +whether the same or different visible people produced recent activity. +Moreover, conventional per-frame average precision can hide the behavioral +failure of entering during an exchange between others. We therefore retain the +dense EgoSpeak target while adding event evaluation of agent entry and yield; +event-oriented work likewise emphasizes recall, false alarms, and latency +rather than a single frame-level score \cite{jiang2026turnbench}. Related work supplies complementary parts of the problem. Egocentric talking-to-me detection predicts whether a completed utterance addresses the @@ -83,29 +111,32 @@ camera wearer and benefits from long-term social context \cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}. The former is generally utterance-level rather than continuous anticipation; -the latter predicts visible participants rather than the floor action of an -invisible camera wearer. Our gap is therefore narrower than ``audio--visual -multiparty turn taking'': we study whether short-term anonymous participant -continuity helps direct, continuous wearer-action projection when the candidate -responder is behind the camera and the visible participant set changes. +the latter predicts visible participants rather than the action of an +egocentric conversational agent represented by a wearer-derived target. Our +gap is therefore narrower than ``audio--visual multiparty turn taking'': we +study whether omni representations support direct, continuous agent +floor-action prediction and whether causal local evidence helps when the +visible participant set changes. We call this task \emph{egocentric floor-action projection}. The study asks -four questions: (RQ1) how much wearer activity is linearly or shallowly -readable from a frozen global omni state; (RQ2) whether standard +four questions: (RQ1) how accurately a frozen global omni state predicts the +wearer-derived agent target and other-person activity; (RQ2) whether standard parameter-efficient tuning recovers missing performance; (RQ3) whether causal -local audio and person-level visual evidence provide complementary information; -and (RQ4) whether any aggregate gain reduces false agent entries without -making the controller globally passive. EasyCom is suitable because it -provides synchronized egocentric audio--video, per-person tracks, and voice +local audio and person-level visual evidence provide complementary +information; and (RQ4) whether any aggregate gain reduces false agent entries +without making the controller globally passive. EasyCom is suitable because +it provides synchronized egocentric audio--video, per-person tracks, and voice activity \cite{donley2021easycom}. Target-of-speech annotations, when used, are analysis strata only and are never model input or required output. -The intended contributions are therefore a controlled cross-backbone -representation diagnosis, a lightweight local-evidence adaptation selected -from pre-registered candidates, and a behavior-aligned event evaluation. The -current Qwen2.5-Omni observations are motivation, not a claim that all omni -models intrinsically fail. Likewise, track-consistent cross-attention becomes -the final method only if it beats pooled, no-ID, and capacity-matched controls. +The intended contributions are therefore: a controlled cross-backbone study +of pretrained omni representations for EgoSpeak-style floor action; a +lightweight local-evidence adaptation selected from +pre-registered candidates; and a behavior-aligned event evaluation that maps +wearer demonstrations to agent entry and yield decisions. Current +Qwen2.5-Omni observations are motivation, not a claim that all omni models +intrinsically fail. Likewise, track-consistent cross-attention becomes the +final method only if it beats pooled, no-ID, and capacity-matched controls. \begin{figure}[t] \centering @@ -115,8 +146,9 @@ the shared experiment interface: raw causal AV, backbone-specific global readout, frozen/PEFT/local-adapter branches, optional recent audio and anonymous person histories, and the common $10\times3$ head. Mark causal cutoffs, trainable modules, parameter budgets, and source-boundary resets.}}} -\caption{Planned model-agnostic floor-action study. The local adapter remains a -candidate family until the controlled train-side comparison is complete.} +\caption{Planned omni-model study for egocentric conversational floor action. +The local adapter remains a candidate family until the controlled train-side +comparison is complete.} \label{fig:method} \end{figure}