diff --git a/manuscript/targeted/README.md b/manuscript/targeted/README.md index 6544312..0554c58 100644 --- a/manuscript/targeted/README.md +++ b/manuscript/targeted/README.md @@ -5,8 +5,18 @@ modify or replace `../main/`. ## Positioning -The paper is deliberately framed as a causal global--local **audit**, because -that is the strongest complete story supported by archived evidence: +The paper now starts from the egocentric conversational agent's when-to-speak +problem, positions it against 27 cited works in turn-taking, first-person +audio-visual conversation, active-speaker modeling, and streaming Omni +interaction, then studies two cutoff-aligned interfaces: + +- **short-window encoder evidence (SWE)**: direct recent audio and optional + person-crop visual encoder features alongside the frozen Omni readout; +- **partial-tail refresh (PTR)**: query only the observed prefix of the current + two-second block at each 200-ms cutoff, reusing completed payloads. + +The evidence remains a causal global--local **audit**, not a claim that all +Omni models lack local perception or that PTR's independent gain is proven: - global Omni context and recent audio are complementary across all horizons; - most measured benefit is dense forecasting and wearer-offset anticipation; @@ -24,6 +34,10 @@ independently audited here. No semantic ENTRY/YIELD, addressee, online-detector, or real-time-throughput claim is made. The deployable matrix is explicitly single-seed and exploratory. +EasyCom has archived screening scores. The Ego4D manifest and extraction / +training workflow exist, but this checkout has no archived Ego4D test metrics; +the manuscript specifies its transfer protocol and explicitly leaves any +cross-dataset performance claim pending. The six-group L/T/C/B/H-Q/S table remains in `main.tex` as an **illustrative, unmeasured planning aid**, excluded from the page-limited PDF. Its directional statements are hypotheses and falsifiers, not EasyCom @@ -46,7 +60,8 @@ the task diagram (Fig. 1) and the system diagram (Fig. 2) occupy one column. The current PDF reserves five numbered result tables: measured deployable and oracle screens share Table 1; L controls extend its first panel. The unmeasured T/C matrix and a separate measured chunk screen share Table 2; -B, H, and S occupy Tables 3--5. Pending cells are explicitly unmeasured. +B (including an Ego4D transfer slot), H, and S occupy Tables 3--5. Pending +cells are explicitly unmeasured. Figure 3 is an H/Q case slot, not a selected or measured example. Its cutoff-marked timelines must be inserted only after the slice protocol and case-selection rule are frozen. The single-column system figure is generated diff --git a/manuscript/targeted/main.tex b/manuscript/targeted/main.tex index 374c993..840412b 100644 --- a/manuscript/targeted/main.tex +++ b/manuscript/targeted/main.tex @@ -17,10 +17,10 @@ \renewcommand{\textfraction}{0.05} \renewcommand{\floatpagefraction}{0.75} \renewcommand{\dblfloatpagefraction}{0.75} -\title{Does an Omni Model Know When the Wearer Will Speak?\\ -A Causal Global--Local Audit of Egocentric Speech Anticipation} -\name{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu, -Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen} +\title{Egocentric Speech Anticipation with Omni Models:\\ +Short-Window Evidence and Partial-Tail Refresh} +\name{\shortstack{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu,\\ +Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}} \address{Shanghai Ideal Information Industry (Group) Co., Ltd., China Telecom} \begin{document} @@ -28,19 +28,21 @@ Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen} \maketitle \begin{abstract} -We test whether a frozen Omni readout exposes sub-second evidence for -forecasting an invisible camera wearer's speech from causal multiparty -audio--video. Our 5-Hz model combines 8-s Qwen2.5-Omni context and recent -1-s audio to predict three activity channels at ten 0.2--2.0-s horizons. -On EasyCom, a seed-0 screen reaches 81.72\% mAP at 0.2~s and 65.11\% -mean mAP, versus 74.20/61.19\% for released EgoSpeak. Wearer offset AP -improves by 7.23 points, onset AP by 1.47. Pooled vision adds only 0.29 -mean-mAP points. A three-seed causal information audit finds that privileged -past visible-person activity adds 4.83 points at 0.2~s and 1.77 mean-mAP -points; visibility and delayed activity add little. A separate no-box -full-frame screen reaches 82.38/65.54\%, but unmatched training and head -capacity prevent visual attribution. Timely evidence access, not semantic -floor understanding, remains the supported conclusion. +An egocentric conversational agent must anticipate when its invisible wearer +will speak while observing a causal multiparty audio--video stream. Although +turn-taking prediction and omni-modal assistants have advanced, the temporal +accessibility of local evidence in an integrated Omni state remains unclear. +We study two interfaces: a short-window encoder-evidence path that exposes +recent audio and optional person-crop vision to a frozen-Omni forecasting +head, and partial-tail refresh that queries the observed prefix of an +incomplete audiovisual block at each 200-ms cutoff. On EasyCom, the +exploratory global-plus-audio screen obtains 81.72\% 0.2-s mAP and 65.11\% +ten-horizon mean mAP, versus 74.20/61.19\% for released EgoSpeak. Pooled +vision adds only 0.29 mean-mAP points, whereas an analysis-only causal +visible-activity oracle reveals additional information. A separate chunk +screen suggests sensitivity to state freshness, but does not isolate the +partial-tail effect. These results motivate finer local evidence access; +they do not establish semantic floor control or cross-dataset transfer. \end{abstract} \begin{keywords} @@ -51,12 +53,16 @@ streaming, voice activity projection, conversational agents \section{Introduction} \label{sec:intro} -Conversational agents must decide not only \emph{what} to say, but also -\emph{when} to participate. This decision is particularly asymmetric in an -egocentric stream: the camera wearer is normally invisible, yet the model must -forecast that wearer's behavior from mixed audio and observations of other -people. As Fig.~\ref{fig:task} illustrates, prediction precedes any known -turn boundary and excludes media after the decision cutoff. +An egocentric conversational agent must decide \emph{when} to speak while its +camera wearer is usually invisible. It sees other participants and hears a +mixed conversation, but must predict the wearer's own future speech before a +turn boundary is known (Fig.~\ref{fig:task}). Continuous turn-taking models +and voice-activity projection (VAP) show the value of anticipatory rather than +silence-triggered decisions \cite{ekstedt2022vap,russell2025mmvap, +elmers2025triadic,lin2025backchannel,lee2024eot,qi2026muvap, +jiang2026turnbench}. Yet these tasks differ in sensor placement, visible +speakers, dialogue roles, and whether the target is a future activity state +or a semantic intervention. \begin{figure}[H] \centering @@ -68,47 +74,56 @@ not semantic floor ownership.} \label{fig:task} \end{figure} -Voice Activity Projection (VAP) casts turn taking as continuous prediction of -future activity \cite{ekstedt2022vap}. EgoSpeak brings this objective to -untrimmed first-person video and defines the EasyCom protocol used here -\cite{kim2025egospeak,donley2021easycom}. Other work predicts future speakers -from synchronized egocentric sensors \cite{serj2024threem}, reconstructs -egocentric conversational graphs \cite{jia2024avconv}, or grounds causal -multiparty projections in visible face tracks \cite{qi2026muvap}. Those -systems establish that neither multiparty audiovisual modeling nor -first-person speech initiation is itself a novelty. Our question is narrower: -does a general-purpose Omni readout make timely evidence accessible for -forecasting the unseen wearer? +\textbf{Egocentric audiovisual conversation.} +EgoSpeak forecasts wearer speech from first-person RGB and audio using +specialized encoders and a temporal fusion model \cite{kim2025egospeak}; +EasyCom and Ego4D supply complementary egocentric conversation settings +\cite{donley2021easycom,grauman2022ego4d}. Multi-view next-speaker +anticipation, egocentric speaking/listening graphs, and addressee detection +study related but different targets \cite{serj2024threem,jia2024avconv, +kong2024sicnet}. Egocentric active-speaker localization and detection +\cite{jiang2022asl,huh2025egoasd,clarke2025asd} and long--short person +modeling \cite{wang2024loconet} estimate who is speaking now; they do not by +themselves forecast an unseen wearer's future activity. Visual cues can help +turn prediction in noise \cite{oconnorrussell2025noise}, but whether pooled +vision conveys the relevant local floor evidence here is empirical. -Qwen2.5-Omni integrates long-form audio and video in a Thinker representation -\cite{xu2025qwenomni}. Such integration is attractive for conversational -context, but a blockwise state can blur or delay the short acoustic and visual -cues needed at 200-ms intervals. We therefore separate an integrated global -state $\G$ from explicit recent evidence $\A$ and optional $\V$, and treat -their value as an empirical question rather than assuming that a larger -backbone solves temporal access. +\textbf{Omni and local evidence.} +Omni systems integrate modalities and streaming interaction +\cite{xu2025qwenomni,cui2026minicpmo45,tian2026roma,yan2026duplug, +yu2025salmonnomni,chen2024videollmonline,wang2025omnimmi}. Encoder-side +audio--visual adaptation offers another route to accessible modality evidence +\cite{lin2023lavish,rouditchenko2024whisperflamingo}. However, neither +large-context integration nor a nominally streaming interface guarantees a +fresh representation at every 200-ms prediction cutoff. In our Qwen2.5-Omni +extraction, completed 2-s blocks and an incomplete current block have +different update rules. We therefore test two distinct interfaces: +\emph{short-window encoder evidence} (SWE) exposes recent modality features +outside the integrated Thinker state, and \emph{partial-tail refresh} (PTR) +queries only the observed current-block prefix. Neither term implies that +all Omni models use the same chunk duration or lack local perception. -We contribute a cutoff-aligned global--local audit, a bounded Omni stream -with observed-tail refresh and recent audio, and a separation of deployable -features from an analysis-only visible-activity oracle. Generic pooled or -current-frame person features recover little of the available activity signal; -timely activity estimation is the next modeling target. +We contribute a causal implementation of SWE and PTR for wearer activity +anticipation; an EasyCom audit separating global, local-audio, pooled-visual, +and privileged information conditions; and a protocol for EasyCom and Ego4D +transfer with endpoint and hard-slice diagnostics. Matched tail and Ego4D +scores are still pending, so our measured claims remain confined to the +available EasyCom screens. \begin{figure}[t] \centering \includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf} -\caption{Deployable global--local path and bounded stream. At each 200-ms -cutoff, the head fuses the frozen integrated Omni state with explicit recent -audio and optional pooled person-crop vision. Completed 2-s payloads are -reused; only the observed partial tail is temporarily recomputed. Privileged -past-activity audit labels are kept outside this graph and are not inputs.} +\caption{Two cutoff-aligned interfaces. SWE supplies recent encoder-side +audio and optional pooled person-crop vision to a frozen-Omni head. PTR +temporarily queries the observed current-block prefix while reusing completed +2-s payloads. Privileged past-activity labels are not model inputs.} \label{fig:overview} \end{figure} \section{Method} \label{sec:method} -\subsection{Task and Causal Inputs} +\subsection{Task and Short-Window Encoder Evidence} At cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s. Each @@ -116,17 +131,16 @@ horizon has three channels in fixed order: background, camera wearer, and aggregate others. Wearer and others may overlap; no output identifies an individual speaker, addressee, or semantic end of turn. -The global feature $\vect g_n$ is extracted from frozen Qwen2.5-Omni-3B. We -concatenate the final four Thinker readout states and project them to 2048 -dimensions. The readout contains both audio and video context; $\G$ is not an -audio-free condition. The local audio feature $\vect a_n$ is obtained by -cropping the most recent one-second mixed waveform \emph{before} -preprocessing, encoding it with the frozen Qwen audio encoder, and averaging -valid output tokens. An optional visual feature $\vect v_n$ averages spatial -tokens inside provided person boxes, people within each causal frame, and -valid frames from the most recent second. It preserves neither identity nor -mouth-motion trajectories, and provided boxes do not constitute an online -detector. +The global feature $\vect g_n$ concatenates the final four frozen +Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It +already contains audio and video context; $\G$ is not audio-free. SWE +adds encoder-side features without requiring the Thinker state to preserve +their fine timing. Local audio $\vect a_n$ crops the latest one-second mixed +waveform \emph{before} preprocessing, runs the frozen Qwen audio encoder, and +averages valid tokens. Optional $\vect v_n$ averages frozen vision tokens +inside provided person boxes, then across people and valid causal frames in +the latest second. This pooling has no identity or mouth-motion trajectory; +provided boxes do not imply an online detector. Each enabled local feature is layer-normalized, projected to 256 dimensions, and paired with a validity bit. The trainable head is @@ -143,15 +157,17 @@ head with the EgoSpeak multi-positive cross-entropy over causal-valid source--horizon pairs. The Omni and modality encoders remain frozen, so this is frozen-backbone adaptation rather than end-to-end training. -\subsection{Cutoff-Aligned Streaming} +\subsection{Causal Partial-Tail Refresh} -Figure~\ref{fig:overview} uses source-aligned 2-s blocks under an 8-s -presentation cap. After warm-up, three completed blocks and the observed -current tail are available; span varies with block phase. Completed -modality payloads are reused. At each cutoff, the tail is cropped at $t_n$ -and queried in a temporary cache, without committing it. Only completed -blocks are committed; session or source discontinuities reset caches. This -refreshes the logical state at 5~Hz without backfilling earlier rows. +PTR uses source-aligned 2-s blocks under an 8-s maximum presentation cap +(Fig.~\ref{fig:overview}). After warm-up, three completed blocks and the +observed current tail are available; span varies with block phase. Reusing +completed modality payloads, we crop the tail at $t_n$ and query it in a +temporary cache. The tail and query are not committed. Only completed +blocks enter persistent history; session and source discontinuities reset +caches. Thus each 5-Hz readout sees the observed prefix without future +backfill. PTR is a causal access rule, not by itself a proven accuracy or +real-time latency improvement. \subsection{Visible-Activity Information Audit} @@ -255,8 +271,8 @@ Completed blocks + observed tail & 70.85 & 59.17 & -- \\ \end{table*} \begin{table}[t] -\caption{B result slots: paired within-backbone local-audio gains (unmeasured). -MiniCPM requires a matched readout/refresh/history audit before comparison.} +\caption{Pending paired $\G\rightarrow\G+\A$ transfer (unmeasured). +EC is EasyCom; MiniCPM first needs readout/refresh/history audits.} \label{tab:backbone} \centering\normalsize \setlength{\tabcolsep}{3pt} @@ -264,9 +280,10 @@ MiniCPM requires a matched readout/refresh/history audit before comparison.} \toprule Backbone & $\G$ & $\G+\A$ & $\Delta$ \\ \midrule -Qwen Omni 3B & -- & -- & -- \\ -Qwen Omni 7B & -- & -- & -- \\ -MiniCPM-o 4.5 & -- & -- & -- \\ +EC: Omni 3B & -- & -- & -- \\ +EC: Omni 7B & -- & -- & -- \\ +EC: MiniCPM-o 4.5 & -- & -- & -- \\ +Ego4D: Omni 3B & -- & -- & -- \\ \bottomrule \end{tabular} \end{table} @@ -320,6 +337,15 @@ single-seed screen; the information audit uses seeds 0, 1, and 2 with an identical fixed recipe. We therefore report standard deviations only for the latter and treat small differences in the former descriptively. +\textbf{Ego4D transfer protocol.} +Ego4D audiovisual clips \cite{grauman2022ego4d} use the existing EgoSpeak +manifest and targets, with 346 training and 87 test clips. The same 5-Hz +cutoffs and local 1-s encoder windows define $\G$, $\G+\A$, $\G+\V$, and +$\G+\A+\V$ comparisons; visual validity is retained rather than silently +dropping no-box rows. The repository contains the extraction and training +workflow but no archived Ego4D score or matched PTR control. Hence this +protocol is specified, not reported as an observed cross-dataset gain. + \textbf{Metrics.} At each horizon we compute AP for the three activity channels over common causal-valid rows, then average classes to obtain mAP and average all ten @@ -345,25 +371,22 @@ transfer, (H/Q) error slices and timelines, and (S) full-path efficiency. They share the official split and physical cutoffs; the 8-s T/C row reuses one run. The source retains unmeasured hypotheses and falsifiers. -\section{Results} +\section{Results and Analysis} \label{sec:results} -\subsection{Global Context and Recent Audio Are Complementary} +\subsection{EasyCom: Local Encoder Evidence} -Table~\ref{tab:main} shows that $\A$ is strong at 0.2~s, while $\G$ is -slightly stronger when mAP is averaged to 2.0~s. Their fusion is better than -$\A$ at every horizon: the mean gain is 4.60 points, with gains of 2.03 at -0.2~s and 4.47 at 2.0~s. It also improves every class when averaged over -horizons (+8.72 background, +2.43 wearer, and +2.65 others). Relative to -released EgoSpeak, $\G+\A$ improves 0.2-s mAP by 7.52 points and mean mAP by -3.92 points. The endpoint effect is asymmetric: offset AP rises by 7.23 -points, while onset AP rises by 1.47. Dense forecasting gains therefore do -not imply equal progress at both behavior boundaries. +Table~\ref{tab:main} shows that recent encoder-side audio $\A$ is strong at +0.2~s while integrated $\G$ is slightly stronger across all horizons. +Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 over +$\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at +0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23 +points, onset AP by only 1.47: dense gains do not imply equal improvement +at both activity boundaries. -The global-only head has more trainable parameters and a longer context than -the local-only head, so this comparison demonstrates predictive -complementarity, not that semantic reasoning alone causes the gain. A wider -history-matched control is required for a representation-level causal claim. +The $\G$-only head has more trainable parameters and longer history than the +$\A$-only head. Thus this screen shows predictive complementarity, not a +capacity-matched proof that the Thinker lacks a specific local cue. % Retained as a planning aid; illustrative, unmeasured patterns do not belong % in the page-limited submission preview. @@ -429,7 +452,7 @@ not real-time deployment. \\ \end{table*} \fi -\subsection{Generic Vision Does Not Recover the Available Signal} +\subsection{EasyCom: Visual Evidence and Ablations} Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and +0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when @@ -470,53 +493,45 @@ behavioral improvement is asserted by this placeholder.} \label{fig:hq_cases} \end{figure} -\subsection{Pending Controls and Limits} +\subsection{Chunk Timing and Cross-Dataset Limits} -Tables~\ref{tab:stream_context}--\ref{tab:efficiency} reserve result slots -for T/C, B, H, and S; Fig.~\ref{fig:hq_cases} reserves Q cases. Panel B of -Table~\ref{tab:stream_context} separately reports a global-only chunk screen: -the 200-ms commits beat 2-s hold after startup exclusion but require about -ten times as many processor calls across 12 sessions. Freshness and chunk -construction both change, and the older tail bank is not extraction-matched; -these values do not fill the pending T/C factorial or establish real-time -throughput. L requires a -trained width-matched global control and past-delayed features on identical -valid pairs; T crosses tail refresh with local audio; C re-extracts each -causal context cap and pairs it with a global-only run. B compares the -within-backbone audio increment only after readout and history audits. -H/Q reports slice counts and cutoff-marked examples; S measures full-path -latency and memory. The oracle uses privileged labels, provided boxes omit -online detection, and aggregate others cannot establish addressee or -semantic floor ownership. +Table~\ref{tab:stream_context}, panel B, reports a separate global-only +chunk-construction screen. Non-native 200-ms commits exceed native 2-s hold +at 0.2~s (67.05 versus 59.62\% mAP), but use 95,453 rather than 9,539 +processor calls over 12 sessions. Completed blocks plus observed tail reach +70.85\%; that older bank is not extraction-matched. Freshness and chunk +construction vary together, so this is motivation for, not evidence of, an +isolated PTR benefit or real-time throughput. Panel A reserves the matched +tail $\times$ audio and context-cap tests. Tables~\ref{tab:backbone}-- +\ref{tab:efficiency} reserve backbone, hard-slice, and cost analyses; the +cutoff-marked H/Q case in Fig.~\ref{fig:hq_cases} remains unmeasured. +Ego4D has no archived test metrics in this checkout, so transfer cannot yet +be claimed. The oracle uses privileged labels, provided boxes omit online +detection, and aggregate others cannot establish addressee or semantic floor +ownership. \subsection{Decision Rules and Reproducibility} -The unit of comparison is the same physical cutoff and future target time, -not an arbitrary feature row. For every new condition we will archive -input-source timestamps, common valid-pair counts, seeds, per-session AP, -trainable parameter counts, processor/model revisions, checkpoint hashes, -and prediction scores. Frames and horizons are correlated, not independent -replicates. - -We retain the fixed four-epoch recipe for the matched matrix. Any loss -weight, threshold, context selection, or case-selection rule is frozen -using training-side evidence before looking at Sessions~1--3. Because -these test sessions have already informed this research program, even -new seed runs remain exploratory rather than restoring an unseen test set. -The raw endpoint AP in Table~\ref{tab:main} may motivate a behavioral -hypothesis, but an ENTRY/YIELD error-rate claim additionally requires -one-event-one-vote attribution with response windows censored at the next -other onset and yield windows at the next other offset. Until that -competing-risk evaluator passes real-timeline audit, we report only dense -forecasting and endpoint anticipation. +Comparisons require identical physical cutoffs and future targets. New runs +must archive source timestamps, common valid-pair counts, seeds, per-session +AP, trainable counts, revisions, checkpoints, and predictions; correlated +frames are not independent replicates. The matched matrix keeps a fixed +four-epoch recipe, with selection rules frozen on training-side evidence. +Sessions~1--3 have already been inspected, so repeat seeds remain exploratory. +Raw endpoint AP is not one-event-one-vote ENTRY/YIELD: response windows must +be censored at the next other onset and yield windows at the next other +offset. Until that evaluator passes a real-timeline audit, we report dense +forecasting and endpoint anticipation only. \section{Conclusion} -In this strictly causal audit, frozen Omni context and recent audio improve -wearer activity forecasting, especially future speech cessation. Generic -pooled vision does not recover the timely visible-activity signal exposed by -the analysis-only oracle. Matched controls and censored event evaluation -remain necessary before claiming improved floor behavior. +On EasyCom, a short-window audio-encoder path complements the frozen Omni +readout for future wearer activity, most clearly at speech offset. Pooled +vision recovers little of the timely visible-activity information exposed by +the analysis-only audit. Partial-tail refresh provides a causal 5-Hz +interface, but its independent predictive and efficiency effects await +matched tests. Ego4D transfer and censored event evaluation remain necessary +before claiming generalizable or improved semantic floor behavior. \section{Acknowledgment} OpenAI Codex assisted with drafting and figure layout; the authors remain diff --git a/manuscript/targeted/references.bib b/manuscript/targeted/references.bib index efbc328..894f47b 100644 --- a/manuscript/targeted/references.bib +++ b/manuscript/targeted/references.bib @@ -61,3 +61,148 @@ year = {2025}, doi = {10.1109/ICASSP49660.2025.10888166} } + +@inproceedings{russell2025mmvap, + author = {Sam O'Connor Russell and Naomi Harte}, + title = {Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction}, + booktitle = {Findings of ACL}, + pages = {209--221}, + year = {2025}, + doi = {10.18653/v1/2025.findings-acl.12} +} + +@inproceedings{elmers2025triadic, + author = {Mikey Elmers and Koji Inoue and Divesh Lala and Tatsuya Kawahara}, + title = {Triadic Multi-party Voice Activity Projection for Turn-taking in Spoken Dialogue Systems}, + booktitle = {Proc. Interspeech}, + year = {2025}, + url = {https://www.isca-archive.org/interspeech_2025/elmers25_interspeech.html} +} + +@inproceedings{lin2025backchannel, + author = {Yuxin Lin and Yinglin Zheng and Ming Zeng and Wangzheng Shi}, + title = {Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals}, + booktitle = {Proc. ACL}, + pages = {15310--15322}, + year = {2025}, + doi = {10.18653/v1/2025.acl-long.743} +} + +@inproceedings{lee2024eot, + author = {Meng-Chen Lee and Zhigang Deng}, + title = {Online Multimodal End-of-Turn Prediction for Three-party Conversations}, + booktitle = {Proc. International Conference on Multimodal Interaction}, + pages = {57--65}, + year = {2024}, + doi = {10.1145/3678957.3685742} +} + +@article{jiang2026turnbench, + author = {Freeman Jiang and others}, + title = {{TurnBench}: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue}, + journal = {arXiv preprint arXiv:2608.25218}, + year = {2026} +} + +@inproceedings{grauman2022ego4d, + author = {Kristen Grauman and Andrew Westbury and Eugene Byrne and others}, + title = {{Ego4D}: Around the World in 3,000 Hours of Egocentric Video}, + booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition}, + pages = {18995--19012}, + year = {2022} +} + +@inproceedings{kong2024sicnet, + author = {Deqian Kong and Furqan Khan and Xu Zhang and Prateek Singhal and Ying Nian Wu}, + title = {Long-Term Social Interaction Context: The Key to Egocentric Addressee Detection}, + booktitle = {Proc. IEEE International Conference on Acoustics, Speech and Signal Processing}, + year = {2024} +} + +@inproceedings{jiang2022asl, + author = {Hao Jiang and Calvin Murdock and Vamsi Krishna Ithapu}, + title = {Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization}, + booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition}, + year = {2022} +} + +@inproceedings{clarke2025asd, + author = {Jason Clarke and Yoshihiko Gotoh and Stefan Goetze}, + title = {Speaker Embedding Informed Audiovisual Active Speaker Detection for Egocentric Recordings}, + booktitle = {Proc. IEEE International Conference on Acoustics, Speech and Signal Processing}, + year = {2025}, + url = {https://arxiv.org/abs/2502.06012} +} + +@inproceedings{wang2024loconet, + author = {Xizi Wang and Feng Cheng and Gedas Bertasius and David Crandall}, + title = {{LoCoNet}: Long-Short Context Network for Active Speaker Detection}, + booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition}, + year = {2024} +} + +@inproceedings{oconnorrussell2025noise, + author = {Sam O'Connor Russell and Naomi Harte}, + title = {Visual Cues Support Robust Turn-taking Prediction in Noise}, + booktitle = {Proc. Interspeech}, + year = {2025}, + url = {https://www.isca-archive.org/interspeech_2025/oconnorrussell25_interspeech.html} +} + +@article{cui2026minicpmo45, + author = {Junbo Cui and Bokai Xu and Chongyi Wang and others}, + title = {{MiniCPM-o 4.5}: Towards Real-Time Full-Duplex Omni-Modal Interaction}, + journal = {arXiv preprint arXiv:2604.27393}, + year = {2026} +} + +@article{tian2026roma, + author = {Xueyun Tian and Wei Li and Bingbing Xu and Heng Dong and Yuanzhuo Wang and Huawei Shen}, + title = {{ROMA}: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding}, + journal = {arXiv preprint arXiv:2601.10323}, + year = {2026} +} + +@article{yan2026duplug, + author = {Ruiqi Yan and Wenxi Chen and Zhanxun Liu and others}, + title = {{SoulX-Duplug}: Plug-and-Play Streaming State Prediction Module for Realtime Full-Duplex Speech Conversation}, + journal = {arXiv preprint arXiv:2603.14877}, + year = {2026} +} + +@article{yu2025salmonnomni, + author = {Wenyi Yu and Siyin Wang and Xiaoyu Yang and others}, + title = {{SALMONN-omni}: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation}, + journal = {arXiv preprint arXiv:2505.17060}, + year = {2025} +} + +@inproceedings{chen2024videollmonline, + author = {Joya Chen and others}, + title = {{VideoLLM-online}: Online Video Large Language Model for Streaming Video}, + booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition}, + year = {2024} +} + +@inproceedings{wang2025omnimmi, + author = {Yuxuan Wang and others}, + title = {{OmniMMI}: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts}, + booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition}, + year = {2025} +} + +@inproceedings{lin2023lavish, + author = {Yan-Bo Lin and Yi-Lin Sung and Jie Lei and Mohit Bansal and Gedas Bertasius}, + title = {Vision Transformers Are Parameter-Efficient Audio-Visual Learners}, + booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition}, + pages = {2299--2309}, + year = {2023} +} + +@inproceedings{rouditchenko2024whisperflamingo, + author = {Andrew Rouditchenko and Yuan Gong and Samuel Thomas and Leonid Karlinsky and Hilde Kuehne and Rogerio Feris and James Glass}, + title = {{Whisper-Flamingo}: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation}, + booktitle = {Proc. Interspeech}, + pages = {2420--2424}, + year = {2024} +}