icassp2027-experiment/manuscript/main/main.tex

% ICASSP 2027 content skeleton. The final paper must use the official 2027
% author kit and contain no visible \todo{} markers. Planned experiments are
% intentionally marked rather than presented as completed work.
\documentclass{article}
\usepackage{spconf}
\usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}

\newcommand{\todo}[1]{\textcolor{red}{\textbf{[TODO: #1]}}}
\newcommand{\vect}[1]{\boldsymbol{#1}}
\setlength{\emergencystretch}{2em}

\title{Is It My Turn? Causal Local Evidence Adaptation of Omni Models for
Egocentric Floor Forecasting}

% TODO: Insert the final author list and affiliations after the ICASSP 2027
% review policy and author order are finalized.
\name{Internal Manuscript Draft}
\address{}

\begin{document}
\ninept
\maketitle

\begin{abstract}
Real-time audiovisual agents must decide not only what to say, but whether an
emerging floor opportunity belongs to them. This is difficult in multiparty
egocentric interaction: the target wearer is behind the camera, while the
visible people provide only indirect evidence about whether the surrounding
floor remains occupied. We study strictly causal forecasting of the wearer's
future speech activity every 200~ms. Although a general-purpose Omni model
provides rich long-context audio--visual representations, its global readout is
not explicitly trained to expose sub-second, wearer-relative floor evidence.
We therefore study a candidate causal local floor adapter that combines the global
Omni state with short-window mixed audio and an anonymous set of visible-person
observations, without assuming persistent speaker identity. On EasyCom,
\todo{report the final three-seed dense and committed-event
results: EOT-response and INT-yield recall, false-positive rate, latency,
confidence intervals, and oracle-gap recovery}. Controlled
\todo{time-shift/remove-vision/same-rate or capacity-matched} interventions
show that \todo{state the supported mechanism, without claiming addressee or
speaker identity}. These results demonstrate \todo{the narrow conclusion
actually supported by the final experiments}.
\end{abstract}

\begin{keywords}
conversational agents, turn-taking, egocentric video, multimodal adaptation,
voice activity projection
\end{keywords}

\section{Introduction}
\label{sec:intro}

Natural turn-taking requires a conversational system to anticipate not only
when speech will stop, but whether a floor opportunity actually belongs to it.
In a dyadic exchange, a pause after one interlocutor finishes often invites the
other to respond. In a group conversation, the same pause may instead mark a
handover between two other participants. An egocentric agent that reacts only
to silence or aggregate speech may therefore interrupt a conversation that
should continue without it. We study \emph{egocentric floor forecasting}:
continuously predicting the near-future speech participation of the
first-person wearer from strictly causal audio--visual observations.

Voice Activity Projection (VAP) introduced continuous prediction of future
voice activity for turn-taking \cite{ekstedt2022vap}. EgoSpeak brought this
formulation to causal, untrimmed first-person streams, using future camera-
wearer activity as an imitation target \cite{kim2025egospeak}. Complementary
work adds visual gaze, head pose, and facial behavior to predictive turn-taking
\cite{russell2025mmvap}, while MuVAP combines global acoustic dynamics with
visible face tracks to forecast visible floor holders in multiparty scenes
\cite{qi2026muvap}. Our target has a different role structure: the participant
whose future action is predicted is the one participant absent from the
visible set. The visible people are contextual evidence for the unseen
wearer's decision, not candidate output identities.

This role asymmetry exposes an adaptation problem for general-purpose Omni
models. Omni pretraining favors globally integrated representations useful for
multimodal understanding and generation. Egocentric floor control instead
depends on timely prosodic and visible-person activity at sub-second scale,
including whether other participants continue to occupy the floor. Increasing
model capacity or applying a generic parameter-efficient update does not
explicitly preserve or route this evidence to a dense wearer-relative
prediction head. The open question is therefore not merely whether an Omni
backbone can be applied to EgoSpeak, but what task structure is still required
to turn general audiovisual understanding into reliable floor control.

Our candidate decomposition separates long-context understanding from
short-window floor evidence. A
causal global Omni state represents the broader scene and conversation. In
parallel, a lightweight local path encodes recent mixed audio and a changing,
anonymous set of visible-person observations. A residual adapter makes this
local evidence accessible to the common ten-horizon activity head. The primary
variant deliberately avoids cross-time identity: the current development
oracle supports timely aggregate visible-other activity but not person
continuity. Track-aware structure is considered only if it defeats no-ID,
identity-corrupted, and capacity-matched controls.

The final paper will claim the following three contributions only if the
corresponding TODO gates pass:
\begin{itemize}
  \item We identify and test a temporal--role mismatch between a global Omni
  readout and causal egocentric floor forecasting for an invisible wearer.
  \item We evaluate a wearer-relative causal local floor adapter that combines
  long-context Omni information with short-window audio and anonymous
  visible-person evidence.
  \item We evaluate whether the adapter reduces erroneous wearer commits
  during multiparty floor transfers under an EOT/INT event protocol with
  explicitly annotated hard-negative regions.
\end{itemize}

\begin{figure}[t]
\centering
\fbox{\parbox[c][1.00in][c]{0.94\columnwidth}{\centering
\todo{Draw only the selected method: causal AV to the long-context Omni path;
the short-window mixed-audio and anonymous person-set path; residual fusion;
the common $10\times3$ head; and the separate EOT-response/INT-yield evaluation.
Mark physical cutoffs, trainable modules, person masks, and NULL/OFFSCREEN.
Do not draw oracle labels, rejected track variants, or multiple backbones.}}}
\caption{Long-context Omni understanding and short-window local floor evidence
for wearer-relative future-activity prediction.}
\label{fig:method}
\end{figure}

\section{Method}
\label{sec:method}

\subsection{Causal wearer-relative projection}

At decision time $t$, the model observes only the stream
$\mathcal X_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s. For
bin $k$, the multilabel target
$\vect y_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
background, camera-wearer activity, and activity by one or more other people.
Wearer--other overlap is allowed. The model outputs
\begin{equation}
 \hat{\vect Y}_t=f_\theta(\mathcal X_{\leq t})
 \in[0,1]^{K\times3}.
 \label{eq:task}
\end{equation}
The wearer-activity forecast also provides a causal control score. When the
wearer is silent, $\hat y^{\mathrm w}_{t,1}$ acts as a near-term take-turn
score; while the wearer is active, $1-\hat y^{\mathrm w}_{t,1}$ acts as a
yield score. These scores are trained only from future activity targets.
Committed EOT-response and INT-yield events used for behavioral evaluation are
defined separately from participant-resolved annotations in
Sec.~\ref{sec:experiments}.

\subsection{Global Omni baseline}

Let $\vect g_t\in\mathbb R^{d_g}$ be the causal global readout of
Qwen2.5-Omni at the physical cutoff $t$ \cite{xu2025qwenomni}. The frozen
baseline trains only a dimension-matched prediction head. A matched PEFT/LoRA
baseline updates a pre-registered subset of the Omni model under the same
examples, context, optimizer budget, head, and evaluator. Its purpose is to
test whether generic parameter adaptation is sufficient; one negative recipe
is not interpreted as a general limitation of LoRA or all Omni models.

\todo{Freeze the exact Qwen2.5-Omni-3B revision, processor, input rates,
streaming/cache semantics, readout layer, context, dtype, LoRA targets/rank,
trainable parameter budget, and cutoff audit. Add Qwen2.5-Omni-7B only as a
capacity-scaling control. MiniCPM-o 4.5 is restricted to the diagnostic
streaming/cache ablation below and does not enter the core method or support a
cross-family superiority claim.}

\subsection{Causal local floor adapter}

The candidate local path operates on the most recent $L_\ell$ seconds. At local instant
$\tau\leq t$, a causal audio encoder produces $\vect a_\tau$ from mixed audio.
For the anonymous visible-person set
$\mathcal P_\tau=\{(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau}$,
a shared visual encoder and masked set operator produce
\begin{equation}
 \vect v_\tau=\mathrm{SetPool}
 \left(\{E_v(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau}
 \cup\{\vect v_\varnothing\}\right),
 \label{eq:set}
\end{equation}
where $\vect v_\varnothing$ handles an empty or offscreen visible set. The
operator is permutation invariant and uses no participant-ID embedding or
cross-time identity. A causal short-window encoder summarizes local dynamics,
\begin{equation}
 \vect \ell_t=F_\ell
 \left(\{[\vect a_\tau;\vect v_\tau]\}_{\tau\in(t-L_\ell,t]}\right).
 \label{eq:local}
\end{equation}
The adapter preserves the global initialization through a zero-initialized
residual output,
\begin{equation}
 \vect z_t=\vect g_t+W_o\,
 \sigma\!\left(W_g\vect g_t+W_\ell\vect\ell_t\right),
 \qquad W_o\vert_{\mathrm{init}}=0,
 \label{eq:fusion}
\end{equation}
and the common head maps $\vect z_t$ to Eq.~\eqref{eq:task}. This implements an
evidence-access hypothesis rather than adding an unconstrained second model:
the global path retains broad context, while the local path preserves timely
floor evidence.

\todo{Freeze $L_\ell$, local audio/video rates, crop source and resolution,
set pooling, causal temporal encoder, masks, reset rules, parameter count, and
latency. If the local path runs faster than the 5-Hz global/output grid, compare
it with a same-parameter 5-Hz variant before using the term ``dual-rate.'' Use
provided causal boxes only as a clearly labeled method-isolation condition;
online detection/tracking is not a paper contribution unless actually run.}

\subsection{Training}

The primary objective is class-balanced binary cross entropy over valid
horizons and activity channels,
\begin{equation}
 \mathcal L_{\mathrm{act}}=
 \sum_{t,k,c}m_{t,k}w_c\,
 \mathrm{BCE}(\hat y^c_{t,k},y^c_{t,k}).
 \label{eq:loss}
\end{equation}
No transcript, target-of-speech label, close microphone, future frame, or
per-person speech label enters the deployable model. \todo{Add an active-speaker
or event auxiliary loss only if it improves the frozen primary protocol and
its label/input boundary is fully reported.}


\section{Experiments}
\label{sec:experiments}

\subsection{Dataset and Data Preparation}

We evaluate on EasyCom following the EgoSpeak split, using Sessions 4--12 as
the training/development pool and Sessions 1--3 only for final evaluation
\cite{donley2021easycom}. The participant-resolved speech annotations are
converted into two parallel representations. First, they are downsampled to
5~Hz to construct the EgoSpeak-compatible ten-horizon targets for background,
camera-wearer, and aggregate non-wearer activity. Second, the original
participant timeline is retained to derive committed interaction events for
behavioral evaluation.

An \emph{EOT-response} is an observed wearer take-turn after a non-wearer floor
release; an \emph{INT-yield} is an observed wearer floor release after a
non-wearer takeover during wearer speech.

\textbf{Hard-negative region definition.}
Committed positive events and hard-negative regions are derived separately
from the participant-resolved speech timeline. Positive EOT-response and
INT-yield events are anchored by observed wearer behavior. In contrast,
hard-negative regions do not require any wearer transition. For example, a
mid-turn pause followed by the same non-wearer resuming speech, or an
other-to-other floor transfer with the wearer remaining silent throughout,
defines an EOT hard-negative region: a take-turn commitment within this region
is counted as a false positive. Likewise, brief backchannels or non-takeover
overlaps during wearer speech define INT hard-negative regions. Ambiguous or
annotation-corrupted regions are excluded from scoring.

Participant identity is used only to construct these evaluation annotations
and is never provided to the prediction model. Provided person boxes are used
only for the causal visible-person input; transcripts, target-of-speech labels,
close-microphone audio, and future observations are unavailable to the model.
\todo{Report duration and event statistics: EOT positives/hard negatives,
INT positives/hard negatives, excluded candidates, and audited-region counts
for train/development and test. Freeze the event-construction and region-audit
protocol before reporting semantic EOT/INT results; until then, retain the
timing-derived proxy terminology.}

\subsection{Model Configuration and Training}

Our main backbone is Qwen2.5-Omni-3B. We compare a frozen global Omni readout,
matched PEFT/LoRA adaptation, global features with recent local audio, a
parameter- and history-matched temporal local branch, and the proposed causal
local audio--visual adapter. Qwen2.5-Omni-7B is used as a capacity-scaling
control. All learned variants use identical causal examples, prediction
horizons, prediction heads, and the class-balanced binary cross-entropy
objective in Eq.~\eqref{eq:loss}.

The diagnostic streaming ablation additionally uses MiniCPM-o 4.5
\cite{cui2026minicpmo45}. Its native cached stream is evaluated at its native
one-second update interval: a state becomes available to the 200-ms row whose
cutoff equals the chunk end and is then causally held until the next update. A
Qwen row subjected to the same 1-Hz update-and-hold schedule separates
refresh-rate effects from backbone effects. A MiniCPM-o
5-Hz cache row is admissible only if a cached branch at every cutoff agrees
with full causal-prefix recomputation within a frozen numerical tolerance and
passes future-input intervention tests. Thus the native 1-Hz condition is a
strictly causal held-state baseline, not a claim of native 5-Hz refresh.

Model selection is performed using training-side data only. Event operating
points are selected from session-level out-of-fold predictions over
Sessions~4--12; the final model is then retrained on all nine sessions before
evaluation on Sessions~1--3. Final results are averaged over three random
seeds.
\todo{Freeze context length, local window/rates, optimizer, epochs, LoRA
configuration, trainable parameter counts, and inference latency.}

\subsection{Evaluation Protocol}

We first retain dense future-activity evaluation for direct comparison with
EgoSpeak, reporting AP at the 0.2--2.0~s horizons and their mean mAP.

Behavioral evaluation follows a committed-event protocol. For both
EOT-response and INT-yield, each gold event is matched to at most one committed
prediction and is scored by recall and signed detection latency.

\textbf{False-positive scoring rule.}
A false positive is defined by the model committing inside a labeled
hard-negative region, regardless of whether the wearer subsequently exhibits
an onset or offset. Thus, false-positive evaluation includes conversational
situations in which the correct wearer behavior is to remain silent or retain
the floor.

We report the recall--false-positive trade-off and a common operating point at
matched recall $R=0.8$. The primary behavioral quantities are therefore EOT
false-positive rate, INT false-positive rate, and signed latency at the frozen
operating point. We additionally report performance on the other-to-other
subset to test whether local visual evidence specifically reduces erroneous
wearer turn-taking during multiparty floor transfers. \todo{Freeze the FPR
denominators and report signed median latency separately for EOT and INT.}


\section{Results and Analysis}
\label{sec:results}

\subsection{Main Comparison}

Table~\ref{tab:main} compares the proposed causal local audiovisual adapter
with an EgoSpeak reproduction, the frozen Qwen2.5-Omni-3B global state $G$,
matched PEFT, $G$ plus recent local audio, and a parameter- and history-matched
temporal local branch. Qwen2.5-Omni-7B is included only as a capacity-scaling
control. All methods use the same split, causal inputs, targets, and evaluation
protocol. Event thresholds are selected on training-side calibration data and
frozen before test evaluation. \todo{Freeze all baseline implementations and
verify identical targets, causal validity, and evaluation.}

\begin{table}[t]
\caption{Confirmatory EasyCom results. EOT and INT false-positive rates are
measured at matched recall $R=0.8$. E/I Lat. gives signed median EOT/INT
latency in seconds; values closer to zero are better.}
\label{tab:main}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccccc}
\hline
Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$
& EOT FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I Lat.\\
\hline
EgoSpeak reproduction       & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-3B frozen $G$          & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-3B matched PEFT        & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$local audio             & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$matched temporal local  & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$causal local AV (ours)  & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-7B scaling control     & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
\hline
\end{tabular}}
\end{table}

\textbf{Dense forecasting.}
The proposed local AV adapter improves 0.2-s mAP from
\todo{$X$ to $Y$} over the strongest matched baseline and improves
mean mAP by \todo{$Z$ points}.
The gain is largest at \todo{short prediction horizons / transition-adjacent
frames / specify actual observation}, indicating that local audiovisual
evidence primarily improves \todo{the interpretation supported by the data}.

\textbf{Committed behavior.}
At matched EOT-response recall $R=0.8$, the proposed adapter reduces EOT FPR
from \todo{$X$} to \todo{$Y$}; at matched INT-yield recall, it changes INT FPR
from \todo{$A$} to \todo{$B$}. Its signed median EOT/INT latencies are
\todo{$+0.XX/+0.XX$~s}. These recall-matched error rates and signed latencies
test whether fewer erroneous commits arise from better floor decisions rather
than indefinite waiting.

\textbf{Comparison with adaptation baselines.}
Matched PEFT yields \todo{result}, whereas adding recent local audio alone
yields \todo{result}.
The parameter- and history-matched temporal branch achieves
\todo{result}.
The remaining gap between this control and the proposed adapter therefore
suggests that \todo{correctly timed audiovisual local evidence contributes
beyond additional temporal capacity}.
\todo{If this difference is not statistically supported, state instead that
the local temporal branch accounts for most of the gain.}

\subsection{Information Opportunity}

Before designing the deployable adapter, we measured whether local floor
information was useful at all using an analysis-only oracle.
The oracle receives perfectly timed visible-other speech activity through a
non-deployable pathway and is never used in the main comparison.

On development Session~12, aggregate visible activity improved 0.2-s mAP by
6.97 points and ten-horizon mean mAP by 2.91 points over the matched
global-plus-local-audio baseline.
In contrast, visibility alone and one-second-delayed activity remained close
to the audio baseline, while person-specific histories did not outperform
identity-shuffled histories.

These controls indicate that the useful signal is primarily
\emph{timely local floor evidence}, rather than participant identity or
static visibility.
The proposed adapter is therefore designed to approximate this information
using only causally available audiovisual observations.

\todo{After completing all seeds, report mean $\pm$ confidence interval here.
If the oracle effect is unstable, describe it qualitatively and omit the exact
numbers from the four-page version.}

\subsection{Mechanism Analysis}

To determine whether the gain arises from correctly aligned visual floor
evidence rather than additional parameters or generic visual context, and to
separate streaming refresh from representation family, we evaluate the
controls in Table~\ref{tab:ablation}.

\begin{table}[t]
\caption{Mechanism and streaming/cache controls. The streaming panel uses
frozen global states and capacity-matched downstream heads; all rows share
targets and evaluation. FPRs are measured at matched recall $R=0.8$.
Cross-family raw scores are diagnostic.}
\label{tab:ablation}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lcccc}
\hline
Condition & Mean mAP $\uparrow$ & EOT FPR $\downarrow$
& O$\rightarrow$O FPR $\downarrow$ & INT FPR $\downarrow$\\
\hline
\multicolumn{5}{l}{\emph{Local-evidence controls (Qwen2.5-Omni-3B)}}\\
Remove vision           & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Time-shift vision       & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Full-frame vision       & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Correct local AV        & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\hline
\multicolumn{5}{l}{\emph{Streaming/cache controls}}\\
Qwen, full causal prefix at 5 Hz (no cache)
                         & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen, cached 5-Hz update & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen, cached 1-Hz + causal hold & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
MiniCPM-o, native cached 1-Hz + causal hold
                         & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
MiniCPM-o, audited cached 5-Hz branch
                         & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}}
\end{table}

Removing local vision changes \todo{EOT/O$\rightarrow$O/INT FPR by $X/Y/Z$},
while shifting the visual stream by \todo{$\Delta t$} produces \todo{result}.
Replacing person-local evidence with full-frame vision produces \todo{result}.
If aligned local AV improves estimates of surrounding floor occupancy, its
largest selective reduction should appear on the O$\rightarrow$O hard-negative
subset. Together, these comparisons \todo{support / do not support} that
mechanism beyond generic visual context.

\todo{Report paired intervals and the supported interpretation. If correct
and shifted/removed vision do not separate, drop the audiovisual
evidence-routing claim. If dense mAP improves but event behavior does not, do
not claim improved floor control. If legitimate onset or yield degrades, do
not describe lower false entry as success.}

\subsection{Scaling and Streaming Controls}

Increasing the frozen Omni backbone from 3B to 7B yields
\todo{$X$ points} in 0.2-s mAP and \todo{EOT/INT FPR of $Y/Z$} at matched
recall.
The proposed 3B adapter \todo{remains better than / approaches / is surpassed
by} the 7B scaling control.

This result suggests that \todo{architectural access to local causal evidence
provides a complementary benefit to backbone scaling}.
Because both backbones belong to the Qwen2.5-Omni family, we do not claim that
this conclusion generalizes to all omni-modal models.

The streaming rows answer two different questions. First, full causal-prefix
recomputation and streaming cache reuse at the same 5-Hz Qwen cutoffs test
whether cache/chunk semantics change the representation; equivalent outputs
make this an efficiency control rather than a distinct predictive method.
Second, the matched Qwen 5-Hz and 1-Hz cached rows test whether an advantage is
explained by more frequent causal state refresh rather than the proposed local
evidence route. For native 1-Hz streams, the state emitted at the end of each
one-second chunk is never backfilled into earlier 200-ms rows; source gaps and
session boundaries reset all modality and language-model caches. MiniCPM-o is
a secondary cross-family diagnostic under the same 1-Hz schedule, not a
same-frequency architecture ranking. Its 5-Hz cached-branch row is omitted if
cached/full-prefix equivalence, cutoff invariance, or reset audits fail.

\todo{Freeze native chunk boundaries, initialization of rows before the first
complete chunk, cache-reset rules, the cached/full-prefix tolerance, and the
future-media intervention test. Report within-backbone 5-Hz versus 1-Hz
deltas plus cache/no-cache throughput and memory; do not interpret unmatched
Qwen--MiniCPM raw scores as a pure backbone effect.}

\subsection{Limitations and Qualitative Analysis}

The results support \todo{precise supported claim}, but several limitations
remain.
First, \todo{EasyCom size/session diversity limitation}.
Second, the adapter relies on \todo{state the actual visible-person box or
tracking assumption}.
Third, \todo{failure condition: occlusion, overlapping speakers, rapid floor
exchange, etc.} remains challenging.

\todo{Add one compact, pre-selected qualitative timeline only if space permits:
one multi-other false-entry improvement paired with one preserved real wearer
onset or one failure. Examples illustrate aggregate rates and never establish
the claim by themselves.}

\section{Conclusion}
\label{sec:conclusion}

This study asks whether a general-purpose Omni representation can be adapted
for causal floor forecasting of an invisible egocentric wearer. The candidate
method separates long-context Omni understanding from short-window local floor
evidence without assuming persistent speaker identity. \todo{State the final
learned-method improvement, matched-recall EOT/INT false-positive rates,
signed latency, causal-control evidence, and main limitation. If no learned
variant passes these tests, reframe the submission as an information audit or
do not make the method claim.}

\appendix
\section{LLM-Assisted Behavioral Event Annotation}
\label{app:event_annotation}

EasyCom provides participant-resolved speech intervals, transcripts, and
target-of-speech metadata, but raw speech transitions do not uniquely specify
conversational floor function. A wearer onset can be a genuine floor entry, a
backchannel, or a within-turn resumption; a wearer offset can reflect yielding,
natural completion, or a temporary pause. We therefore construct a separate
behavioral-event layer while preserving the original speech intervals used for
dense activity targets. The pipeline has three stages: deterministic candidate
construction, language-model-assisted semantic pre-labeling, and mandatory
human adjudication. The language model never supplies final ground truth.

\subsection{Deterministic candidate construction}

\textbf{Floor-level wearer entries.}
Raw wearer onsets are conditionally grouped into floor-level episodes. Let
$g$ be the silence from the preceding wearer offset to a new raw onset. In the
current audit configuration, an onset is stored as a
\textsc{Wearer-Resume} control rather than proposed as an EOT entry when
$g\leq0.8$~s and the gap contains no plausible competing non-wearer floor
activity. A non-feedback contribution that occupies at least $0.4$~s of the
gap, or continues across the resumed wearer onset, conservatively preserves a
new entry candidate. A short ($\leq1.0$~s) feedback-like lexical or nonverbal
contribution is retained as evidence but does not by itself split the wearer
episode. These rules group events only; they do not modify the source
annotations.

EOT candidate construction starts from each observed floor-level wearer onset,
which is retained as the immutable \emph{behavior anchor}. The generator then
searches backward for preceding non-wearer releases. A release remains an
admissible \emph{opportunity-anchor} option only when the wearer stays silent
and no new non-wearer onset or already-active other floor invalidates the
intervening interval. Silence duration alone does not reject an option. Thus
each wearer onset contributes at most one primary review item rather than one
item per preceding release. The semantic annotator must either select one of
the supplied release options as the trigger or reject the EOT interpretation;
it cannot introduce a new timestamp. The selected opportunity and observed
behavior anchors deterministically define the valid prediction interval. A
wearer onset without an admissible release is retained only as an audit control.

\textbf{EOT hard negatives.}
Two participant-resolved timing rules independently propose regions in which
the wearer should remain silent. A \textsc{Same-Speaker-Pause} region is
created when a non-wearer resumes within 2~s and both the wearer and all other
participants remain silent in the intervening gap. An
\textsc{Other-to-Other} region is created when a different non-wearer begins
within 1.5~s, the original speaker does not first resume, and the wearer stays
silent through the handover. The semantic pre-labeler can additionally assign
\textsc{EOT-Wearer-Backchannel} to a floor-level onset that is interpretable
as feedback rather than a response.

\textbf{INT candidates and hard negatives.}
Let $t_w$ be the start of the current floor-level wearer episode, $t_o$ a new
non-wearer onset, and $t_y$ the episode-ending wearer offset. A potential
\textsc{INT-Yield} is proposed only when
\begin{equation}
 t_w < t_o < t_y < t_o^{\mathrm{off}},
 \label{eq:int_candidate}
\end{equation}
with a one-frame (50-ms) tolerance only at the offset boundary in the current
audit. Thus the competing activity must begin after the wearer acquired the
floor and must continue across the wearer yield. Pre-existing non-wearer
activity, post-completion onsets, offsets internal to merged wearer episodes,
and offsets without crossing activity are retained separately as
\textsc{Wearer-Offset-Control} records and are not sent for semantic
pre-labeling. Thus ordinary completion-to-silence cases do not enter the
primary INT review queue.

INT hard-negative regions are also proposed without requiring a wearer
offset. If a non-wearer contribution ends while the wearer remains active
throughout it and for at least another 0.4~s, the region is labeled by rule as
\textsc{INT-Other-Backchannel} when it is short and feedback-like, and
\textsc{INT-Non-Takeover-Overlap} otherwise. These timing labels are candidate
priors, not final semantic annotations.

\subsection{Semantic pre-labeling}

For every primary candidate, the annotation assistant receives the candidate
source and rule type, immutable timestamps, participant-resolved activity,
neighboring onsets and offsets, gap and overlap measurements, transcript
context, and target-of-speech metadata when available. Participant identities
are used only inside this annotation package. The assistant is instructed to
treat timing as factual, not to invent prosody, gaze, gesture, intention, or
other unavailable cues, and not to equate onset with floor entry, offset with
yield, silence with EOT, or overlap with interruption. EasyCom's \texttt{[L]}
marker is treated as evidence of laughter only: it is not by itself a sustained
floor take or takeover. For EOT, a positive decision must name one supplied
opportunity-anchor option; for all other EOT decisions no trigger is selected.

The allowed decisions are \textsc{Positive}, \textsc{Hard-Negative},
\textsc{Not-Applicable}, \textsc{Exclude}, and \textsc{Unsure}. Positive
events are typed as \textsc{EOT-Response} or \textsc{INT-Yield}; hard
negatives use a fixed subtype vocabulary. The current pre-labeling run uses
\texttt{deepseek-v4-flash}, temperature zero, and two passes: a primary pass
and a skeptical pass that seeks alternative explanations. If they disagree on
the verdict, event type, or hard-negative subtype, the combined suggestion is
\textsc{Unsure}. Model identifier, prompt version, per-request prompt hashes,
both raw decisions, confidence values, and explanations are stored with every
candidate.

\subsection{Human adjudication and scoring eligibility}

Every AI-pre-labeled candidate enters an event-centered interface containing
the original video, local participant timeline, transcript, deterministic
timing evidence, and the AI suggestion. The adjudicator performs one primary
action: accept the complete AI label and selected EOT trigger, relabel it, or
exclude an unscorable
sample. Relabeling exposes the same constrained verdict, event-type, and
hard-negative subtype vocabulary. Relabeling a positive EOT also requires one
of the displayed release options. A confidence level is required; notes are
optional, and an optional $\pm0.2$~s boundary correction is available only for
an evident source-annotation error. The original event time is always retained.

Only the completed human decision is eligible to become semantic ground truth.
\textsc{Not-Applicable} denotes a valid observation outside the committed
EOT/INT task, whereas \textsc{Exclude} denotes a region that cannot be scored
reliably because of annotation corruption, a recording boundary, missing
context, timestamp corruption, or another unresolved defect. Neither category
enters the primary event denominator. Transcripts, participant identities,
target-of-speech metadata, annotation-model outputs, and future observations
are never inputs to the evaluated forecasting model.

\todo{Before freezing this annotation protocol, audit randomly sampled
continuous timeline segments without candidate hints to estimate candidate
recall for EOT, INT, and each hard-negative family. Independently double-label
a pre-registered subset, report its size and agreement statistic, freeze all
candidate parameters and prompt/model identifiers on Sessions 4--12, and
report final candidate, control, exclusion, and adjudicated-label counts.}

\bibliographystyle{IEEEbib}
\bibliography{references}

\end{document}