Build ICASSP manuscript and experiment framework

manuscript/main/REVIEWER_CHECKLIST.md

@@ -0,0 +1,141 @@
+# ICASSP 2027 completion checklist
+
+This is an internal execution checklist. It is not part of the submitted PDF.
+
+## Reviewer verdict on the current draft
+
+The research question is potentially publishable: **at matched wearer-response
+recall, can correctly aligned person-level vision reduce false wearer entries
+during third-party conversation without degrading legitimate wearer responses?**
+
+The contribution is not “the first audio-visual multiparty turn model.” VAP,
+EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible
+gap is continuous, causal floor-entry projection for an **invisible camera
+wearer**, conditioned on addressee evidence from a variable participant set.
+
+The current aggregate result (G+A+pooled-V minus G+A: +0.30 points at 0.2 s,
++0.29 mean mAP, one seed) motivates the question but does not establish social
+grounding. Acceptance depends on the event protocol and identity controls.
+
+## P0 — Freeze the task before training a new model
+
+- [ ] Define the exact 5 Hz causal cutoff and physical target interval for each
+  of the ten 0.2 s horizons.
+- [ ] Freeze addressed-response and third-party event definitions.
+- [ ] Add competing-risk rules: an addressed response ends at the first wearer
+  onset, unrelated-other onset, timeout, or invalid annotation boundary.
+- [ ] Specify overlap, minimum silence, maximum response delay, duplicate
+  detections, clip boundaries, group/none/multiple/unknown targets, and
+  offscreen speakers.
+- [ ] Produce train/test event-count tables, duration, class prevalence, and
+  visibility coverage before examining model scores.
+- [ ] Mark all already viewed Sessions 1–3 results as exploratory. From this
+  point forward, lock the event protocol and select thresholds, checkpoints,
+  and all hyperparameters using a training-side fold or cross-fitting over
+  Sessions 4–12. Run the fixed final model on Sessions 1–3 only after that
+  lock; do not describe the historically viewed split as untouched.
+
+**Gate:** two independent code paths or a manual sample audit must agree on at
+least 100 randomly selected events. Do not start the main model sweep before
+this gate passes.
+
+## P1 — Establish matched baselines
+
+Run three seeds with identical optimizer, history, parameter budget, and event
+evaluator:
+
+1. G: frozen streaming omni state.
+2. G+A: global state plus recent audio.
+3. G+A+V-pool: the current pooled visual baseline.
+4. Temporal control: G+A plus a parameter-matched non-person temporal branch.
+
+Report original ten-horizon AP/mAP and the event metrics. Confirm that changing
+batching, cached features, or track padding does not change baseline outputs.
+
+**Gate:** baseline standard deviations are known, and the event evaluator
+reproduces identical scores from saved predictions.
+
+## P2 — Implement the minimum person-set model
+
+- [ ] Shared temporal encoder for every face track.
+- [ ] Each person representation cross-attends to the same recent mixed-audio
+  token sequence; no separated/close-mic audio at inference.
+- [ ] Masked permutation-equivariant set encoder.
+- [ ] Learned OFFSCREEN/NULL element.
+- [ ] Fixed floor/address/third-party role queries.
+- [ ] Zero-initialized gated residual into the frozen global state.
+- [ ] Unit tests for variable `N`, empty visible set, missing frames, track-slot
+  permutation, padding invariance, and strict causal cutoff.
+
+**Gate:** with residual gate zero, logits are numerically identical to G+A; a
+random permutation of track slots leaves logits unchanged.
+
+## P3 — Train only the ablations that answer the claim
+
+Use the same three seeds:
+
+1. WearerSet with activity loss only.
+2. WearerSet + per-person active-speaker auxiliary loss.
+3. WearerSet + active-speaker + pure-target addressee auxiliary loss.
+
+Mask group/unknown/offscreen target annotations from addressee loss. They may
+remain in the activity loss. Do not use LLM pseudo-labels as primary gold.
+
+Stop after these stages unless a clear event-level gain appears. Broad Thinker
+LoRA, 7B, and ROMA experiments are diagnostics, not the core method, because
+the existing runs show negative transfer or no consistent 0.2 s benefit.
+
+## P4 — Run causal visual counterfactuals
+
+For the frozen best checkpoint, evaluate:
+
+- correct person tracks;
+- wrong-person tracks from the same scene;
+- time-shifted tracks with unchanged audio;
+- removed/zero visual features;
+- visibility-conditioned subsets.
+
+All controls must reuse the same model weights. Retraining each control would
+mix representation and optimization effects.
+
+**Acceptance gate:** at matched addressed-response recall, correct vision must
+reduce third-party false entries more than wrong/time-shifted/removed vision,
+without worse response latency or wearer-yield recall. Aggregate mAP alone is
+not sufficient.
+
+## P5 — Statistics and analysis
+
+- [ ] Three training seeds, mean ± sample standard deviation.
+- [ ] Paired cluster bootstrap by session/conversation block for 95% confidence
+  intervals; do not bootstrap adjacent 5 Hz frames independently.
+- [ ] Operating points at addressed recall 0.7, 0.8, and 0.9; headline at 0.8.
+- [ ] Error strata: target visibility, number of visible people, overlap,
+  backchannel/interruption proxy, and offscreen target.
+- [ ] P50/P95 decision latency, throughput, peak GPU memory, and trainable
+  parameters.
+- [ ] Ego4D only as secondary aggregate activity generalization; it cannot
+  validate addressee grounding without comparable target labels.
+
+## Paper assembly order
+
+1. Finalize the event protocol and insert counts in Experimental Setup.
+2. Produce the main matched-recall table before polishing prose.
+3. Produce the correct/wrong/time-shift/remove counterfactual table.
+4. Draw one compact method figure with causal cutoffs and the variable set.
+5. Collapse the preliminary aggregate table into the final main table if page
+   pressure is high.
+6. Add one short failure analysis, not a catalog of all earlier experiments.
+7. Replace the abstract and conclusion TODOs with exact numbers and confidence
+   intervals.
+8. Remove every visible `\todo{}` marker and verify the official ICASSP 2027
+   template/page limit once released.
+
+## Stop/go decision
+
+- **Go:** correct person-level vision provides a statistically credible
+  event-level advantage over G+A, pooled-V, temporal capacity control, and
+  wrong/time-shifted vision.
+- **Reframe:** all visual variants improve similarly. Claim better local
+  temporal context, not addressee grounding.
+- **Stop this method:** aggregate mAP rises but third-party false entries do
+  not fall at matched recall, or legitimate response latency/recall degrades.

manuscript/main/main.tex

@@ -1,12 +1,17 @@
-% Draft ICASSP 2027 manuscript using the official ICASSP 2026 template
+% Internal ICASSP 2027 draft using the official ICASSP 2026 template bundled
-% currently bundled in this repository. Replace the template when the ICASSP
+% in this repository. Replace the class/style files when the 2027 author kit
-% 2027 author kit is released.
+% is released, and remove every visible \todo{} marker before submission.
 % Compile from this directory with: make
 \documentclass{article}
 \usepackage{spconf}
-\usepackage{amsmath,graphicx,hyperref}
+\usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}
 
-\title{NOT EVERY TURN IS MINE: ADDRESSEE-GROUNDED FLOOR-ENTRY PROJECTION IN EGOCENTRIC MULTIPARTY CONVERSATION}
+\newcommand{\todo}[1]{\textcolor{red}{\textbf{[TODO: #1]}}}
+\newcommand{\method}{WearerSet}
+\newcommand{\vect}[1]{\boldsymbol{#1}}
+
+\title{NOT EVERY OPEN FLOOR IS MINE: ADDRESSEE-CONDITIONED FLOOR-ENTRY
+PROJECTION IN EGOCENTRIC MULTIPARTY CONVERSATION}
 
 % TODO: Insert the final author list and affiliations after the ICASSP 2027
 % review policy and author order are finalized.
@@ -18,22 +23,23 @@
 \maketitle
 
 \begin{abstract}
-Egocentric conversational agents must decide not only when a speaking
+An egocentric conversational agent must decide not only when a speaking
-opportunity arises, but also whether that opportunity belongs to them.
+opportunity arises, but also whether that opportunity belongs to it. Existing
-Existing speech-initiation forecasting collapses all non-wearer speakers, so
+speech-activity projection collapses all non-wearer speakers, allowing strong
-strong frame-level accuracy can coexist with intrusive entries while two
+frame-level accuracy to coexist with intrusive entries during conversations
-bystanders converse. We study addressee-grounded floor-entry projection in
+between bystanders. We formulate addressee-conditioned floor-entry projection:
-causal multiparty audio-video streams. Our formulation retains dense forecasts
+dense causal forecasts of wearer and non-wearer activity at 200-ms resolution,
-of wearer and non-wearer activity at 200-ms resolution over a two-second
+conditioned on a variable-cardinality set of visible participants. Our model
-horizon, while grounding the wearer-entry decision in person-aligned visual
+augments a frozen streaming omni-model with person-aligned audio--visual
-context. Using EasyCom target-of-speech annotations, we introduce an event-level
+representations and permutation-invariant role queries for floor state,
-protocol that separates addressed-to-wearer responses from third-party
+wearer address, and third-party interaction. We further introduce an EasyCom
-exchanges and measures false entries at matched response recall, together with
+event protocol that separates addressed-to-wearer responses from third-party
-decision latency. Wrong-person and time-shifted visual controls test whether
+exchanges and reports false entries at matched response recall and decision
-any improvement follows correct social grounding rather than additional model
+latency. Correct-, wrong-person, time-shifted, and removed-video controls test
-capacity. Experiments on EasyCom, with Ego4D as an activity-forecasting
+whether a gain follows social grounding rather than model capacity.
-generalization set, will evaluate this hypothesis under a strictly causal
+\todo{Replace this sentence with the main result: FP/min reduction at 80\%
-protocol.
+response recall, response latency/recall non-degradation, and 95\% confidence
+interval.}
 \end{abstract}
 
 \begin{keywords}
@@ -45,65 +51,308 @@ audio-visual learning
 \label{sec:intro}
 
 Conversational agents must continuously decide whether to speak or keep
-listening. In a dyadic exchange, a pause or an apparent end of turn is
+listening. In a dyadic exchange, a pause or an apparent end of turn is often
-often treated as an invitation to respond. This assumption fails in
+treated as an invitation to respond. This assumption fails in multiparty
-multiparty interaction: one participant may finish an utterance addressed to
+interaction: one participant may finish an utterance addressed to another
-another participant, and an egocentric agent that reacts only to acoustic
+participant, and an egocentric agent that reacts only to acoustic availability
-availability will enter a floor that was never offered to it. The resulting
+will enter a floor that was never offered to it. Such an error is socially more
-error is socially more disruptive than a small framewise prediction error.
+disruptive than a small framewise prediction error. A useful agent must
-Accordingly, a useful model must distinguish \emph{when the floor may change}
+therefore distinguish \emph{when the floor may change} from \emph{whether the
-from \emph{whether the change is directed to the camera wearer}.
+change is directed to the camera wearer}.
 
-Voice Activity Projection (VAP) models future joint speech activity and derives
+Voice Activity Projection (VAP) models future joint speech activity and
-turn-taking events without manually labeling every event
+derives turn-taking events without manually labeling each event
 \cite{ekstedt2022vap}. EgoSpeak extends future activity prediction to causal,
-untrimmed egocentric video, forecasting wearer and non-wearer activity over the
+untrimmed egocentric video, forecasting wearer and aggregate non-wearer
-next two seconds \cite{kim2025egospeak}. Its role-relative output is attractive
+activity over the next two seconds \cite{kim2025egospeak}. Its role-relative
-for a wearable agent, but aggregating all visible participants into a single
+output is attractive for a wearable agent, but an ``others'' channel cannot
-``others'' channel removes the distinction between an utterance addressed to
+distinguish an utterance addressed to the wearer from an exchange between
-the wearer and an exchange among bystanders. Consequently, conventional
+bystanders. Conventional per-frame average precision can consequently be
-per-frame average precision can be dominated by ordinary speech activity and
+dominated by ordinary activity while hiding precisely the false-entry behavior
-can hide precisely the false-entry behavior that visual social cues should
+that visual social cues should prevent. Event-oriented evaluation likewise
-prevent. Recent event-oriented evaluation likewise emphasizes the joint
+emphasizes recall, false alarms, and latency rather than a single frame-level
-trade-off among recall, false positives, and latency rather than a single
+score \cite{jiang2026turnbench}.
-frame-level score \cite{jiang2026turnbench}.
 
-Related work provides two complementary pieces of the problem but does not
+Related work supplies complementary parts of the problem. Egocentric
-unify them. Egocentric talking-to-me detection predicts whether a completed
+talking-to-me detection predicts whether a completed utterance addresses the
-utterance addresses the camera wearer and benefits from long-term social
+camera wearer and benefits from long-term social context
-context \cite{kong2024sicnet}. Multiparty VAP grounds future activity in
+\cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face
-visible face tracks to predict floor shifts and next speakers
+tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}.
-\cite{qi2026muvap}. The former is generally evaluated as utterance-level
+The former is generally utterance-level rather than continuous anticipation;
-addressee classification rather than continuous anticipation, whereas the
+the latter does not ask whether an invisible camera wearer should enter. Our
-latter selects among visible participants and does not ask whether an
+gap is therefore narrower than ``audio--visual multiparty turn taking'': we
-invisible camera wearer should enter. We target the intersection: causal
+study continuous, wearer-relative floor action when the candidate responder is
-floor-entry forecasting for an egocentric agent whose decision depends on the
+behind the camera and the number of visible participants changes over time.
-speaker--addressee relation among the people in view.
 
-We formulate this problem as \emph{addressee-conditioned egocentric activity
+We call this task \emph{addressee-conditioned egocentric floor-entry
-projection}. A global streaming audio-video representation predicts future
+projection}. A global streaming audio--video representation supplies broad
-wearer and non-wearer activity, while a person-aligned visual pathway preserves
+context, while a set branch preserves individual visual histories and their
-participant identity and recent social context. The final wearer-entry score
+alignment to recent mixed audio. Fixed semantic role queries compress a
-can therefore represent both floor availability and evidence that the current
+variable participant set into wearer-address and third-party evidence without
-utterance is directed to the wearer. EasyCom is particularly suitable for this
+assuming a fixed seating order. EasyCom is suitable because it provides
-analysis because it provides synchronized egocentric audio-video, per-person
+synchronized egocentric audio--video, per-person tracks, voice activity, and
-tracks, voice activity, and target-of-speech annotations
+target-of-speech annotations \cite{donley2021easycom}. Target annotations are
-\cite{donley2021easycom}. Target annotations are used for training-side
+used only for training-side supervision and evaluation strata, never as input
-auxiliary supervision and evaluation strata, never as model input at test
+at test time.
-time.
 
-Our contributions are threefold. First, we cast speech initiation as
+Our contributions are threefold. (1) We formulate continuous speech
-recipient-aware floor-entry projection, connecting dense egocentric activity
+initiation as recipient-aware floor-entry projection for an invisible camera
-forecasting with multiparty addressee grounding. Second, we propose an
+wearer. (2) We propose a causal set-based fusion module that grounds mixed
-event-level evaluation that contrasts addressed-to-wearer response events with
+audio in a variable number of visible participants and preserves the original
-third-party exchanges and reports false entries at matched response recall,
+dense EgoSpeak target. (3) We introduce an event evaluation that reports
-alongside latency and the original EgoSpeak metrics. Third, we use
+third-party false entries at matched addressed-response recall, accompanied by
-wrong-person, temporally shifted, and visibility-conditioned visual controls to
+latency and visual counterfactual controls. This design makes the scientific
-test whether improvements arise from the correct social evidence instead of
+claim falsifiable: visual fusion is useful only if \emph{correctly aligned}
-extra feature capacity or globally conservative predictions.
+vision improves the operating trade-off without suppressing legitimate wearer
+responses.
 
-% TODO: Add Method, Experimental Setup, Results, and Conclusion after the
+\begin{figure}[t]
-% event protocol and person-aligned model are frozen.
+\centering
+\fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
+\todo{Draw the final method figure: causal omni state $\vect g_t$; recent
+mixed-audio tokens; a masked set of face-track histories; shared per-person
+audio--visual grounding; three role queries; zero-initialized residual; and
+the $10\times3$ activity head. Mark all causal cutoffs and masks.}}}
+\caption{Proposed wearer-relative set fusion. The number and ordering of
+visible participants may change, while the output remains fixed-size.}
+\label{fig:method}
+\end{figure}
+
+\section{Task and Method}
+\label{sec:method}
+
+\subsection{Wearer-relative activity projection}
+
+At decision time $t$, the model observes only the causal stream
+$\mathcal{X}_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s.
+For bin $k$, the target
+$\vect{y}_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
+background, camera-wearer activity, and activity by one or more other people.
+Wearer--other overlap is allowed; hence this is not a three-way softmax. The
+output is
+\begin{equation}
+  \hat{\vect{Y}}_t=f_\theta(\mathcal{X}_{\leq t})\in[0,1]^{K\times3},
+  \qquad q_t=\hat y^{\mathrm{w}}_{t,1},
+  \label{eq:task}
+\end{equation}
+where $q_t$ is the immediate wearer-entry score used by the event protocol.
+It is interpreted as an \emph{entry} score only in wearer-silent opportunity
+windows; elsewhere it remains a future-activity probability. This retains the
+EgoSpeak target and evaluation while exposing a deployable 5-Hz decision
+signal.
+
+\subsection{Frozen global stream and local participant set}
+
+We obtain a causal global state $\vect{g}_t$ from the streaming Thinker of
+Qwen2.5-Omni \cite{xu2025qwenomni}. The backbone is frozen in the main
+experiment because preliminary LoRA adaptation caused negative transfer
+(Sec.~\ref{sec:results}). From the most recent $L_a$ seconds, a shared audio
+encoder produces token sequence $\vect{A}_t$. For each visible track
+$i\in\{1,\ldots,N_t\}$, a shared temporal visual encoder produces
+$\vect{v}_{t,i}$ from its recent face history. Person-aligned evidence is
+\begin{equation}
+ \vect{p}_{t,i}=\vect{v}_{t,i}+
+ \mathrm{CA}(\vect{v}_{t,i},\vect{A}_t,\vect{A}_t),
+ \label{eq:person}
+\end{equation}
+so every participant queries the same mixed audio rather than requiring
+unavailable separated microphones. A masked set encoder models interactions
+among $\mathcal{P}_t=\{\vect{p}_{t,i}\}_{i=1}^{N_t}$ and is permutation
+equivariant \cite{lee2019settransformer}. Missing detections use a learned
+\textsc{offscreen/null} element. In minibatches, sets are padded to
+$N_{\max}$ with an attention mask; no slot-ID embedding is used, and track
+slots are randomly permuted during training.
+
+\todo{Implement and verify the shared track encoder, mixed-audio
+cross-attention, OFFSCREEN token, padding mask, and numerical invariance under
+track-slot permutation. Freeze $L_a$, visual crop history, $N_{\max}$, and
+face-track confidence filtering from train-side experiments only.}
+
+\subsection{Role-query fusion}
+
+Mean pooling can erase which participant is speaking to whom. Instead, three
+learned semantic queries represent floor state, address-to-wearer evidence,
+and third-party interaction. Each query is conditioned on $\vect{g}_t$ and
+cross-attends to the participant set:
+\begin{align}
+ \vect{Q}_t &= [\vect{q}_{\mathrm{floor}};
+               \vect{q}_{\mathrm{addr}};
+               \vect{q}_{\mathrm{third}}]+W_g\vect{g}_t,\\
+ \vect{R}_t &= \mathrm{MHA}(\vect{Q}_t,\widetilde{\mathcal P}_t,
+                            \widetilde{\mathcal P}_t),\\
+ \vect{z}_t &= \vect{g}_t+\tanh(\alpha)W_r\mathrm{vec}(\vect{R}_t),
+ \label{eq:fusion}
+\end{align}
+where $\alpha$ is initialized to zero. The zero-initialized residual makes the
+starting model identical to the frozen global baseline and allows the visual
+branch to be removed exactly. A two-layer prediction head maps $\vect{z}_t$ to
+Eq.~\eqref{eq:task}.
+
+\subsection{Training objectives}
+
+The main loss is class-balanced binary cross entropy over all horizons and
+activity channels. When annotations are available, auxiliary heads predict
+per-track active-speaker state and whether the current source addresses the
+wearer:
+\begin{equation}
+ \mathcal{L}=\mathcal{L}_{\mathrm{act}}+
+ \lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}+
+ \lambda_{\mathrm{addr}}\mathcal{L}_{\mathrm{addr}}.
+ \label{eq:loss}
+\end{equation}
+Ambiguous, group-addressed, and offscreen-target samples are masked from
+$\mathcal{L}_{\mathrm{addr}}$, but remain in activity training. No language
+model or pseudo-label is used to define primary test events.
+
+\todo{Run the staged ablation: activity only; +ASD; +addressee. Select
+$\lambda$ values and early stopping on a training-side calibration fold. Do
+not inspect Sessions 1--3 during selection.}
+
+\section{Experimental Setup}
+\label{sec:setup}
+
+\textbf{Data and splits.}
+We use the EgoSpeak EasyCom protocol: Sessions 4--12 for model development and
+Sessions 1--3 for final reporting. All architecture, checkpoint, threshold,
+and hyperparameter choices use training-side folds only. Inputs contain the
+camera stream and shared array audio available to the wearer; close-talking
+microphones and target-of-speech labels are supervision only. We also report aggregate
+activity forecasting on the EgoSpeak Ego4D split \cite{grauman2022ego4d}, but
+do not claim addressee grounding there because comparable target annotations
+are absent.
+
+\textbf{Event construction.}
+An \emph{addressed response} starts from a pure other-to-wearer utterance and
+requires a wearer onset before an unrelated other onset. A \emph{third-party
+exchange} starts from a pure other-to-other target while the wearer remains
+silent. Group, none, multiple, and unknown targets form a separate diagnostic
+stratum. Threshold $\tau_r$ is selected on the development split to obtain
+addressed-response recall $r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary
+metric is false wearer entries per minute of third-party opportunity at
+matched recall; the headline operating point is $r=0.8$. We also report
+addressed-response recall, signed detection latency, wearer yield recall, and
+the original ten-horizon per-frame mAP.
+
+\todo{Freeze the event pairing and competing-risk rules, including pre/post
+windows, minimum silence, maximum response delay, overlap handling, and one
+detection per event. Publish counts for train/test positive and negative
+events, visibility, duration, and excluded target types. Treat all previously
+viewed Sessions 1--3 scores as exploratory; lock the final protocol and model
+before the confirmatory three-seed run.}
+
+\textbf{Baselines and controls.}
+We compare: global omni state (G); G plus recent audio (G+A); G+A plus pooled
+visual context (G+A+V-pool); a parameter-matched temporal branch without
+person identity; and the proposed set branch. Required counterfactuals replace
+correct tracks with (i) wrong participants from the same scene, (ii)
+time-shifted tracks, and (iii) zeros. These controls share parameter count,
+audio history, and optimization budget.
+
+\textbf{Implementation and statistics.}
+All models run at 5~Hz with an 8-s causal memory. The 3B Thinker is frozen and
+local audio/face histories are encoded causally. We report three seeds and
+mean$\pm$standard deviation. Confidence intervals for event metrics use paired
+cluster bootstrap over sessions/conversation blocks rather than treating
+adjacent frames as independent. Runtime includes P50/P95 decision latency,
+peak memory, and trainable parameter count.
+
+\todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions,
+$L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter
+count, hardware, and real-time latency after the architecture is frozen.}
+
+\section{Results and Analysis}
+\label{sec:results}
+
+\begin{table}[t]
+\caption{Current aggregate evidence on EasyCom test Sessions 1--3. Values are
+percentages from one seed and are motivation, not the final main result.}
+\label{tab:preliminary}
+\centering
+\small
+\begin{tabular}{lccc}
+\hline
+Model & 0.2-s mAP & 0.2-s wearer AP & Mean mAP\\
+\hline
+G+A             & 81.72 & 85.07 & 65.11\\
+G+A+V-pool      & 82.03 & 85.09 & 65.40\\
+\hline
+$\Delta$ visual & +0.30 & +0.02 & +0.29\\
+\hline
+\end{tabular}
+\end{table}
+
+Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
+Pooled vision gives a small gain, while immediate wearer AP is nearly
+unchanged. This is compatible with either weak but useful social grounding or
+generic visual regularization; only matched-recall event evaluation and
+counterfactual identity controls can distinguish them.
+
+\begin{table}[t]
+\caption{Primary EasyCom event evaluation. Lower $\mathrm{FP}_{\mathrm{3p}}$
+is better; all rows use the same development-selected operating protocol.}
+\label{tab:main}
+\centering
+\small
+\resizebox{\columnwidth}{!}{%
+\begin{tabular}{lccccc}
+\hline
+Model & $\mathrm{FP}_{\mathrm{3p}}$/min@$R=.8$ & $R_{\mathrm{addr}}$ &
+Latency & Yield R & Mean mAP\\
+\hline
+G                 & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+G+A               & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+G+A+V-pool        & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+Temporal control  & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+\method{}         & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+\method{}+ASD     & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+\method{}+Addr    & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+\hline
+\end{tabular}}
+\end{table}
+
+\todo{Report Table~\ref{tab:main} as mean$\pm$standard deviation over three
+seeds, plus 95\% paired cluster-bootstrap confidence intervals for the
+proposed-vs-G+A difference. The paper's main claim is supported only if
+third-party FP/min decreases at matched recall without worse response latency
+or wearer-yield recall.}
+
+\begin{table}[t]
+\caption{Visual grounding counterfactual on the frozen best model.}
+\label{tab:counterfactual}
+\centering
+\small
+\begin{tabular}{lccc}
+\hline
+Visual evidence & $\mathrm{FP}_{\mathrm{3p}}$/min & Addr. recall & 0.2-s mAP\\
+\hline
+Correct tracks    & \todo{--} & \todo{--} & \todo{--}\\
+Wrong person      & \todo{--} & \todo{--} & \todo{--}\\
+Time shifted      & \todo{--} & \todo{--} & \todo{--}\\
+Removed           & \todo{--} & \todo{--} & \todo{--}\\
+\hline
+\end{tabular}
+\end{table}
+
+Table~\ref{tab:counterfactual} is the identity test. A true social-grounding
+effect should be largest for correct tracks and should collapse for wrong or
+time-shifted people. If every visual variant improves equally, the defensible
+conclusion is additional capacity/history, not addressee grounding.
+
+\todo{Add one compact error analysis split by target visibility, number of
+visible people, overlap, and backchannel/interruption proxy. Include event
+counts with every rate. Move negative LoRA/7B/ROMA diagnostics to supplementary
+material unless they directly explain the final method.}
+
+\section{Conclusion}
+\label{sec:conclusion}
+
+We study when an egocentric agent should enter a multiparty floor, not merely
+whether someone will speak. The proposed formulation retains dense future
+activity prediction while grounding wearer entry in a variable set of visible
+participants. Its event protocol tests the intended behavior directly:
+reducing intrusions into third-party exchanges at matched legitimate-response
+recall. \todo{Replace with one quantitative conclusion and one limitation;
+do not claim success from aggregate mAP alone.}
 
 \bibliographystyle{IEEEbib}
 \bibliography{references}

manuscript/main/references.bib

@@ -40,3 +40,26 @@
   journal = {arXiv preprint arXiv:2107.04174},
   year    = {2021}
 }
+
+@article{xu2025qwenomni,
+  author  = {Jin Xu and Zhifang Guo and Jinzheng He and Hangrui Hu and Ting He and Shuai Bai and Keqin Chen and Jialin Wang and Yang Fan and Kai Dang and Bin Zhang and Xiong Wang and Yunfei Chu and Junyang Lin},
+  title   = {{Qwen2.5-Omni Technical Report}},
+  journal = {arXiv preprint arXiv:2503.20215},
+  year    = {2025}
+}
+
+@inproceedings{lee2019settransformer,
+  author    = {Juho Lee and Yoonho Lee and Jungtaek Kim and Adam Kosiorek and Seungjin Choi and Yee Whye Teh},
+  title     = {Set Transformer: A Framework for Attention-Based Permutation-Invariant Neural Networks},
+  booktitle = {Proceedings of the 36th International Conference on Machine Learning},
+  pages     = {3744--3753},
+  year      = {2019}
+}
+
+@inproceedings{grauman2022ego4d,
+  author    = {Kristen Grauman and Andrew Westbury and Eugene Byrne and others},
+  title     = {{Ego4D}: Around the World in 3,000 Hours of Egocentric Video},
+  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
+  pages     = {18995--19012},
+  year      = {2022}
+}