icassp2027-experiment/reports/omni_turn_research_direction_2026-08-26.md

Frozen Omni turn anticipation: evidence, trends, and next direction

Date: 2026-08-26

This memo combines the current EasyCom experiments, a read-only audit of the local EasyCom/Ego4D/EgoSpeak copies under /Volumes/MX500B/Code, and recent work on streaming multimodal interaction. Nothing on the NTFS volume was modified.

Executive decision

The most defensible ICASSP direction is not to continue scaling the Qwen2.5-Omni backbone or to fine-tune more of it. The evidence instead supports:

A frozen causal Omni backbone with a fine-grained, transition-aware multimodal decision readout for egocentric future-speaker anticipation.

The current 4-epoch last-four-layer concat-MLP reaches 60.87144 mean mAP, slightly above EgoSpeak's reported EasyCom GRU A+V result (60.6), while using the Omni Transformer itself for temporal modeling and no additional RNN. It is not yet a clean paper result: it is single-seed and the epoch budget was chosen after looking at Sessions 1--3. Its main scientific value is the diagnosis:

1. useful turn information is distributed across the upper Omni layers;
2. a final-layer query alone under-exposes that information;
3. unconstrained high-capacity readout training overfits the nine training conversations very quickly;
4. larger 7B features and Thinker LoRA do not solve the problem.

The next contribution should therefore target how to expose and supervise the pretrained representation, not how to make the backbone larger.

Current result ladder

Selected EasyCom full-timeline results on Sessions 1--3:

System	0.2s mAP	Mean mAP	Interpretation
EgoSpeak Transformer A+V	73.8	58.7	best reported EgoSpeak 0.2s result
EgoSpeak GRU A+V	73.5	60.6	best reported EgoSpeak mean result
Frozen Omni-3B, final-layer query	71.99768	58.51937	initial 8s reference
Frozen Omni-3B, last-4 scalar mixture	73.45085	60.53423	recovers the short-horizon deficit
Frozen Omni-3B, last-4 concat-MLP, 4 epochs	73.42020	60.87144	exploratory project best
Same concat-MLP, 12 epochs	72.44507	57.98809	clear overfitting
Same concat-MLP, 50 epochs	66.54645	53.03515	severe overfitting
Frozen Omni-7B, reported run	69.80133	57.67367	larger backbone is worse

The concat-MLP result is archived in results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/.

Compared with the scalar mixture, concat-MLP gains only +0.33721 mean mAP and loses 0.03065 at 0.2 seconds. The added capacity mainly helps around 1.2--1.6 seconds. This is useful but too small to justify a 16.8M-parameter fusion module without a cleaner validation protocol and stronger regularization.

What the local data actually imply

EasyCom: many frames, few independent conversations

The EgoSpeak EasyCom targets contain:

Split	Sessions	5Hz steps	Duration	Background	Wearer	Others
Train (4--12)	9	72,405	4.0225h	22.57%	24.93%	60.50%
Test (1--3)	3	23,048	1.2804h	22.91%	26.74%	58.71%
Total	12	95,453	5.3029h	--	--	--

Wearer and other-speaker labels overlap in about 8.08% of all steps, so this is intrinsically a multi-label conversational-state problem even though the EgoSpeak implementation applies a three-way softmax objective.

This exposes an immediate objective mismatch. The observed target rows contain exactly four conversational states:

1. [1, 0, 0]: silence/background;
2. [0, 1, 0]: wearer only;
3. [0, 0, 1]: other people only;
4. [0, 1, 1]: wearer and others overlap.

A three-way softmax cannot assign high probability to both active-speaker classes. In the overlap state, the LSTR loss used by the current project sums -log p_wearer - log p_others, forcing the two positive classes to compete. This is especially relevant because wearer AP is the class that collapses most under overfitting.

A structured four-state head is a cleaner formulation: predict a softmax over silence / wearer-only / others-only / overlap, then expose evaluator scores as p_background=p_silence, p_wearer=p_wearer-only+p_overlap, and p_others=p_others-only+p_overlap. This retains a normalized state model while matching the multi-positive EgoSpeak labels exactly.

The important limitation is the effective sample size. Adjacent 200ms anchors are strongly correlated, and there are only nine independent training sessions. Session-level variation is large:

• wearer prevalence ranges from 10.37% to 37.70%;
• other-speaker prevalence ranges from 47.71% to 77.17%;
• overlap ranges from 1.42% to 20.35%;
• label-change frequency ranges from about 26 to 35 events per minute.

This explains why a 16.8M concat projection can reduce training loss from 0.806973 to 0.300277 while test mean mAP falls by 7.83629 points. The head can learn conversation/session priors rather than a transferable switching rule.

The data are nevertheless rich in transitions. Depending on the session, roughly 54%--69% of anchors have at least one label change in their next two seconds. The next method should not simply oversample all transition-near frames; it should explicitly model persistence versus switching.

Existing annotations support stronger auxiliary supervision

The raw EasyCom copy contains 323 transcript JSON files with:

• 6,981 utterances and about 52,171 transcribed words;
• speaker ID and frame-accurate start/end timestamps;
• Target-of-Speech labels on 6,963 utterances;
• 6DoF tracked participant pose at the original 20 FPS;
• participant-linked head and face boxes.

These annotations permit auxiliary targets without new manual labeling:

• current conversational state;
• keep/switch state between adjacent 200ms steps;
• time-to-next-change bins;
• utterance completion/onset proximity;
• addressee type (group versus individual);
• head-orientation change and target-of-speech prediction.

Transcript text should be used as training-only privileged information or a distillation target if the final online model must consume raw audio/video only. That would demonstrate a semantic advantage without giving Omni unavailable future text at test time.

Ego4D is the more convincing generalization test

The split used by EgoSpeak's reported experiment contains 346 train and 87 test clips, almost all 5 minutes long:

Split	Clips	5Hz steps	Duration	Background	Wearer	Others	Overlap
Train	346	518,881	28.83h	43.31%	23.24%	40.53%	5.58%
Test	87	130,472	7.25h	42.41%	26.30%	38.81%	5.63%

This provides far more independent clips and substantially less split shift than EasyCom at the session level. A method that improves only EasyCom may be learning its fixed around-table conversational geometry. A method that also improves Ego4D supports the stronger claim that the readout exposes general Omni interaction cues.

Research trend and what transfers to this project

1. Explicit decision heads are replacing implicit generation triggers

ROMA processes aligned streaming audio-video units, preserves 40ms audio temporal position IDs, and places a lightweight speak head parallel to the LM head. It forms the head input as a learned mixture of the last four layers. This strongly validates the project's last-four-layer query experiment, while also suggesting that the low-capacity scalar mixture is a more natural primary model than concat-MLP.

Proact-VL inserts a dedicated FLAG token, reads it with a small gated MLP, and separates the decision of when to respond from response generation. JoyAI-VL-Interaction similarly makes silence an explicit learned action rather than an external heuristic. The shared trend is a small, explicit decision interface on top of a general multimodal backbone.

2. Transition supervision is becoming central

Proact-VL argues that the important imbalance is not merely class frequency, but state transition versus state persistence. It upweights switch steps and adds two stability terms: local probability consistency during persistent states and global speaking-rate calibration. This maps directly onto the EasyCom structure and is the highest-priority idea for improving 0.2-second prediction.

The BadCat unit-based agent reaches a similar conclusion from a systems perspective: it represents dialogue control as keep-listen, listen-to-speak, keep-speak, and speak-to-listen actions. That is a strong conceptual justification for adding a keep/switch auxiliary head rather than another generic hidden layer.

3. Frozen backbones plus task-specific fusion are credible

MM-When2Speak formulates timing as dense response-type classification and uses adaptive attention over visual, audio, and textual tokens. Its response fusion stage freezes the LLM and modality encoders and optimizes a lightweight attention module. This is closely aligned with our empirical finding that frozen Omni is more stable than Thinker LoRA.

The important next ablation is therefore not another larger query MLP. It is a comparison between:

1. final query state;
2. learned last-four query mixture;
3. recent 200ms audio-token pooling;
4. recent 200ms audio-video token attention;
5. query plus recent-token pool.

For token pooling, only payload tokens in the causal interval should be used, selected by TMRoPE time IDs and excluding padding/template tokens.

4. Fine temporal cadence is a differentiator

Most proactive video systems make decisions at one-second cadence. EgoSpeak and this project operate at 200ms. That is not merely an implementation detail: it creates a credible ICASSP contribution around fine-grained conversational switch anticipation, especially because Omni audio tokens retain finer 40ms temporal positions.

The paper should emphasize that a two-second multimodal ingestion unit does not force a two-second decision rate. Causal readout at each 200ms anchor can still exploit fine-grained audio timing, provided the state/readout does not collapse the current-tail tokens.

Recommended paper contribution

Working framing:

Fine-Grained Causal Readout of Frozen Omni Models for Egocentric Turn Anticipation

Possible three-part contribution:

1. Streaming formulation. Raw egocentric audio/video is encoded by a frozen Omni Transformer with bounded causal memory, producing decisions at 5Hz without an external RNN.
2. Hierarchical temporal readout. A parameter-efficient upper-layer query mixture is fused with recent TMRoPE-localized audio/video payload tokens, exposing both semantic context and immediate prosodic/nonverbal cues.
3. Structured transition-aware decoding. A four-state overlap-aware output is marginalized to EgoSpeak's three AP classes, while existing frame labels generate current, keep/switch, and time-to-change supervision with persistence consistency and session-aware validation.

The negative results become useful evidence rather than discarded runs:

• 7B is worse than 3B;
• Thinker LoRA causes representation drift and worse test mAP;
• concat-MLP helps briefly but collapses with longer training;
• a tiny last-four mixture recovers almost all of the useful gain.

Together they support the thesis that the bottleneck is task-aligned causal readout and supervision, not foundation-model capacity.

Prioritized experiment plan

P0: make the current gain publishable

1. Train on Sessions 4--11 and use Session 12 only as development data.
2. Use one fixed 12-epoch schedule and save every epoch. Do not compare runs whose cosine schedule changes with the requested epoch count.
3. Select epoch and threshold using a predeclared development metric.
4. Retrain on Sessions 4--12 for the selected epoch count, then evaluate Sessions 1--3 once.
5. Run at least three seeds; EgoSpeak reports five-seed summary statistics.
6. Match single-query, fixed mean, scalar mixture, and concat-MLP schedules.

Primary selection metric should be 0.5 * mAP@0.2s + 0.5 * mean_mAP, or both metrics should be reported as a Pareto pair. Selecting only mean mAP can hide the exact responsiveness regression that matters most to this project.

Before changing the representation, run an identical-output-objective ablation on the frozen final-layer and scalar-mixture features:

Output formulation	Training loss	Evaluation scores
Official control	current three-logit LSTR MCE	three-way softmax
Independent multi-label	BCE on wearer/others	sigmoid scores; background derived from neither
Structured state	four-way CE on silence/wearer/others/overlap	three marginals

The structured state variant is preferred because it represents overlap without abandoning a coherent normalized conversational state. If it improves the scalar-mixture model, carry it into every later experiment.

P1: low-capacity temporal readout

Run these in order:

Readout	Approx. new fusion parameters	Purpose
Last-4 scalar mixture	4	robust reference
Feature-wise last-4 gate	8,192	per-channel layer choice
Low-rank concat, rank 128	about 1.31M	cross-layer interaction
Low-rank concat, rank 256	about 2.62M	capacity ablation
Current concat-MLP	about 16.8M	upper-capacity control

Initialize feature-wise/low-rank fusion as a residual around the scalar mean, so the first optimizer step preserves the strong pretrained readout. Use weight decay, dropout, and dev early stopping, but do not make generic regularization the main novelty.

P1: transition-aware supervision

Convert each target vector to the corresponding four-state variable q_t, then derive:

• current_t = q_t;
• switch_{t,h} = 1[q_{t+h} != q_t] for short horizons;
• time_to_change_t capped at 2 seconds;
• persistence_t = 1 - switch_{t,1}.

Train the existing ten-horizon head jointly with current/switch/time-to-change heads. Apply larger weight only to true switch steps, plus a consistency loss when the ground-truth state persists. This should be evaluated first on the camera-wearer AP at 0.2 seconds, where representation drift and head overfitting have repeatedly caused the largest damage.

P2: expose recent token evidence

Add a readout query that attends only to payload tokens from [t-0.2s, t], identified through TMRoPE positions. Compare audio-only recent tokens, AV recent tokens, and query-plus-token fusion. Keep the backbone frozen.

This is the most distinctive method contribution, but it should follow P0/P1 because token masking and timestamp correctness are easier to get wrong than layer fusion.

P2: use existing semantics without test leakage

Use EasyCom transcription and Target-of-Speech only during training:

• distill utterance-completion likelihood into the AV state;
• predict group-versus-individual addressee;
• predict a coarse semantic continuation state such as incomplete/complete;
• drop the privileged text branch at inference.

This makes the claimed "Omni semantic advantage" testable. Merely replacing wav2vec/ResNet with an Omni hidden state does not demonstrate semantic use.

P3: generalization and ablations

• EasyCom audio-only, video-only, and AV under identical readout capacity;
• Ego4D 346/87 split with the same method and no EasyCom-specific tuning;
• 4s, 8s, and 20s memory with fixed 5Hz output;
• 2 FPS versus 5 FPS video ablation;
• runtime, memory, and causal verification;
• per-phase and transition-conditioned AP in addition to EgoSpeak Table 2.

Stop conditions

Do not spend another full training cycle on:

• full Thinker LoRA on EasyCom alone;
• Qwen2.5-Omni-7B without a readout change;
• 25--50 epoch high-capacity heads without development early stopping;
• more overlapping-window extraction as the main architecture;
• prompt wording as the primary method.

Those directions are weakly supported by the current evidence and make the paper harder to interpret. The clean next experiment is the matched P0 readout ablation followed by transition-aware supervision.