icassp2027-experiment/README.md

# Omni-only Turn-Taking Experiment

This project tests whether **Qwen2.5-Omni-3B can replace both the handcrafted
audio/visual feature extractors and the downstream RNN/Transformer used by
EgoSpeak** for egocentric turn-taking anticipation.

The persistent paper index and Chinese summaries are maintained in
[literature_notes](literature_notes/README.md).

The pooled local-feature B1 experiment (frozen Qwen face crops + recent audio,
native 2048D sidecars, optional trainable projection, four-epoch ablations) is
documented in [docs/local_feature_baseline.md](docs/local_feature_baseline.md).

The main experiment is deliberately simple:

```text
causal audio + video up to time t
                |
                v
     Qwen2.5-Omni-3B Thinker
                |
        fused state q_t [2048]
                |
                v
       trainable prediction head
                |
                v
  future logits [10 horizons, 3 classes]
```

There is **no RNN after Qwen**. The Omni Thinker is the temporal model. The
only supervised component in the first experiment is a small `10 x 3`
classification head.

## Current status

| File | Status | Purpose |
|---|---|---|
| `smoke_test.py` | Verified on one NVIDIA L20 | Text and 12-second EasyCom audio-video inference |
| `easycom_manifest.py` | Verified on EasyCom | Discover one-minute clips, preserve session timecodes, and check 5 Hz targets |
| `qwen_omni_features.py` | Implemented, not executed here | Load only the 3B Thinker and extract fused states from one AV window |
| `streaming_turn_model.py` | Verified on EasyCom | Produce one causal Omni state every 200 ms, with causal-prefix and open-stream modes |
| `TurnPredictionHead` | Trained and evaluated | Map `[2048]` to `[10, 3]` logits |
| `train_turn_head.py` | Verified on EasyCom | Train the frozen-Omni anticipation head with the official EasyCom split |
| `infer_turn_head.py` | Verified on EasyCom | Save EgoSpeak-format predictions and reproduce per-horizon AP/mAP |
| `eval_omni_turn_lora.py` | Implemented, remote validation pending | Evaluate Thinker LoRA from dense AV caches with the same AP/mAP protocol |

The full Thinker input path, embedded audio, direct video decoding,
FlashAttention-2, 5 Hz streaming extraction, cache replay, target alignment,
head training, and EgoSpeak-compatible evaluation have now been exercised on
EasyCom. The first Ego4D 8-second versus 20-second context ablation has also
been recorded. Multi-seed evaluation remains in progress.

## Current EasyCom results

The current single-run comparison was recorded on 2026-08-19. Both modes use
the frozen Qwen2.5-Omni-3B Thinker, 2 FPS video, 5 Hz predictions, a 20-second
causal context budget, the same `10 x 3` head architecture, and EgoSpeak's
official EasyCom train/test split.

| Representation | EgoSpeak full-timeline mean mAP | Strict causal mean mAP |
|---|---:|---:|
| Open-stream replay | 58.316 | 58.326 |
| Causal-prefix | 58.830 | 58.841 |

Causal-prefix is 0.514 mAP points higher in the exact Table 2 protocol, but
the difference is concentrated at 0.2--0.6 seconds. Open-stream is slightly
higher at 1.6--2.0 seconds. These are not yet final paper values because the
head-training variance has not been measured across multiple seeds.

The full per-horizon and per-class snapshot, machine-readable CSV, provenance,
and follow-up checks are in
[`results/easycom_open_stream_vs_causal_prefix_2026-08-19/`](results/easycom_open_stream_vs_causal_prefix_2026-08-19/README.md).

### EasyCom 8-second versus 20-second context

An additional single-run open-stream comparison recorded on 2026-08-23 gives
**58.51937** full-timeline mean mAP with an 8-second context budget versus
**58.316** with 20 seconds, a difference of **+0.20337** points. The aggregate
difference is small and hides a clear horizon tradeoff: 8 seconds is stronger
at 0.2--0.8 seconds, whereas 20 seconds is stronger from 1.0 through 2.0
seconds. These runs must be repeated with identical additional seeds before
choosing a context budget for the paper.

The raw 8-second console output, machine-readable comparison, and detailed
interpretation are archived in
[`results/easycom_open_stream_8s_vs_20s_2026-08-23/`](results/easycom_open_stream_8s_vs_20s_2026-08-23/README.md).

### Last-4 query-layer learned mixture

A frozen Qwen2.5-Omni-3B experiment now reads the query states from the final
four Thinker layers and learns four global softmax mixture weights before the
same `10 x 3` anticipation head. With an 8-second open-stream context, 2 FPS
video, 5 Hz outputs, and 25 head-training epochs, it reaches **73.45085** mAP
at 0.2 seconds and **60.53423** mean mAP on official test Sessions 3, 2, and 1.
Relative to the recorded single-final-layer 8-second run, this is **+1.45317**
at 0.2 seconds and **+2.01486** mean mAP, with positive gains at every horizon.

The final oldest-to-newest layer weights are
`[0.246384, 0.248327, 0.241774, 0.263515]`. Because they remain almost uniform,
the result currently demonstrates the value of multi-layer query readout, not
yet the value of learning the mixture. A fixed uniform last-4 mean is the next
required control. The complete table, machine-readable metrics, provenance,
and naming caveat are archived in
[`results/easycom_frozen_3b_8s_last4_learned_mix_2026-08-26/`](results/easycom_frozen_3b_8s_last4_learned_mix_2026-08-26/README.md).

#### Higher-capacity last-4 concat-MLP ablation

`train_turn_head.py` also supports a higher-capacity fusion control. It
flattens each `[4, 2048]` last-layer stack to `8192` dimensions, applies
`LayerNorm -> Linear(8192, 2048) -> GELU`, and then uses the unchanged
anticipation MLP. This adds about 16.8 million fusion parameters, compared
with four logits in the scalar-mixture experiment. Keep the seed and training
recipe fixed for a clean capacity ablation:

```bash
CUDA_VISIBLE_DEVICES=0 \
OMP_NUM_THREADS=4 \
MKL_NUM_THREADS=4 \
OPENBLAS_NUM_THREADS=4 \
python train_turn_head.py \
  --feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack \
  --target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
  --output checkpoints/easycom_frozen_3b_8s_last4_concat_mlp_25ep.pth \
  --sessions 4 5 6 7 8 9 10 11 12 \
  --epochs 25 \
  --batch-size 512 \
  --learning-rate 7e-5 \
  --weight-decay 5e-5 \
  --warmup-epochs 10 \
  --warmup-factor 0.3 \
  --dropout 0.2 \
  --loss lstr-mce \
  --layer-fusion concat-mlp \
  --expected-mode open-stream-replay \
  --seed 0 \
  --device cuda:0
```

Evaluate with the same dense stacked test features:

```bash
CUDA_VISIBLE_DEVICES=0 python infer_turn_head.py \
  --feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack \
  --target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
  --checkpoint checkpoints/easycom_frozen_3b_8s_last4_concat_mlp_25ep.pth \
  --output-dir results/easycom_frozen_3b_8s_last4_concat_mlp_25ep \
  --expected-mode open-stream-replay \
  --batch-size 1024 \
  --device cuda:0
```

Treat this as a capacity ablation. If it improves, compare it against a
parameter-matched single-layer MLP and use stronger regularization or a clean
development session before attributing the gain to cross-layer fusion.

#### Four-epoch loss-structure ablation

Using the same frozen 8-second last-4 stack and concat-MLP readout, a
three-seed fixed-budget comparison tested LSTR-MCE, independent sigmoid/BCE,
and four-state softmax. All nine training sessions were used for exactly four
epochs with no development set. On the causal-valid test metric, no objective
is a decisive winner: BCE has the highest mean mAP (`60.91733`), while
four-state CE has the highest 0.2-second wearer AP (`78.72040`). Relative to
LSTR-MCE, both alternatives slightly improve the two speech classes while
reducing background AP. The full protocol, per-seed summary, seed-0 horizon
curves, and machine-readable data are archived in
[`results/easycom_loss_structure_4ep_2026-08-26/`](results/easycom_loss_structure_4ep_2026-08-26/README.md).

#### R1/R2 transition-aware head ablation

The transition target for current state t and future horizon h is one exactly
when the complete three-label activity row changes between t+h-1 and t+h.
This includes onset, offset, speaker handover, and entering or leaving overlap.
It never uses media or labels after the prediction target t+h.

Both variants share the activity MLP trunk and add a small binary 10-horizon
transition readout:

- R1 supervised uses
  L = L_activity + 0.2 L_transition,
  where L_transition is BCEWithLogits with positive weight 3.
- R2 expected reward uses
  L = L_activity + 0.2 L_transition - 0.1 E[R_transition].
  For transition probability p and target z, correctness reward is
  z p + (1-z)(1-p). Its expectation is computed exactly, weighted with the
  same positive weight 3, and normalized by total weight. There is no sampled
  policy gradient, so this is the low-variance control rather than a claim of
  interactive reinforcement learning.

Run the fixed four-epoch, no-development-set comparison with the same schedule
as the loss-structure ablation:

~~~bash
FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame

for seed in 0 1 2
do
  CUDA_VISIBLE_DEVICES=0 \
  OMP_NUM_THREADS=4 \
  MKL_NUM_THREADS=4 \
  OPENBLAS_NUM_THREADS=4 \
  python train_turn_head.py \
    --feature-root "$FEATURE_ROOT" \
    --target-root "$TARGET_ROOT" \
    --output "checkpoints/easycom-transition-r1-4ep-seed-$seed.pth" \
    --sessions 4 5 6 7 8 9 10 11 12 \
    --epochs 4 \
    --batch-size 512 \
    --learning-rate 7e-5 \
    --weight-decay 5e-5 \
    --warmup-epochs 1.6 \
    --warmup-factor 0.3 \
    --dropout 0.2 \
    --loss lstr-mce \
    --transition-objective r1-supervised \
    --transition-loss-weight 0.2 \
    --transition-pos-weight 3 \
    --layer-fusion concat-mlp \
    --expected-mode open-stream-replay \
    --seed "$seed" \
    --device cuda:0

  CUDA_VISIBLE_DEVICES=0 \
  OMP_NUM_THREADS=4 \
  MKL_NUM_THREADS=4 \
  OPENBLAS_NUM_THREADS=4 \
  python train_turn_head.py \
    --feature-root "$FEATURE_ROOT" \
    --target-root "$TARGET_ROOT" \
    --output "checkpoints/easycom-transition-r2-4ep-seed-$seed.pth" \
    --sessions 4 5 6 7 8 9 10 11 12 \
    --epochs 4 \
    --batch-size 512 \
    --learning-rate 7e-5 \
    --weight-decay 5e-5 \
    --warmup-epochs 1.6 \
    --warmup-factor 0.3 \
    --dropout 0.2 \
    --loss lstr-mce \
    --transition-objective r2-expected-reward \
    --transition-loss-weight 0.2 \
    --transition-pos-weight 3 \
    --transition-reward-weight 0.1 \
    --layer-fusion concat-mlp \
    --expected-mode open-stream-replay \
    --seed "$seed" \
    --device cuda:0
done
~~~

Evaluate all six checkpoints. In addition to the unchanged EgoSpeak activity
metrics, inference writes transition_pred arrays and prints transition AP,
precision, recall, and F1 at threshold 0.5:

~~~bash
for variant in r1 r2
do
  for seed in 0 1 2
  do
    CUDA_VISIBLE_DEVICES=0 python infer_turn_head.py \
      --feature-root "$FEATURE_ROOT" \
      --target-root "$TARGET_ROOT" \
      --checkpoint "checkpoints/easycom-transition-$variant-4ep-seed-$seed.pth" \
      --output-dir "results/easycom-transition-$variant-4ep-seed-$seed" \
      --expected-mode open-stream-replay \
      --batch-size 1024 \
      --device cuda:0
  done
done
~~~

Compare R1 and R2 against the existing three-seed LSTR-MCE baseline under the
same four-epoch budget. The primary paper outcome remains activity mAP; the
transition metrics diagnose whether an activity improvement is genuinely
associated with better boundary prediction.

The preliminary seed-0/1 results are archived in
[results/easycom_transition_auxiliary_4ep_2026-08-27/](results/easycom_transition_auxiliary_4ep_2026-08-27/README.md).
Neither R1 nor R2 establishes a meaningful activity-mAP gain. At 0.2 seconds,
transition prevalence is 11.21718%, while transition AP is approximately
15.3%, indicating weak but non-random ranking. R2 improves activity mean mAP
over R1 by only 0.01162 points and transition AP by 0.16660 points. Seed 2 is
still required before freezing this as a complete null ablation.

## Ego4D local-feature ablation family

To test whether the recent-audio / per-person-visual gains transfer from EasyCom,
see [the Ego4D local-feature experiment guide](docs/ego4d_local_baselines.md).
It reuses existing global Omni features and the EgoSpeak manifest, extracts local
A/V directly from raw clips, and runs **G / G+A / G+V / G+A+V** for four epochs,
without a dev set. `run_ego4d_local_baselines.py` preflights all train/test files
and encoder contracts before training; the summary reports matched-seed deltas,
including AV minus audio. No new accuracy claim is implied by adding the code.

The global `train/<uid>.pt` and `test/<uid>.pt` layout is now read directly by
extraction, training and inference; no symlink index is needed.
`extract_ego4d_local_multigpu.py` distributes full clips across eight independent
GPU workers, with per-GPU logs, validated resume, and foreground supervision.
See the guide's L20 commands for the `/data1/yfzx/wangx` paths.

For **encoder-only A / V / AV**, without Thinker states, see
[the encoder-only comparison guide](docs/ego4d_encoder_only.md).
`prepare_encoder_only_features.py` packs complete local vectors and validity
bits on CPU; `run_ego4d_encoder_only.py` trains/evaluates four-epoch heads with
no dev set. This native-concatenation baseline is not capacity- or
context-matched to the Thinker baseline.

## Current Ego4D context-length results

The following single-seed snapshot was recorded on 2026-08-22. Both variants
use open-stream replay, 2 FPS Decord video, 5 Hz predictions, the same frozen
Qwen2.5-Omni-3B Thinker and independently trained `10 x 3` heads. They differ
only in the rolling causal-memory budget: 8 seconds versus 20 seconds. Both
runs use the same EgoSpeak Ego4D train/test manifest and test timeline.

| Context | EgoSpeak full-timeline mean mAP | Strict causal mean mAP |
|---|---:|---:|
| 8 seconds | **65.5943** | **65.6920** |
| 20 seconds | 63.1408 | 63.2298 |
| 20s minus 8s | -2.4535 | -2.4622 |

The strict-causal per-horizon comparison is:

| Horizon | 8s mAP | 20s mAP | 20s minus 8s |
|---:|---:|---:|---:|
| 0.2s | **74.2213** | 71.7403 | -2.4810 |
| 0.4s | **71.0180** | 68.5945 | -2.4234 |
| 0.6s | **68.3746** | 65.9287 | -2.4459 |
| 0.8s | **66.3190** | 63.8606 | -2.4584 |
| 1.0s | **64.7470** | 62.3329 | -2.4140 |
| 1.2s | **63.6416** | 61.2290 | -2.4126 |
| 1.4s | **62.8697** | 60.4035 | -2.4662 |
| 1.6s | **62.3142** | 59.7870 | -2.5272 |
| 1.8s | **61.8705** | 59.3678 | -2.5027 |
| 2.0s | **61.5444** | 59.0541 | -2.4903 |
| Mean | **65.6920** | 63.2298 | -2.4622 |

For the exact full-timeline protocol, averaging class AP over the ten
horizons gives:

| Class | 8s mean AP | 20s mean AP | 20s minus 8s |
|---|---:|---:|---:|
| Background | **69.5063** | 67.2409 | -2.2653 |
| Camera wearer speaking | **59.4005** | 56.5784 | -2.8222 |
| Other people speaking | **67.8760** | 65.6030 | -2.2730 |

The 8-second representation is better at every horizon and for every class.
The approximately 2.4--2.5 point gap is nearly constant across anticipation
horizons, while the largest class degradation is for camera-wearer speech.
This pattern does not support a long-horizon benefit from the additional 12
seconds of context. A plausible hypothesis is that excessive history dilutes
locally predictive turn-taking cues in the final frozen Thinker state, but the
single-seed comparison cannot establish that mechanism. Both context settings
must be repeated with the same additional seeds before treating the difference
as a final paper result.

## Why Omni-only instead of Omni + RNN

EgoSpeak separately extracts visual and audio features and then uses GRU,
LSTR, or Mamba for temporal modeling:

```text
ResNet50 video feature + wav2vec2 audio feature -> temporal model -> prediction
```

Qwen2.5-Omni already contains:

- audio and visual encoders;
- joint audio-video temporal positions;
- a causal decoder-only Thinker with KV cache;
- pretrained acoustic, visual, and semantic knowledge.

Adding a GRU after a continuously processed Omni sequence would model history
twice and would make it harder to attribute an improvement to Omni. The primary
comparison is therefore:

```text
EgoSpeak: ResNet50 + wav2vec2 + GRU
Ours:     Qwen2.5-Omni Thinker + small prediction head
```

An Omni-feature-plus-GRU model can be kept as an ablation, but it is not the
main architecture.

## EgoSpeak contract that this project reproduces

The inspected reference repository is `/Volumes/760P/Code/EgoSpeak`.
It contains no Qwen or Omni implementation and no raw-media feature extraction
scripts. It distributes precomputed arrays instead:

```text
DATASET/
|- rgb_kinetics_resnet50/   # [T, 2048]
|- audio_wav2vec2/          # [T, 5120]
`- target_perframe/         # [T, 3]
```

The useful part for this project is the exact task interface.

### Time and classes

Both EasyCom and the EgoSpeak `EGO4DSHUFFLE` split use **5 FPS**, so one task
timestep is **0.2 seconds**. The three output columns are:

```text
0: background
1: camera wearer speaking
2: other people speaking
```

They are not strictly one-hot. Background is exclusive, but simultaneous
camera-wearer and other-person speech is stored as `[0, 1, 1]`. The inspected
EasyCom features contain 7,711 such overlap steps. Preserve the full
multi-positive target rather than applying `argmax`.

Relevant EgoSpeak files:

- `LSTR/data/data_info.json`: FPS, classes, and session splits;
- `MiniROAD/configs/EasyCom/miniroad_aa_easycom.yaml`: 128-step training
  windows, stride 4, three classes, and anticipation length 10;
- `MiniROAD/datasets/dataset.py`: feature loading and future-label indexing;
- `MiniROAD/criterions/loss.py`: anticipation loss;
- `MiniROAD/trainer/eval.py`: per-horizon AP and mean anticipation mAP.

### Future target alignment

For current task index `i`, the prediction target is:

```python
target[i + 1 : i + 11]
```

Thus the ten output rows represent:

```text
t+0.2, t+0.4, t+0.6, ..., t+2.0 seconds
```

If `states[i]` is verified to correspond to EgoSpeak task index `i`, the
offline head-training tensors should be constructed as follows:

```python
target = target_perframe                            # [T, 3], multi-positive
usable = min(len(states), len(target)) - 10

x = states[:usable]                                 # [N, 2048]
y = stack(
    [target[h : h + usable] for h in range(1, 11)],
    axis=1,
)                                                   # [N, 10, 3]
```

Do not silently assume that the first saved Omni state matches label index
zero. EgoSpeak does not publish the script that generated its original
ResNet/wav2vec2 features, so the exact zero-time convention is not documented.
Before training, inspect a short clip with known speech boundaries and test the
candidate mappings `states[i] -> target[i]` and `states[i] -> target[i+1]`.
Store the selected offset in every feature file/checkpoint.

### Splits

Use the published split, not a random frame split.

EasyCom:

- train, preserving EgoSpeak's list: `Session_7`, `Session_11`, `Session_10`,
  `Session_6`, `Session_9`, `Session_4`, `Session_8`, `Session_5`, `Session_12`;
- test, preserving EgoSpeak's list: `Session_3`, `Session_2`, `Session_1`.

This is the **EgoSpeak comparison split**. The EasyCom dataset repository
itself recommends a different train/validation/test split; do not substitute
that split when reproducing EgoSpeak Table 2.

Ego4D Audio-Visual Diarization:

- use the `EGO4DSHUFFLE` key;
- 346 training clips;
- 87 test clips;
- each clip is approximately five minutes.

Never mix timesteps from the same session across training and validation.

## EasyCom raw-media layout and manifest

The extracted release used here has this relevant structure:

```text
extracted_easycom/
`- Main/
   |- Video_Compressed/
   |  |- Session_1/
   |  |  |- 00-00-000.mp4
   |  |  |- 01-00-000.mp4
   |  |  `- ...
   |  `- Session_2 ... Session_12/
   |- Glasses_Microphone_Array_Audio/
   |- Close_Microphone_Audio/
   |- Speech_Transcriptions/
   `- Metadata/
```

`Video_Compressed` is the AR-glasses wearer's egocentric stream with embedded
stereo AAC audio. Each session is divided into approximately one-minute clips;
the `mm-ss-SSS` filename is the clip's start time on the original recording
timeline. Some intervals are absent because EasyCom redacted one-minute
segments. These raw time gaps are expected.

EgoSpeak does not preserve those empty intervals. Its appendix states that it
crops every EasyCom clip to retain only the 1920-pixel-wide video region and
then merges the retained clips into one dense video per session. Therefore the
EgoSpeak-compatible label time for a clip is its cumulative position in the
concatenation, not the `mm-ss-SSS` source time. The manifest stores both.

Build a raw-video manifest and, when the EgoSpeak labels are available, check
their duration at 5 Hz:

```bash
python easycom_manifest.py \
  --dataset-root /data1/yfzx/wangx/Datasets/easycom/extracted_easycom \
  --target-root /path/to/EasyCom/target_perframe \
  --output manifests/easycom.json
```

Omit `--target-root` for the first media-only inventory. The manifest reports
raw-source discontinuities as informational and computes each clip's dense
EgoSpeak concatenation offset and corresponding 5 Hz target index. For the
audio-only ablation, use the same embedded glasses audio as A+V (extract it to
mono WAV if needed); do not switch to close microphones, because that changes
the sensor and makes the modality ablation unfair.

Audit downloaded EgoSpeak arrays, and optionally compare every session length
against the raw manifest:

```bash
python audit_egospeak_features.py \
  --feature-root /path/to/EgoSpeak/dataset/EasyCom \
  --raw-manifest manifests/easycom.json \
  --output reports/egospeak_easycom_audit.json
```

The inspected release contains 95,453 synchronized steps across all 12
sessions: 5.302944 hours at 5 Hz. Every session has identical RGB, flow, audio,
and target length. Shapes are `[T,2048]`, `[T,2048]`, `[T,5120]`, and `[T,3]`.

## Recommended comparison matrix

| Experiment | Visual sampling | Audio | Output rate |
|---|---:|---|---:|
| EgoSpeak official A+V | 5 Hz ResNet features | 5 Hz wav2vec2 features | 5 Hz |
| Omni audio-only | none | raw glasses-wearer waveform | 5 Hz |
| Omni A+V primary | 2 FPS | raw embedded waveform | 5 Hz |
| Omni A+V FPS ablation | 5 FPS | raw embedded waveform | 5 Hz |

Two FPS is the primary Omni setting because it matches Qwen's default video
sampling and permits longer context. Five FPS is the visual-rate ablation.
Input feature rates need not match EgoSpeak; the split, causal prediction
times, three classes, ten target horizons, and evaluator must match exactly.

## Prepare an EgoSpeak-compatible EasyCom session

Do not pass the raw one-minute files independently to Omni and do not preserve
the gaps implied by their filenames. EgoSpeak concatenates the retained clips
into a dense session timeline and crops the participant-ID banner on the left.

`prepare_easycom_session.py` reproduces this policy and creates videos whose
stored frames already lie on a fixed session-global 2 FPS or 5 FPS grid. This
prevents Qwen's per-window video loader from choosing a new adaptive set of
frames for every overlapping prefix.

Session 1 has been checked against the local release:

```text
24 retained clips
27,768 source frames at 20 FPS
/ 4 frames per task step
= 6,942 steps at 5 Hz
= target_perframe/Session_1.npy length exactly
```

Preview the commands and validate frames against the target without encoding:

```bash
python prepare_easycom_session.py \
  --session-dir /Volumes/MX500B/Code/easycom/Main/Video_Compressed/Session_1 \
  --output-dir prepared_easycom/Session_1 \
  --target /Volumes/MX500B/Code/EgoSpeak/dataset/EasyCom/target_perframe/Session_1.npy \
  --fps 2 5 \
  --dry-run
```

Remove `--dry-run` to run FFmpeg. The outputs are:

```text
Session_1_2fps.mp4     primary Omni A+V input
Session_1_5fps.mp4     visual-rate ablation
Session_1.json         clip order, exact T, crop and sampling provenance
Session_1.ffconcat     reproducible dense concatenation order
```

The target length must equal the number of complete 5 Hz intervals,
`floor(sum(source_frames) / 4)`. If a session ends with one to three residual
20 Hz frames, the target-derived duration discards only that incomplete final
interval; the discarded frame count is recorded in the sidecar.

Each clip's decoded video and audio are trimmed to the exact duration implied
by its video frame count before concatenation. This prevents AAC priming/tail
padding from accumulating at the 24 clip boundaries and causing progressive
audio/video drift. The session-global visual grid is padded defensively at EOF
and then trimmed by frame index, so it contains exactly
`ceil(target_steps / 5 * video_fps)` frames on every FFmpeg version.

On the remote L20 machine, use its own dataset paths. For example:

```bash
python prepare_easycom_session.py \
  --session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_1 \
  --output-dir /data1/yfzx/wangx/Datasets/easycom/omni_prepared/Session_1 \
  --target /path/to/EgoSpeak/dataset/EasyCom/target_perframe/Session_1.npy \
  --fps 2 5
```

This is the command that performs the potentially long FFmpeg processing; it
has not been run by the code authoring workflow.

If encoding completed but a later validation/metadata step failed, update the
script and reuse the existing MP4 without encoding it again:

```bash
python prepare_easycom_session.py ... --fps 2 --reuse-existing
```

## What Qwen's native 2-second streaming design means

Qwen2.5-Omni was designed for **block-wise multimodal prefill**. Its
[technical report](https://arxiv.org/html/2503.20215) states that the audio
and visual encoders are modified for block-wise processing and that audio
self-attention is performed in independent two-second blocks. The released
[3B configuration](https://huggingface.co/Qwen/Qwen2.5-Omni-3B/raw/main/config.json)
uses `seconds_per_chunk=2`, `position_id_per_seconds=25`, and audio
`n_window=100`.

Conceptually, a native online input is organized as follows:

```text
open one user/media turn
  encode/append AV [0,2]       -> commit Thinker KV
  encode/append AV [2,4]       -> commit Thinker KV
  encode/append AV [4,6]       -> commit Thinker KV
  ...
close the media turn           -> generate text and/or speech
```

Within each temporal group, video tokens precede the aligned audio tokens.
TMRoPE assigns 25 temporal positions per second, or 50 positions per native
two-second group, so audio and video share a physical-time coordinate system.
Completed blocks are appended to one causal Thinker sequence; the long-range
history is modeled by the Thinker rather than by cross-block full attention in
the modality encoders.

This architecture is block-causal, not frame-causal. Attention inside an
audio block is non-causal. It is therefore invalid to encode the complete
`[0,2]` block and claim that its representation at 0.2 seconds saw no future:

```text
encode complete [0,2] AV -> read state at 0.2 s  # future leakage
```

For this project's 5 Hz targets, the current incomplete block must instead be
encoded only through the prediction time. These 0.2-second partial-block
readouts are an extension made for EgoSpeak; the Qwen paper does not define a
canonical hidden state every 200 ms.

The public Transformers and local Qwen web-demo APIs normally receive a
completed media object and call the processor/model once. They retain the
model's block-wise masks and token organization, but do not expose the full
persistent live-session implementation used by the hosted real-time product.

## Native Qwen streaming versus `causal-prefix`

The two extraction modes are both causal with respect to raw observations,
but they define different model inputs. `causal-prefix` is a sliding-window
reference invented for this experiment; it is not Qwen's native streaming
execution model.

| Property | `causal-prefix` | Native Qwen streaming idea |
|---|---|---|
| Session | New inference at every 0.2 s | One open media turn |
| Media organization | One complete `[max(0,t-M),t]` item | Fixed source-aligned 2 s blocks |
| Modality encoding | Re-encode the complete window | Encode/commit each completed block once |
| Thinker KV | Start empty for every state | Append to persistent KV |
| TMRoPE | Recomputed from each window origin | Advances with the stream |
| History | Exact rolling `M` seconds | Accumulates until a serving/context policy intervenes |
| 5 Hz readout | Close/read a new turn each time | Not specified by the original model |

The difference becomes especially important after a sliding window fills. At
`t=20.8` with `M=20`, `causal-prefix` constructs one new media item covering:

```text
[0.8,20.8]
  -> encoder blocks [0.8,2.8], [2.8,4.8], ..., [18.8,20.8]
```

At `t=21.0`, every encoder boundary moves again:

```text
[1.0,21.0]
  -> encoder blocks [1.0,3.0], [3.0,5.0], ..., [19.0,21.0]
```

Native streaming keeps the source-aligned partition instead:

```text
[0,2], [2,4], ..., [18,20], current partial [20,20.8]
```

Because Qwen's audio encoder uses block-local non-causal attention, moving a
block boundary changes the representation even when most raw samples are the
same. Window-relative TMRoPE, convolution/padding boundaries, chat closure,
and BF16/attention execution add further differences. This is why exact KV
replay can be numerically identical to its own from-scratch replay while still
differing from `causal-prefix`; the modes are evaluating different token and
encoder histories.

Use the modes according to the scientific question:

- use `open-stream-replay` for the primary online experiment, training, and
  deployment-aligned evaluation;
- use `causal-prefix` as an expensive continuous-window reference and
  short-clip ablation;
- never train a head on one representation and deploy it on the other without
  retraining, because their feature distributions are not interchangeable.

## Causal 5 Hz extraction modes

`streaming_turn_model.py` implements two modes.

### 1. `causal-prefix`: continuous-window reference

At every 200 ms, independently process the causal context ending at the
current time:

```text
[max(0, t-20 s), t] AV -> Omni -> q_t
```

Properties:

- strict no-future input;
- one continuous media window per prediction;
- no cross-forward cache;
- very expensive because overlapping context is recomputed;
- useful for checking continuous sliding-window semantics and causality;
- not the native Qwen block-streaming execution model.

Example:

```bash
python streaming_turn_model.py \
  --video /path/to/easycom_session.mp4 \
  --use-audio-in-video \
  --mode causal-prefix \
  --start 0 \
  --end 20 \
  --memory-seconds 20 \
  --output features/easycom_20s_prefix.pt
```

### 2. `open-stream-replay`: recommended efficient mode

This mode keeps one user/media turn open across the whole bounded-memory
segment. The first block contributes the chat and AV opening markers once;
later completed two-second blocks contribute only interleaved audio/video
payload tokens:

```text
open AV prefix
  completed [0,2] payload       -> commit KV
  current   [2,2.2] payload     -> temporary branch -> close/read q_t
  current   [2,2.4] payload     -> temporary branch -> close/read q_t
  ...
  completed [2,4] payload       -> commit KV
AV/chat closure                 -> never committed; readout branch only
```

This matches the core Qwen design choices: fixed two-second AV groups, one
open media turn, completed-block commits, and incremental Thinker KV. It makes
two practical changes for this dataset:

1. the current 0--2 second partial block is evaluated speculatively every
   0.2 seconds and discarded unless it reaches a complete block boundary;
2. the KV history is bounded and rebuilt from recent complete blocks rather
   than accumulating for an entire multi-minute session. Complete-block AV
   embeddings are retained in a bounded ring, so rolling rebuilds rerun only
   the Thinker and do not rerun the processor, audio tower, or video tower.

It should therefore be described as **bounded Qwen-style open streaming**, not
as bit-for-bit reproduction of an unbounded official serving session.

The audio encoder accepts a shorter final chunk with padding and an
actual-length mask. For the two-frame visual patch requirement, strict-grid
mode duplicates only the most recent past frame when necessary.

Run a four-second validation before using it for extraction:

```bash
python streaming_turn_model.py \
  --model-id /path/to/Qwen2.5-Omni-3B \
  --video prepared_easycom/Session_1/Session_1_2fps.mp4 \
  --use-audio-in-video \
  --mode open-stream-replay \
  --start 0 --end 4 \
  --video-fps 2 \
  --strict-fixed-video-grid \
  --verify-every 1 \
  --verify-continuous-every 1 \
  --flash-attention-2 \
  --output features/Session_1_open_first_4s.pt
```

`cache_replay` must be identical to replaying the same open stream from an
empty cache. `continuous_window_semantics` compares it with one normal
continuous Qwen AV input. Do not assume equivalence: the modality encoders are
still invoked per tail block in the efficient path, so this numerical check is
required even though artificial user-turn boundaries have been removed.

## Whole-session prediction and the 20-second limit

Twenty seconds is a practical memory/context setting for the 3090, not the
maximum video duration and not a limit specified by Qwen. The original
append-only idea eventually reaches both linear KV-memory growth and the
Thinker's 32,768-token context limit, so an unbounded multi-minute stream is
not practical. The complete session is processed sequentially and a
prediction is emitted every 0.2 seconds.

`open-stream-replay` uses blockwise rolling memory. With a 20-second
budget it rebuilds the open cache from the latest nine completed blocks
(18 seconds) when a new block starts, then adds the current 0--2 second tail.
Its effective history therefore varies from 18 to 20 seconds and never drops
to an empty 0.2-second context at a 20-second boundary. The nine blocks'
prepared AV embeddings and local TMRoPE coordinates are cached once; strict
window replay still recomputes the Thinker KV so retained blocks no longer
carry indirect influence from observations that fell out of the window.

For a short native-stream fidelity check, set `--memory-seconds` to at least
the tested clip duration. For example, the options
`--start 0 --end 40 --memory-seconds 40` prevent a rolling rebuild during that
40-second clip.
Do not extend this policy to complete EasyCom/Ego4D sessions: KV memory grows
linearly, and this implementation clones the cache for speculative partial
blocks and readout branches.

For the paper experiment, drive the output length from the preparation sidecar
rather than a floating MP4 duration:

```bash
python streaming_turn_model.py \
  --session-metadata prepared_easycom/Session_1/Session_1.json \
  --use-audio-in-video \
  --mode open-stream-replay \
  --memory-seconds 20 \
  --video-fps 2 \
  --strict-fixed-video-grid \
  --verify-every 25 \
  --verify-continuous-every 25 \
  --output features/Session_1_omni_2fps.pt
```

The sidecar selects the matching fixed-FPS MP4 and sets `N` to its exact
`num_steps`. Alternatively pass `--video ... --target ...` explicitly. A raw
20 FPS MP4 with `--video-fps 2` is rejected when
`--strict-fixed-video-grid` is enabled.

In strict A+V mode the extractor does not delegate temporal frame selection to
the generic video utility. It reads the already-fixed MP4 grid explicitly and
assigns both video frames and audio samples with half-open intervals
`[start, end)`. Thus a frame exactly on a two-second boundary belongs only to
the following block. It duplicates the most recent available frame when the
visual tower needs an even frame count. Audio is loaded once at 16 kHz and
sliced at the same causal boundary. The processor still receives
`use_audio_in_video=True`, so Qwen's audio/video token interleaving is retained.
Long runs use `tqdm` and update the progress bar after every completed state,
showing percentage, media time, elapsed time, ETA, and throughput. Pass
`--no-progress` to disable it. The Thinker-only path also suppresses Qwen's
irrelevant speech-output system prompt warning.

Before a full session, run only the first four seconds and inspect both check
types:

```bash
python streaming_turn_model.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --video /data1/yfzx/wangx/Datasets/easycom/omni_prepared/Session_1/Session_1_2fps.mp4 \
  --use-audio-in-video \
  --mode open-stream-replay \
  --start 0 --end 4 \
  --video-fps 2 \
  --strict-fixed-video-grid \
  --verify-every 1 \
  --verify-continuous-every 1 \
  --flash-attention-2 \
  --output features/Session_1_first_4s_check.pt
```

Do not launch all sessions unless `cache_replay` cosine is nearly 1. For
`open-stream-replay`, a materially lower `continuous_window_semantics` cosine
means that the custom payload stitching is not equivalent to one continuous
Qwen input; it must not be dismissed as a media-turn-boundary effect because
this mode intentionally keeps a single media turn open.

Video can be sampled at 2 FPS while predictions remain at 5 Hz. Audio and the
causal boundary update every 200 ms; consecutive prediction times may use the
same most-recent visual frame. A frame with timestamp later than `t` must never
be included.

## Direct extraction without a prepared 2 FPS MP4

`streaming_turn_model_directly.py` removes the offline video-transcoding
requirement. It reads original frame PTS with Decord, constructs a global
2 FPS lattice, maps every lattice point to the latest source frame whose PTS
is not later than that point, and passes explicit frame lists to Qwen. Audio is
decoded once to 16-kHz mono and sliced at exact causal boundaries. Therefore
no FFmpeg `fps` filter or second H.264/AAC encode can alter the experiment.

Validate one Ego4D clip without loading Qwen:

```bash
python streaming_turn_model_directly.py \
  --video /data/wangx/Datasets/ego4d/v1/clips/CLIP_UID.mp4 \
  --target /data/wangx/Datasets/ego4d/ego4d_target_frame/CLIP_UID.npy \
  --video-fps 2 \
  --media-only
```

Run a short native persistent-KV check:

```bash
python streaming_turn_model_directly.py \
  --model-id /data/wangx/Models/Qwen2.5-Omni-3B \
  --video /data/wangx/Datasets/ego4d/v1/clips/CLIP_UID.mp4 \
  --end 40 \
  --mode open-stream-replay \
  --memory-seconds 0 \
  --video-fps 2 \
  --flash-attention-2 \
  --device cuda:0 \
  --output features/direct/CLIP_UID_first_40s.pt
```

Here `--memory-seconds 0` means that completed-block Thinker KV persists for
the entire requested interval. Do not use this setting blindly for complete
multi-minute recordings: it has linear KV growth and remains subject to the
Thinker's context limit. A full bounded-memory dataset run should explicitly
use the experimental budget, for example `--memory-seconds 20`.

EasyCom can be read directly as the dense concatenation of retained clips;
filename timecode gaps are ignored and the rightmost 1920x1080 region is kept:

```bash
python streaming_turn_model_directly.py \
  --model-id /path/to/Qwen2.5-Omni-3B \
  --easycom-session-dir /path/to/Main/Video_Compressed/Session_1 \
  --target target_frame/Session_1.npy \
  --mode open-stream-replay \
  --memory-seconds 20 \
  --video-fps 2 \
  --flash-attention-2 \
  --device cuda:0 \
  --output features/direct/Session_1.pt
```

The direct reader defaults to Decord. On NVIDIA systems, install
`PyNvVideoCodec` and select the NVDEC backend explicitly:

```bash
python -m pip install PyNvVideoCodec

CUDA_VISIBLE_DEVICES=0 python streaming_turn_model_directly.py \
  --model-id /path/to/Qwen2.5-Omni-3B \
  --easycom-session-dir /path/to/Main/Video_Compressed/Session_1 \
  --end 24 \
  --mode open-stream-replay \
  --memory-seconds 20 \
  --video-fps 2 \
  --frame-cache-size 64 \
  --decoder pynvcodec \
  --decoder-gpu-id 0 \
  --flash-attention-2 \
  --device cuda:0 \
  --output features/direct/Session_1_24s_pynvcodec.pt
```

The NVDEC backend uses ffprobe for exact display-order PTS indexing, one
reconfigurable `SimpleDecoder` for pixel decode, and the existing ffmpeg PCM
audio path. `--decoder-gpu-id` is relative to `CUDA_VISIBLE_DEVICES`, so it is
normally `0` even when the worker is assigned a nonzero host GPU. Decoder RGB
conversion can differ slightly; validate a short Decord/PyNvVideoCodec pair
before extracting every session, and do not mix decoder backends across
train/test splits.

### Persistent-model Ego4D batch extraction

`extract_ego4d_batch.py` loads Qwen and its processor once, then extracts the
selected EgoSpeak manifest rows sequentially. Each clip receives a new media
reader, Thinker KV cache, TMRoPE origin, and rolling-memory state; only the
immutable model weights and processor are reused. Manifest indices are
one-based data-row indices and do not count the TSV header.

For example, process manifest samples 81 through the end on visible GPU 0:

```bash
CUDA_VISIBLE_DEVICES=0 \
OMP_NUM_THREADS=4 \
MKL_NUM_THREADS=4 \
OPENBLAS_NUM_THREADS=4 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
python extract_ego4d_batch.py \
  --data-root /data1/yfzx/wangx/Datasets/ego4d \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --start-index 81 \
  --memory-seconds 20 \
  --video-fps 2 \
  --decoder decord \
  --dtype bfloat16 \
  --flash-attention-2 \
  --device cuda:0
```

Completed non-empty `.pt` files are skipped. Each new output is first written
to a process-specific `.partial` path and atomically renamed after extraction,
so rerunning the same range resumes safely.

For CPU Decord, the direct reader intentionally indexes each clip and releases
that temporary reader, then keeps an LRU of at most two live readers during
extraction. Each reader defaults to one decode thread. These limits avoid one
worker retaining a thread pool for every one-minute EasyCom clip and can be
overridden with `--decoder-threads` and `--max-open-readers`. The recommended
full-run settings are:

```text
--decoder decord --decoder-threads 1 --max-open-readers 2
--frame-cache-size 64
```

Matching `states.shape[0]` to the target is necessary but not sufficient.
The direct reader also validates EasyCom's 20-Hz source inventory, source PTS
monotonicity, dense duration, crop bounds, audio sample count, and causal
frame-to-grid mapping. The original prepare scripts remain useful only as an
optional decode-speed/reproducibility cache.

## Saved streaming tensors

Each `.pt` file contains:

- `times`: `[N]`, prediction times in seconds;
- `states`: `[N, 2048]`, causal fused Thinker states;
- `label_indices`: `[0, ..., N-1]`, direct `target_perframe` row alignment;
- `valid_10_horizon_steps`: `N-10`, because the last ten current rows do not
  have all future labels;
- `mode`: `causal-prefix` or `open-stream-replay`;
- `stream_representation`: explicit input-organization metadata;
- `config`: step, native block, memory, FPS, and pixel settings;
- `verification`: cached-versus-exact numerical checks;
- `input_video_fps` and `video_sampling_grid`: sampling provenance;
- `media_interval_policy`: fixed-grid AV uses half-open `[start, end)` ownership;
- `logits`: optional `[N, 10, 3]` output if a trained head is supplied;
- `probabilities`: optional softmax probabilities.

## Training the `10 x 3` head

The head must be trained. It is not part of pretrained Qwen.

The first experiment should freeze all Qwen parameters and train only:

```text
LayerNorm(2048)
-> Linear(2048, 1024)
-> GELU
-> Dropout
-> Linear(1024, 30)
-> reshape [10, 3]
```

`easycom_head.py` defines the head and loss; they remain re-exported by
`streaming_turn_model.py` for compatibility:

```python
from streaming_turn_model import (
    TurnPredictionHead,
    egospeak_multihorizon_loss,
)
```

The supplied LSTR config uses Adam, 50 epochs, learning rate `7e-5`, weight
decay `5e-5`, dropout `0.2`, and 10 warmup epochs followed by cosine decay.
`train_turn_head.py` uses those defaults. Its batch size is 512 causal states,
not LSTR's 16 temporal windows, because the two model inputs have different
shapes.

Train once on the official nine sessions with a fixed 50-epoch protocol:

```bash
python train_turn_head.py \
  --feature-root features/open_2fps \
  --target-root target_frame \
  --output checkpoints/open_2fps_head.pth \
  --device cuda:0
```

The default loss is `lstr-mce`, exactly matching LSTR's MCE treatment of a
multi-positive `[0,1,1]` row: it sums both active negative-log-softmax terms
without normalizing the target. The previous L2-normalized objective belongs
to MiniROAD and is available only as the explicit
`--loss miniroad-normalized` ablation.

Two label-structure ablations are also available without changing the saved
three-column target files:

- `--loss independent-bce` keeps a `10 x 3` head, trains the three labels with
  independent sigmoid/BCE, and evaluates the sigmoid marginals directly;
- `--loss four-state-ce` changes the head to `10 x 4` and maps each target to
  `silence`, `wearer_only`, `others_only`, or `overlap`. At evaluation time it
  returns the original three marginal scores as `P(silence)`,
  `P(wearer_only)+P(overlap)`, and `P(others_only)+P(overlap)`.

The four-state marginals are intentionally not renormalized. Keep features,
split, fusion, optimizer, schedule, and seeds fixed when comparing these losses.
For checkpoint selection, reserve one official training session such as
Session 12 with `--dev-sessions 12`; Sessions 1--3 remain test-only.

Recommended experimental procedure:

1. Freeze Qwen and extract all training-session states offline.
2. Verify state/label time offset and truncate only the final ten invalid
   current positions.
3. Keep EgoSpeak targets as float multi-positive tensors `[N, 10, 3]`; never
   convert them with `argmax`.
4. For comparison with the supplied LSTR shell, train with the default
   `egospeak_multihorizon_loss`/`lstr-mce` objective.
5. Select hyperparameters using only training sessions. For EasyCom, reserve a
   training session for development or use session-level cross-validation;
   never tune on Sessions 1--3.
6. After choosing the epoch/hyperparameters, optionally retrain on all nine
   EasyCom training sessions and evaluate the fixed model once on the test set.

Class-balanced/focal loss, a learned turn-query token, and LoRA are later
ablations. They should not replace the frozen-Omni linear/MLP-head baseline.

### Offline frozen states with transition-sparse supervision

This is the clean sampling-only control.  It does not reconstruct cached AV
tokens and does not run the Thinker during training.  It loads the same dense
5 Hz frozen states used by the standard head baseline, then retains only the
`target_index` values listed by the existing transition-sparse cache
manifests.  The representation, head architecture, optimizer, 50 epochs, loss,
targets, and final dense test evaluator are otherwise unchanged.

Set `FROZEN_FEATURE_ROOT` to the exact feature directory used by the matching
dense-supervision baseline.  Do not compare sparse 8-second states with a
dense 20-second baseline (or vice versa):

```bash
FROZEN_FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_20s_decord
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
SPARSE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s

CUDA_VISIBLE_DEVICES=3 \
python train_turn_head.py \
  --feature-root "$FROZEN_FEATURE_ROOT" \
  --target-root "$TARGET_ROOT" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_7" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_11" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_10" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_6" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_9" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_4" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_8" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_5" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_12" \
  --output checkpoints/easycom_frozen_states_sparse_supervision_50ep.pth \
  --epochs 50 \
  --batch-size 512 \
  --expected-mode any \
  --device cuda:0
```

Evaluation remains dense 5 Hz on the official test sessions:

```bash
CUDA_VISIBLE_DEVICES=3 \
python infer_turn_head.py \
  --feature-root "$FROZEN_FEATURE_ROOT" \
  --target-root "$TARGET_ROOT" \
  --checkpoint checkpoints/easycom_frozen_states_sparse_supervision_50ep.pth \
  --output-dir results/easycom_frozen_states_sparse_supervision_50ep \
  --expected-mode any \
  --device cuda:0
```

Compare this result against a head trained for the same 50 epochs on every
dense 5 Hz anchor from the same `FROZEN_FEATURE_ROOT`.  A large drop isolates
the problem to sparse supervision; a close result means sparse supervision is
not sufficient to explain the LoRA regression.

## Thinker-LoRA sampling and cache policy

This is an implemented second-stage experiment, not yet a reported benchmark
result.
Dense 5 Hz partial-prefix caching is fast during training but does not fit the
combined EasyCom and Ego4D storage budget.  The 24-second Session 7 pilot used
0.115 GiB, which extrapolates to about 17.25 GiB per media hour.  In the latest
controlled single-L20 test without gradient checkpointing, 200 updates took
57.718 s (3.465 update/s) with every partial cached, 130.151 s (1.537
update/s) with current partials recomputed using processor defaults, and
111.146 s (1.799 update/s) after CUDA log-Mel and bounded-audio-padding
optimization.  The optimized online run spent 53.408 s in 163 frozen-AV calls;
the cached run therefore remains the preferred training path when storage is
available.

The proposed training sampler is therefore:

1. retain a uniform low-rate set of ordinary anchors, initially 1 Hz;
2. retain every 5 Hz anchor within a two-second band around annotated speaker
   onset, offset, handover, and overlap-state transitions;
3. optionally add high-loss or high-uncertainty anchors after a first training
   pass, while selecting them only from the training split;
4. cache exact partial AV embeddings only for these selected anchors, plus one
   reusable complete embedding for every required 2-second history block;
5. enforce a 240 GiB cache target and a hard 300 GB upper bound, leaving room
   for manifests, checkpoints, and extracted evaluation states.

Transition-centered selection is training-example sampling, not inference
look-ahead: model inputs must still end at the current time.  Final extraction,
head training, and Table 2 evaluation remain on the complete 5 Hz timeline.
The paper must report the sampling rule and compare uniform-only sampling with
transition-enhanced sampling so that any gain is not attributed solely to
additional transition examples.

`cache_omni_av_pieces.py --anchor-policy transition-sparse` implements this
policy without changing the trainer cache schema.  A transition is any change
in the three-class EasyCom target vector.  The default two-second dense band is
defined explicitly as one second before through one second after that target
row.  Every complete two-second source block is still cached once as reusable
history.  Ordinary selected anchors and transition-band anchors cache only
their exact current causal phase.  Label information selects training examples
offline but never changes the media endpoint supplied to Qwen.

Inspect the selection counts for the 24-second pilot without loading media or
the model:

```bash
python cache_omni_av_pieces.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_7 \
  --target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_7.npy \
  --output-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_7_24s \
  --end 24 \
  --anchor-policy transition-sparse \
  --ordinary-anchor-hz 1 \
  --transition-before-seconds 1 \
  --transition-after-seconds 1 \
  --selection-only
```

After auditing the counts, remove `--selection-only` and add the normal AV
extraction options.  Use `--end 0` only after the 24-second cache and replay
verification pass; zero selects the complete target timeline.  The resulting
manifest contains only selected training anchors, so `train_omni_turn_lora.py`
uses the sparse dataset automatically and must be run without
`--online-current-tail`.

### Multi-GPU Thinker-LoRA training

Fully cached AV training supports native single-node DDP through `torchrun`.
Each rank holds one complete 3B Thinker replica, reads a disjoint shard of the
selected cache anchors, and synchronizes only the trainable LoRA and turn-head
gradients.  Model checkpoint loading is deliberately staggered by rank so four
processes do not read all checkpoint shards simultaneously.  During gradient
accumulation, DDP synchronization occurs only on the final microbatch.  Only
rank zero writes the adapter, head, summary, and progress log.

For four GPUs, use per-GPU batch 2 and accumulation 2 for global batch 16:

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
OMP_NUM_THREADS=2 \
MKL_NUM_THREADS=2 \
OPENBLAS_NUM_THREADS=2 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
torchrun --standalone --nproc_per_node=4 train_omni_turn_lora.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_7 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_11 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_10 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_6 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_9 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_4 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_8 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_5 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_12 \
  --output-dir checkpoints/omni_turn_lora_easycom_sparse_8s_ddp4 \
  --max-steps 10170 \
  --batch-size 2 \
  --gradient-accumulation-steps 2 \
  --num-workers 0 \
  --block-cache-size 32 \
  --learning-rate 2e-5 \
  --head-learning-rate 1e-4 \
  --warmup-steps 500 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --log-every 20
```

`10170` is only the current three-epoch estimate for roughly 54,000 selected
anchors.  Recompute it from the completed manifests as
`ceil(num_examples / 16) * 3`.  Do not pass `--device`: under `torchrun`, each
process uses its `LOCAL_RANK`.  DDP intentionally rejects
`--online-current-tail`; the online producer-consumer path remains a separate
benchmark.

### Dense low-drift LoRA probe

The first follow-up to the sparse-LoRA regression should remove sparse
supervision as a variable and constrain adaptation near the Thinker output.
Use dense 5-Hz caches, reserve `Session_12` as development data, and train on
`Session_4` through `Session_11`.  `--lora-last-n-layers 8` discovers the
language-layer indices and passes the 24 selected full module paths directly
to PEFT for the 36-layer Qwen2.5-Omni-3B Thinker.  Explicit paths are required
because the extracted Thinker exposes names beginning with `layers.0...`,
which some PEFT `layers_pattern` versions fail to recognize. Combining it with
`--lora-target-modules q_proj,v_proj,o_proj` reduces the adapter from all 252
attention/MLP projections to 24 output-side attention projections.

Start with the successful frozen 8-second head fixed.  `--freeze-head`
requires `--head-checkpoint`, excludes the head from DDP and AdamW, but still
backpropagates through it into the Thinker LoRA.  This isolates representation
adaptation from head decision-boundary drift.  Do not continue from a regressed
adapter: create this adapter from the original Thinker.

For four GPUs with global batch 16, compute exactly one epoch from the dense
manifest anchor counts and run:

```bash
cd /data1/yfzx/wangx/Code/icassp2027-experiment

MAX_STEPS=$(python - <<'PY'
import json
import math
from pathlib import Path

root = Path("/data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s")
sessions = (4, 5, 6, 7, 8, 9, 10, 11)
examples = sum(len(json.loads((root / f"Session_{s}" / "manifest.json").read_text())["anchors"])
               for s in sessions)
print(math.ceil(examples / 16))
PY
)
WARMUP_STEPS=$(( (MAX_STEPS + 19) / 20 ))

CUDA_VISIBLE_DEVICES=0,1,2,3 \
OMP_NUM_THREADS=2 \
MKL_NUM_THREADS=2 \
OPENBLAS_NUM_THREADS=2 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
torchrun --standalone --nproc_per_node=4 train_omni_turn_lora.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_4 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_5 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_6 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_7 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_8 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_9 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_10 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_11 \
  --head-checkpoint checkpoints/easycom_open_stream_2fps_8s_head_seed0.pth \
  --freeze-head \
  --lora-target-modules q_proj,v_proj,o_proj \
  --lora-last-n-layers 8 \
  --lora-rank 8 \
  --lora-alpha 16 \
  --learning-rate 5e-6 \
  --output-dir checkpoints/omni_turn_lora_dense_8s_last8_qvo_frozen_head_epoch1 \
  --max-steps "${MAX_STEPS}" \
  --batch-size 2 \
  --gradient-accumulation-steps 2 \
  --num-workers 0 \
  --block-cache-size 32 \
  --warmup-steps "${WARMUP_STEPS}" \
  --dtype bfloat16 \
  --flash-attention-2 \
  --log-every 20
```

Select this recipe on dense `Session_12` using 0.2-second mAP as the primary
metric and mean mAP as the tie-breaker.  Only after the recipe is fixed should
it be evaluated on official test `Sessions_1--3`.  A second controlled run may
omit `--freeze-head` and use `--head-learning-rate 1e-5`; all other settings
must remain identical.

For a continuous five-epoch run, set
`TOTAL_STEPS=$((5 * MAX_STEPS))` and pass
`--max-steps "${TOTAL_STEPS}" --save-every-steps "${MAX_STEPS}"`.  The trainer
keeps one AdamW/cosine-scheduler trajectory and writes evaluation-ready
`adapter/` plus `turn_head.pt` pairs under `checkpoint-step-MAX_STEPS`,
`checkpoint-step-2*MAX_STEPS`, through the fifth epoch.  The top-level adapter
and head duplicate the final fifth checkpoint.  Evaluate all five intermediate
checkpoints only on held-out `Session_12`, select one, then run that checkpoint
once on `Sessions_1--3`.

### Frozen-Thinker sparse head-only control

To determine whether the regression comes from transition-sparse sampling or
from updating the Thinker representation, rerun the same sparse-anchor
training with `--freeze-thinker`.  This still forwards every reconstructed
8-second sequence through the original Thinker, but runs that forward under
`no_grad` and updates only the turn head initialized from the frozen 8-second
checkpoint.  It does not create an adapter directory.

For a schedule-matched comparison with a one-epoch LoRA run at global batch
16, use the same cache directories, head learning rate, warmup, seed, and
approximately `ceil(num_examples / 16)` updates:

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
OMP_NUM_THREADS=2 \
MKL_NUM_THREADS=2 \
OPENBLAS_NUM_THREADS=2 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
torchrun --standalone --nproc_per_node=4 train_omni_turn_lora.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_7 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_11 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_10 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_6 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_9 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_4 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_8 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_5 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_12 \
  --freeze-thinker \
  --head-checkpoint checkpoints/easycom_open_stream_2fps_8s_head_seed0.pth \
  --output-dir checkpoints/easycom_sparse_head_only_from_frozen_8s_ddp4_epoch1 \
  --max-steps 3390 \
  --batch-size 2 \
  --gradient-accumulation-steps 2 \
  --num-workers 0 \
  --block-cache-size 32 \
  --head-learning-rate 1e-4 \
  --warmup-steps 500 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --log-every 20
```

Evaluate the resulting head on the same dense test caches without an adapter:

```bash
CUDA_VISIBLE_DEVICES=0,1,2 \
OMP_NUM_THREADS=1 \
MKL_NUM_THREADS=1 \
OPENBLAS_NUM_THREADS=1 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
torchrun --standalone --nproc_per_node=3 eval_omni_turn_lora.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_3 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_2 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_1 \
  --head-checkpoint checkpoints/easycom_sparse_head_only_from_frozen_8s_ddp4_epoch1/turn_head.pt \
  --output-dir results/easycom_sparse_head_only_dense_eval_epoch1 \
  --batch-size 2 \
  --block-cache-size 32 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --ddp-timeout-minutes 60
```

If this head-only control also falls well below the frozen `58.16` dense-cache
control, sparse sampling/head optimization is sufficient to explain the
regression.  If it stays near `58.16` while LoRA falls, the main failure is
Thinker representation drift.

To continue from a completed run, pass both its `adapter/` directory through
`--adapter-checkpoint` and its `turn_head.pt` through `--head-checkpoint` while
writing to a new output directory.  This is weights-only continuation: the
adapter and head weights resume, but AdamW moments and the cosine scheduler are
reinitialized.  Preserve the previous global batch when changing GPU count;
for example, change 8 GPUs with batch 2 and accumulation 1 to 4 GPUs with batch
2 and accumulation 2.  Use zero warmup and a reduced learning rate for a short
post-hoc continuation because a one-epoch cosine schedule has already reached
zero.

### Evaluate a Thinker-LoRA checkpoint

Use `eval_omni_turn_lora.py` for the formal EasyCom Table 2 evaluation.  It
loads the trained Thinker LoRA and `10 x 3` turn head, reconstructs every causal
state from dense AV caches, and reports both the EgoSpeak full-timeline and
strict causal-valid protocols.  Sparse training caches are rejected because
they do not contain predictions at every 200 ms timestamp.

The following command evaluates the official EasyCom test sessions on three
GPUs, with one complete session assigned to each rank:

```bash
CHECKPOINT=/data1/yfzx/wangx/Code/icassp2027-experiment/checkpoints/omni_turn_lora_easycom_sparse_8s_ddp8_epoch1

CUDA_VISIBLE_DEVICES=0,1,2 \
OMP_NUM_THREADS=1 \
MKL_NUM_THREADS=1 \
OPENBLAS_NUM_THREADS=1 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
torchrun --standalone --nproc_per_node=3 eval_omni_turn_lora.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_3 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_2 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_1 \
  --adapter-checkpoint "${CHECKPOINT}/adapter" \
  --head-checkpoint "${CHECKPOINT}/turn_head.pt" \
  --output-dir results/omni_turn_lora_easycom_sparse_8s_ddp8_epoch1 \
  --batch-size 2 \
  --block-cache-size 32 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --ddp-timeout-minutes 60
```

The evaluator saves per-session probabilities and valid masks under `pred/`
and `valid/`, and writes the aggregate metrics to `metrics.json`.  Add
`--overwrite` only when intentionally replacing an existing result directory.
Treat Sessions 1, 2, and 3 as the official test set: inspect their metrics only
after the training recipe and hyperparameters have been fixed.

To isolate whether a score change comes from LoRA or from the dense-cache
reconstruction path, run the same evaluator without `--adapter-checkpoint` and
load the original frozen 8-second head:

```bash
FROZEN_HEAD=checkpoints/easycom_open_stream_2fps_8s_head_seed0.pth

CUDA_VISIBLE_DEVICES=0,1,2 \
OMP_NUM_THREADS=1 \
MKL_NUM_THREADS=1 \
OPENBLAS_NUM_THREADS=1 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
torchrun --standalone --nproc_per_node=3 eval_omni_turn_lora.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_3 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_2 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_1 \
  --head-checkpoint "${FROZEN_HEAD}" \
  --output-dir results/easycom_frozen_8s_dense_cache_control \
  --batch-size 2 \
  --block-cache-size 32 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --ddp-timeout-minutes 60
```

This control should approach the previously reported frozen 8-second score of
58.51937 full-timeline mean mAP.  A material mismatch indicates a representation
difference between dense-cache reconstruction and the earlier open-stream
feature files; a close match isolates the regression to LoRA/head training.

### Recorded sparse-LoRA results

The current single-run EasyCom results confirm negative transfer from this
sparse LoRA recipe:

| Checkpoint | Full-timeline mean mAP | Strict causal mean mAP |
|---|---:|---:|
| Frozen Thinker dense-cache control | 58.16091 | 58.17146 |
| Frozen states + sparse supervision, 50 epochs | 57.20379 | 57.21367 |
| Sparse LoRA epoch1 | 56.52480 | 56.53450 |
| Frozen-head initialized LoRA, 3 epochs | 56.19608 | 56.20535 |
| Sparse LoRA weights-only epoch2 | 54.19850 | 54.20711 |

Epoch2 is a weights-only continuation: adapter/head weights resume, but AdamW
and the scheduler restart. It is 2.32630 mAP points below epoch1 and 3.96241
below the frozen dense-cache control in the full-timeline protocol. The largest
regression is camera-wearer speaking. Do not continue the same recipe to a
third run. The complete snapshot and per-class metrics are archived in
[`results/easycom_omni_lora_sparse_epoch2_2026-08-24/`](results/easycom_omni_lora_sparse_epoch2_2026-08-24/).

The frozen-state sparse-supervision and frozen-head initialized three-epoch
LoRA tables are archived in
[`results/easycom_sparse_supervision_vs_lora_2026-08-24/`](results/easycom_sparse_supervision_vs_lora_2026-08-24/README.md).
At 0.2 seconds, sparse supervision alone changes mAP from `71.69695` to
`71.69194` (effectively unchanged), while the three-epoch LoRA result is
`71.08589`. This isolates the short-horizon regression primarily to Thinker
updating rather than sparse target selection.

For a speed-only alternative, `train_omni_turn_lora.py` supports a two-GPU
producer-consumer benchmark.  The producer GPU runs the frozen AV towers and
places current-tail embeddings in a bounded CPU queue while the consumer GPU
runs Thinker LoRA.  This preserves exact online partial semantics; it changes
only scheduling.  Compare its reported update rate and consumer queue-wait
time with the synchronous 0.787 update/s pilot before adopting it.

## EgoSpeak-compatible evaluation

Run inference on the official test sessions (`Session_3`, `Session_2`, and
`Session_1`):

```bash
python infer_turn_head.py \
  --feature-root features/open_2fps \
  --target-root target_frame \
  --checkpoint checkpoints/open_2fps_head.pth \
  --output-dir results/open_2fps_head \
  --device cuda:0
```

This writes `pred/Session_X.npy` with the exact LSTR/EgoSpeak layout
`[target_time, class, future_horizon]`, i.e. `[T,3,10]`, plus `metrics.json`.
For every valid current timestep, the head itself produces probabilities
shaped:

```text
[number of timesteps, 10 future horizons, 3 classes]
```

Then reproduce EgoSpeak's Table 2 procedure:

1. for each horizon independently, concatenate test-session predictions and
   ground truth;
2. compute per-class frame-level average precision;
3. average the three class AP values to obtain horizon mAP;
4. report all ten horizon mAP values;
5. average the ten horizon mAP values for the final score.

With EgoSpeak's `ignore_index=-100`, the released evaluator does not remove
class index zero, so background is included. The script reports two protocols:

- `egospeak_full_timeline`: the same full `[T,3,10]` array and evaluator as
  LSTR, with zeros where a prediction would require pre-session context;
- `causal_valid_only`: excludes only those unavailable first `h` rows for
  horizon `h`. This is the primary strictly causal score.

The difference involves at most the first ten rows of each session, but both
are saved so the comparison is explicit.

The convenience wrapper mirrors EgoSpeak's train/inference shells:

```bash
chmod +x run_easycom_head_experiment.sh
CUDA_VISIBLE_DEVICES=0 ./run_easycom_head_experiment.sh all
```

Reference Table 2 targets from the EgoSpeak paper:

- EasyCom GRU A+V: **60.6 mean mAP**;
- EasyCom Transformer A+V: **58.7 mean mAP**;
- Ego4D Transformer audio: **69.2 mean mAP**;
- Ego4D Transformer A+V: **69.0 mean mAP**;
- Ego4D GRU audio: **69.2 mean mAP**;
- Ego4D GRU A+V: **68.2 mean mAP**.

Do not compare these numbers with EgoSpeak Table 3 target-speaker-only AP;
Table 3 uses a different metric.

## Experiments that demonstrate Omni's semantic advantage

The headline comparison alone cannot establish that improvements come from
semantics. Run the following controlled experiments:

1. **Omni audio vs Omni audio+video**: measure whether visual grounding helps.
2. **Per-horizon curve**: semantic completion should be especially visible at
   longer horizons such as 1.2--2.0 seconds.
3. **Context length**: compare 4, 10, 15, and 20 seconds without adding an RNN.
4. **Layer ablation**: compare audio-encoder output and several Thinker layers,
   because an intermediate layer may be more discriminative than the final
   generation-oriented layer.
5. **Semantic-completion subset**: contrast completed utterances with
   hesitation/incomplete pauses while matching pause duration and basic
   prosody as closely as possible.
6. **Question/statement/backchannel subsets**: report target-speaker AP and
   premature-interruption rate on linguistically meaningful events.
7. **Causal transcript baseline**: compare Omni audio with causal ASR text plus
   a language model to isolate lexical/semantic contribution.

The frozen-Omni head is the cleanest representation test. LoRA can be reported
as a second-stage ceiling, not as the only result.

## Transcript-derived interaction-event proxies

EasyCom's released human transcriptions contain inclusive 20 FPS
`Start_Frame`/`End_Frame`, `Participant_ID`, `Transcription`, and
`Target_of_Speech`.  They do **not** contain a semantic backchannel class.
[`derive_easycom_interaction_events.py`](derive_easycom_interaction_events.py)
therefore separates observed timing relations from weak semantic proxies:

- `ordinary_switch`: a non-overlapping recent speaker change followed by a
  sustained incoming turn;
- `overlap_takeover`: an observed overlapping onset followed by sustained
  takeover, without claiming interruption intent;
- `overlap_hold`: overlap without enough evidence for takeover;
- `backchannel_proxy_high`: a conservative short lexical response while the
  same other participant dominates immediately before and after it;
- `interruption_proxy_high`: the previous transcript has a non-boundary
  trailing cutoff and the incoming participant takes over;
- `nonverbal_feedback`: laughter-only feedback around another participant's
  floor; pure `[H]`, `[C]`, `[T]`, and `[U]` records are ambiguous candidates.

`backchannel_candidate`, `interruption_candidate`, and
`nonverbal_candidate` are retained for auditing but masked from auxiliary
training.  `[X]` continuations across adjacent one-minute files are not new
events.  Source-time gaps larger than one second create separate segments, so
the dense EgoSpeak concatenation does not create a false switch across a
redacted interval.  A one- or two-frame annotation/media endpoint discrepancy
is clamped and reported; larger disagreement is fatal.

A read-only audit on the raw-data machine is:

```bash
cd /mnt/58462F4C462F29E8/Code/icassp2027-experiment

python3 derive_easycom_interaction_events.py \
  --dataset-root /mnt/58462F4C462F29E8/Datasets/easycom \
  --target-root /mnt/58462F4C462F29E8/Code/EgoSpeak/dataset/EasyCom/target_perframe \
  --examples-per-event 10
```

To persist JSONL/CSV reports outside the dataset, add:

```bash
  --output-dir reports/easycom_interaction_events_v1 \
  --overwrite
```

In an environment with NumPy, add `--write-frame-arrays` to also write one
`Session_*.aux_targets.npz` per session.  It contains 5 Hz `event_onsets`,
`event_roles`, collision flags, and event-name metadata aligned to
`target_perframe`.  The raw EasyCom tree is never modified.

The initial all-session audit found 6,981 utterances, including 530
high-confidence backchannel proxies, 78 high-confidence interruption proxies,
937 overlap-takeovers, 686 overlap-holds, and 349 laughter-feedback events.
These counts establish usable weak-supervision coverage, not label accuracy.
Before training, inspect a stratified sample from Sessions 4--12 and freeze the
rules; do not tune thresholds on test Sessions 1--3.  The primary evaluator
remains the original EgoSpeak three-label AP/mAP protocol.

### Optional DeepSeek adjudication of ambiguous candidates

DeepSeek is not required for the deterministic timing labels above.  It is an
optional second-stage weak-label adjudicator for only
`backchannel_candidate` and `interruption_candidate`; sending all 6,981
utterances to an LLM would add cost and noise without adding observable
prosody, gaze, or intent.  The current audit contains 614 such candidates over
all sessions and 497 in the training split (Sessions 4--12).

[`adjudicate_easycom_interaction_events.py`](adjudicate_easycom_interaction_events.py)
uses DeepSeek JSON Output through the standard-library HTTP client, so no
additional Python package is required.  It makes two differently instructed
conservative passes.  A candidate is promoted only if both passes return the candidate's
matching semantic label with high confidence; disagreements, medium/low
confidence, short answers, cooperative overlaps, and uncertain cases remain
masked.  The result is still a weak proxy and must not be presented as human
ground truth.  The append-only JSONL records prompt/model metadata and resumes
completed candidates, while rejecting a mixed model or prompt configuration.

First inspect exactly what would leave the machine.  This makes no API call and
does not require a key:

```bash
python3 adjudicate_easycom_interaction_events.py \
  --input-dir reports/easycom_interaction_events_v1 \
  --output reports/easycom_interaction_llm_v1.jsonl \
  --max-events 16 \
  --dry-run
```

If external transcript upload is authorized, enter the key without placing it
in shell history and run a 32-candidate pilot:

```bash
read -s -p "DeepSeek API key: " DEEPSEEK_API_KEY; echo
export DEEPSEEK_API_KEY

python3 adjudicate_easycom_interaction_events.py \
  --input-dir reports/easycom_interaction_events_v1 \
  --output reports/easycom_interaction_llm_pilot_v1.jsonl \
  --max-events 32 \
  --selection-seed 2027 \
  --batch-size 8 \
  --passes 2 \
  --allow-external-transcript-upload

unset DEEPSEEK_API_KEY
```

Audit the pilot rationales and precision manually before running the full
training split.  For the full pass, use a new output path, remove
`--max-events`, and keep the default `--splits train`.  The default model is
`deepseek-v4-flash`; it can be changed with `--model` or `DEEPSEEK_MODEL` as
the provider's available model names evolve.  Never tune the prompt or
acceptance rule using Sessions 1--3.  LLM-adjudicated candidates are deliberately
kept in a separate JSONL until pilot precision is accepted; they are not yet
silently merged into the training arrays.

## Candidate improvement: explicit working and long memory

Motivated by EgoSpeak's separation of long-term and working memory, test an
explicit two-timescale representation instead of compressing the entire causal
window into one undifferentiated final Thinker state.  At prediction time `t`,
align long memory to fixed global 2-second boundaries and let working memory
slide with the current prediction time:

```text
b = floor(t / 2 s) * 2 s
long memory:     complete 2-second chunks ending at b, ..., [b-4, b-2], [b-2, b]
working memory:  the sliding recent interval [t-2, t], updated every 0.2 s
```

For example, at `t = 20.4 s`, long memory contains the fixed chunks
`[16,18]`, `[18,20]`, and any earlier retained chunks, while the working
partial is `[18.4,20.4]`.  The overlap between `[18,20]` and `[18.4,20.4]` is
intentional: long memory provides a stable completed-chunk representation,
whereas working memory preserves the exact high-resolution context ending at
the current prediction time.  Cache one representation per completed
long-memory chunk, recompute the working-memory representation at each 5 Hz
prediction step, and fuse the two branches with a small gated/attention
pooling module before the same `10 x 3` prediction head.  Keep the Omni
backbone frozen in the first test so that any gain can be attributed to memory
organization rather than additional backbone capacity.

This is a candidate score-improvement idea, not yet an experimental result.
The key hypothesis is that a dedicated recent branch preserves local prosody,
speaker-change, and utterance-completion cues while long memory supplies
conversation context without diluting them in one final state.  Evaluate it
against matched-compute baselines: (1) the current single-state 8-second model,
(2) working-memory-only `[t-2,t]`, and (3) long+working memory with the same
total causal context.  Report parameter count, throughput, and per-horizon and
per-class AP over identical seeds.  A useful result requires the fused model to
beat both controls consistently; otherwise a gain could merely come from the
extra fusion parameters or the stronger recent-window shortcut.

### Frozen serialized-recap ablation (implemented)

Before adding a learned two-branch fusion module, test a stricter frozen-Omni
variant with exactly the same 2048-dimensional readout and prediction head as
the current 8-second baseline. Mode `overlap-working-memory` serializes the two
timescales inside one still-open Thinker media turn:

```text
total AV presentation budget = 8 s
fixed history budget         = 6 s = three cached 2 s blocks
working recap                = 2 s = [t-2,t]

t = 20.4 s:
fixed:   [14,16] [16,18] [18,20]
working:                   [18.4,20.4]
                                      -> one frozen Thinker state [2048]
```

The overlap is intentional. It presents recent evidence twice but does not add
AV-token duration beyond the matched 8-second budget. Thus this ablation trades
older unique context for stronger recent evidence instead of silently giving
the candidate more input tokens. Fixed blocks retain the global half-open 2 FPS
grid. The working recap reuses three causal global-grid history frames and
replaces the oldest visual slot with the current causal frame:

```text
fixed [18,20):       18.0  18.5  19.0  19.5
working [18.4,20.4]: 18.5  19.0  19.5  20.4
```

Keeping four frames means the oldest visual tick is replaced by the current
frame rather than adding a fifth frame; this isolates endpoint recency without
increasing visual tokens. Historical target times are causally floored to the
session-global 2 FPS grid. This lets them share the complete-block frame cache
and avoids making Decord seek backward through a long-GOP H.264 stream for four
new endpoint-shifted timestamps every 200 ms. Audio remains the full, exactly
half-open `[start,end)` slice. Consequently, call this a global-history plus
current-frame recent-video intervention, not an exact timestamp-preserving
resampling of both modalities. The recap is serialized as the next TMRoPE
chunk; it is deliberately a recent-evidence recap, not a claim that duplicated
samples occur later in physical time. This mode is an ablation, not native Qwen
streaming semantics, and the Omni backbone stays frozen.

One deliberate side effect is a 2-second phase cycle: overlap falls toward
0.2 seconds just before a fixed boundary and jumps to 2.0 seconds at the
boundary. Every output stores this value in
`overlap_working_memory_policy.layout`. If aggregate mAP changes, also inspect
performance by `step_index % 10`; a boundary-only gain or failure is a chunk
phase artifact rather than general evidence for better recent semantics.

Run a 24-second L20 smoke test first:

The sliding 2-second recap otherwise invokes Whisper's model-maximum audio
padding 120 times. Run log-Mel extraction on the model GPU and set explicit
right padding to zero. Every EasyCom 0.2-second increment contains 3200 samples
at 16 kHz and is already divisible by Whisper's 160-sample hop, so no external
silence is needed. This preserves the feature extractor's own STFT boundary
handling while avoiding repeated processing of tens of seconds of zeros.

```bash
mkdir -p features/easycom_overlap_working_8s_smoke

CUDA_VISIBLE_DEVICES=3 \
OMP_NUM_THREADS=4 \
MKL_NUM_THREADS=4 \
OPENBLAS_NUM_THREADS=4 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
python streaming_turn_model_directly.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_7 \
  --end 24 \
  --mode overlap-working-memory \
  --memory-seconds 8 \
  --working-memory-seconds 2 \
  --video-fps 2 \
  --frame-cache-size 128 \
  --audio-feature-device model \
  --audio-right-padding-samples 0 \
  --decoder decord \
  --decoder-threads 1 \
  --max-open-readers 1 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --device cuda:0 \
  --verify-every 0 \
  --output features/easycom_overlap_working_8s_smoke/Session_7_24s_global_history_current.pt
```

Expected shape is `(120, 2048)`. This speed smoke test disables replay
verification; after its throughput is acceptable, rerun a short four-second
check with `--verify-every 5`, where `overlap_cache_replay` should be
numerically exact or effectively exact. Inspect the saved timing policy and
decoder cache counters with:

```bash
python - <<'PY'
import torch

p = torch.load(
    "features/easycom_overlap_working_8s_smoke/Session_7_24s_global_history_current.pt",
    map_location="cpu",
    weights_only=False,
)
print(p["mode"], tuple(p["states"].shape))
print(p["overlap_working_memory_policy"]["working_media"])
print(p["direct_media"]["runtime_counters"])
for row in p["overlap_working_memory_policy"]["layout"][98:102]:
    print(row)
PY
```

For one complete session, replace `--end 24` with its target and use a
session-level output filename:

```bash
mkdir -p /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s

CUDA_VISIBLE_DEVICES=3 \
OMP_NUM_THREADS=4 \
MKL_NUM_THREADS=4 \
OPENBLAS_NUM_THREADS=4 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
python streaming_turn_model_directly.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_7 \
  --target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_7.npy \
  --mode overlap-working-memory \
  --memory-seconds 8 \
  --working-memory-seconds 2 \
  --video-fps 2 \
  --frame-cache-size 128 \
  --audio-feature-device model \
  --audio-right-padding-samples 0 \
  --decoder decord \
  --decoder-threads 1 \
  --max-open-readers 1 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --device cuda:0 \
  --output /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s/Session_7.pt
```

After all train/test sessions have been extracted, train the unchanged frozen
head by setting only the expected feature mode:

```bash
python train_turn_head.py \
  --feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s \
  --target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
  --output checkpoints/easycom_overlap_working_frozen_8s.pt \
  --expected-mode overlap-working-memory \
  --epochs 50 \
  --batch-size 512 \
  --device cuda:0

python infer_turn_head.py \
  --feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s \
  --target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
  --checkpoint checkpoints/easycom_overlap_working_frozen_8s.pt \
  --output-dir results/easycom_overlap_working_frozen_8s \
  --expected-mode overlap-working-memory \
  --device cuda:0
```

The decisive comparison is against the frozen single-state 8-second baseline
using the same split, head, optimization, and seeds. Report the full
per-horizon curve: the hypothesis predicts the clearest gain at 0.2--1.0
seconds; degradation at 1.2--2.0 seconds would indicate that repeated local
cues displaced too much useful long context. Only if this parameter-matched
test is positive should the learned two-branch fusion experiment be added.

## Single-window feature inspection

`qwen_omni_features.py` is useful for inspecting a short window or layer, but
it is not the main whole-session experiment.

If audio is embedded in the video:

```bash
python qwen_omni_features.py \
  --video /path/to/window.mp4 \
  --use-audio-in-video \
  --output features/window.pt
```

If audio and video are separate:

```bash
python qwen_omni_features.py \
  --video /path/to/window.mp4 \
  --audio /path/to/window.wav \
  --output features/window.pt
```

Do not provide the same audio both as a separate file and via
`--use-audio-in-video`.

Saved single-window fields are:

- `token_features`: `[B, token_length, 2048]`;
- `last_token`: `[B, 2048]`;
- `tail_mean`: `[B, 2048]`;
- `attention_mask`: valid-token mask.

## Installation

Install `ffmpeg`, then:

```bash
pip install -r requirements.txt
```

The implementation is pinned to `transformers==4.52.3` because the
open-stream replay path uses version-specific Thinker internals. On Linux,
`qwen-omni-utils[decord]` is recommended for faster video decoding. On a
compatible NVIDIA GPU, optionally install `flash-attn` and pass
`--flash-attention-2`.

The separate B1 local extractor, `extract_turn_local_features.py`, is tested
with both Transformers 4.52.3 and 4.57.6. An existing 4.57.6 GPU container does
not need to be downgraded for local extraction. This does not change the replay
API's dependency assumptions. Keep all local sidecars for one experiment on
the same version; their metadata records the actual runtime version. See
[the local feature guide](docs/local_feature_baseline.md).

## Ego4D Audio-Visual Diarization download

Do not download the complete Ego4D release for this experiment. Download only
the compressed Audio-Visual Diarization benchmark clips and annotations:

```bash
ego4d \
  --output_directory=/path/to/ego4d \
  --datasets clips annotations \
  --benchmarks AV
```

The approximately 1 TB figure on the Ego4D download page refers to all
benchmark clips combined, not only the AV subset. The annotated AVD subset has
about 572 five-minute clips (roughly 50 hours). EgoSpeak uses a custom 346/87
train/test split, about 36.1 hours total. Reproducing its Table 2 requires the
exact IDs in EgoSpeak's `EGO4DSHUFFLE` split.

## Required validation before a full run

Do not launch full EasyCom/Ego4D extraction until all of these pass:

- no input audio sample or video frame has timestamp later than prediction
  time `t`;
- `states` and `target_perframe` lengths/time origins are reconciled and the
  chosen label offset is saved;
- open-stream cache replay is exactly equivalent to its from-scratch replay on
  short clips;
- open-stream and causal-prefix predictions are compared on identical clips;
- a 20-second run fits the 3090 with the selected pixel budget and attention
  implementation;
- saved outputs contain exactly ten future horizons in the order 0.2--2.0 s;
- the local metric reproduces EgoSpeak's evaluator on known predictions;
- test session IDs never enter training, validation, normalization statistics,
  or hyperparameter selection.

Once these checks pass, the first publishable experiment is:

```text
frozen Qwen2.5-Omni-3B Thinker
+ trained 10x3 head
+ bounded Qwen-style 2-second open streaming
+ 20-second rolling causal memory
+ full-session 5 Hz output
+ exact EgoSpeak split and Table 2 metric
```

Repository workflow verified on 2026-08-19.