2122 lines
88 KiB
Markdown
2122 lines
88 KiB
Markdown
# Omni-only Turn-Taking Experiment
|
||
|
||
This project tests whether **Qwen2.5-Omni-3B can replace both the handcrafted
|
||
audio/visual feature extractors and the downstream RNN/Transformer used by
|
||
EgoSpeak** for egocentric turn-taking anticipation.
|
||
|
||
The persistent paper index and Chinese summaries are maintained in
|
||
[literature_notes](literature_notes/README.md).
|
||
|
||
The pooled local-feature B1 experiment (frozen Qwen face crops + recent audio,
|
||
native 2048D sidecars, optional trainable projection, four-epoch ablations) is
|
||
documented in [docs/local_feature_baseline.md](docs/local_feature_baseline.md).
|
||
|
||
For **EasyCom encoder-only A / V / AV** with no Thinker states, start with
|
||
[the EasyCom comparison commands](docs/easycom_encoder_only.md).
|
||
They reuse the verified local sidecars, train on Session 4–12, test on 3/2/1,
|
||
and keep four epochs with no dev set.
|
||
|
||
The main experiment is deliberately simple:
|
||
|
||
```text
|
||
causal audio + video up to time t
|
||
|
|
||
v
|
||
Qwen2.5-Omni-3B Thinker
|
||
|
|
||
fused state q_t [2048]
|
||
|
|
||
v
|
||
trainable prediction head
|
||
|
|
||
v
|
||
future logits [10 horizons, 3 classes]
|
||
```
|
||
|
||
There is **no RNN after Qwen**. The Omni Thinker is the temporal model. The
|
||
only supervised component in the first experiment is a small `10 x 3`
|
||
classification head.
|
||
|
||
## Current status
|
||
|
||
| File | Status | Purpose |
|
||
|---|---|---|
|
||
| `smoke_test.py` | Verified on one NVIDIA L20 | Text and 12-second EasyCom audio-video inference |
|
||
| `easycom_manifest.py` | Verified on EasyCom | Discover one-minute clips, preserve session timecodes, and check 5 Hz targets |
|
||
| `qwen_omni_features.py` | Implemented, not executed here | Load only the 3B Thinker and extract fused states from one AV window |
|
||
| `streaming_turn_model.py` | Verified on EasyCom | Produce one causal Omni state every 200 ms, with causal-prefix and open-stream modes |
|
||
| `TurnPredictionHead` | Trained and evaluated | Map `[2048]` to `[10, 3]` logits |
|
||
| `train_turn_head.py` | Verified on EasyCom | Train the frozen-Omni anticipation head with the official EasyCom split |
|
||
| `infer_turn_head.py` | Verified on EasyCom | Save EgoSpeak-format predictions and reproduce per-horizon AP/mAP |
|
||
| `eval_omni_turn_lora.py` | Implemented, remote validation pending | Evaluate Thinker LoRA from dense AV caches with the same AP/mAP protocol |
|
||
|
||
The full Thinker input path, embedded audio, direct video decoding,
|
||
FlashAttention-2, 5 Hz streaming extraction, cache replay, target alignment,
|
||
head training, and EgoSpeak-compatible evaluation have now been exercised on
|
||
EasyCom. The first Ego4D 8-second versus 20-second context ablation has also
|
||
been recorded. Multi-seed evaluation remains in progress.
|
||
|
||
## Current EasyCom results
|
||
|
||
The current single-run comparison was recorded on 2026-08-19. Both modes use
|
||
the frozen Qwen2.5-Omni-3B Thinker, 2 FPS video, 5 Hz predictions, a 20-second
|
||
causal context budget, the same `10 x 3` head architecture, and EgoSpeak's
|
||
official EasyCom train/test split.
|
||
|
||
| Representation | EgoSpeak full-timeline mean mAP | Strict causal mean mAP |
|
||
|---|---:|---:|
|
||
| Open-stream replay | 58.316 | 58.326 |
|
||
| Causal-prefix | 58.830 | 58.841 |
|
||
|
||
Causal-prefix is 0.514 mAP points higher in the exact Table 2 protocol, but
|
||
the difference is concentrated at 0.2--0.6 seconds. Open-stream is slightly
|
||
higher at 1.6--2.0 seconds. These are not yet final paper values because the
|
||
head-training variance has not been measured across multiple seeds.
|
||
|
||
The full per-horizon and per-class snapshot, machine-readable CSV, provenance,
|
||
and follow-up checks are in
|
||
[`results/easycom_open_stream_vs_causal_prefix_2026-08-19/`](results/easycom_open_stream_vs_causal_prefix_2026-08-19/README.md).
|
||
|
||
### EasyCom 8-second versus 20-second context
|
||
|
||
An additional single-run open-stream comparison recorded on 2026-08-23 gives
|
||
**58.51937** full-timeline mean mAP with an 8-second context budget versus
|
||
**58.316** with 20 seconds, a difference of **+0.20337** points. The aggregate
|
||
difference is small and hides a clear horizon tradeoff: 8 seconds is stronger
|
||
at 0.2--0.8 seconds, whereas 20 seconds is stronger from 1.0 through 2.0
|
||
seconds. These runs must be repeated with identical additional seeds before
|
||
choosing a context budget for the paper.
|
||
|
||
The raw 8-second console output, machine-readable comparison, and detailed
|
||
interpretation are archived in
|
||
[`results/easycom_open_stream_8s_vs_20s_2026-08-23/`](results/easycom_open_stream_8s_vs_20s_2026-08-23/README.md).
|
||
|
||
### Last-4 query-layer learned mixture
|
||
|
||
A frozen Qwen2.5-Omni-3B experiment now reads the query states from the final
|
||
four Thinker layers and learns four global softmax mixture weights before the
|
||
same `10 x 3` anticipation head. With an 8-second open-stream context, 2 FPS
|
||
video, 5 Hz outputs, and 25 head-training epochs, it reaches **73.45085** mAP
|
||
at 0.2 seconds and **60.53423** mean mAP on official test Sessions 3, 2, and 1.
|
||
Relative to the recorded single-final-layer 8-second run, this is **+1.45317**
|
||
at 0.2 seconds and **+2.01486** mean mAP, with positive gains at every horizon.
|
||
|
||
The final oldest-to-newest layer weights are
|
||
`[0.246384, 0.248327, 0.241774, 0.263515]`. Because they remain almost uniform,
|
||
the result currently demonstrates the value of multi-layer query readout, not
|
||
yet the value of learning the mixture. A fixed uniform last-4 mean is the next
|
||
required control. The complete table, machine-readable metrics, provenance,
|
||
and naming caveat are archived in
|
||
[`results/easycom_frozen_3b_8s_last4_learned_mix_2026-08-26/`](results/easycom_frozen_3b_8s_last4_learned_mix_2026-08-26/README.md).
|
||
|
||
#### Higher-capacity last-4 concat-MLP ablation
|
||
|
||
`train_turn_head.py` also supports a higher-capacity fusion control. It
|
||
flattens each `[4, 2048]` last-layer stack to `8192` dimensions, applies
|
||
`LayerNorm -> Linear(8192, 2048) -> GELU`, and then uses the unchanged
|
||
anticipation MLP. This adds about 16.8 million fusion parameters, compared
|
||
with four logits in the scalar-mixture experiment. Keep the seed and training
|
||
recipe fixed for a clean capacity ablation:
|
||
|
||
```bash
|
||
CUDA_VISIBLE_DEVICES=0 \
|
||
OMP_NUM_THREADS=4 \
|
||
MKL_NUM_THREADS=4 \
|
||
OPENBLAS_NUM_THREADS=4 \
|
||
python train_turn_head.py \
|
||
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack \
|
||
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
|
||
--output checkpoints/easycom_frozen_3b_8s_last4_concat_mlp_25ep.pth \
|
||
--sessions 4 5 6 7 8 9 10 11 12 \
|
||
--epochs 25 \
|
||
--batch-size 512 \
|
||
--learning-rate 7e-5 \
|
||
--weight-decay 5e-5 \
|
||
--warmup-epochs 10 \
|
||
--warmup-factor 0.3 \
|
||
--dropout 0.2 \
|
||
--loss lstr-mce \
|
||
--layer-fusion concat-mlp \
|
||
--expected-mode open-stream-replay \
|
||
--seed 0 \
|
||
--device cuda:0
|
||
```
|
||
|
||
Evaluate with the same dense stacked test features:
|
||
|
||
```bash
|
||
CUDA_VISIBLE_DEVICES=0 python infer_turn_head.py \
|
||
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack \
|
||
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
|
||
--checkpoint checkpoints/easycom_frozen_3b_8s_last4_concat_mlp_25ep.pth \
|
||
--output-dir results/easycom_frozen_3b_8s_last4_concat_mlp_25ep \
|
||
--expected-mode open-stream-replay \
|
||
--batch-size 1024 \
|
||
--device cuda:0
|
||
```
|
||
|
||
Treat this as a capacity ablation. If it improves, compare it against a
|
||
parameter-matched single-layer MLP and use stronger regularization or a clean
|
||
development session before attributing the gain to cross-layer fusion.
|
||
|
||
#### Four-epoch loss-structure ablation
|
||
|
||
Using the same frozen 8-second last-4 stack and concat-MLP readout, a
|
||
three-seed fixed-budget comparison tested LSTR-MCE, independent sigmoid/BCE,
|
||
and four-state softmax. All nine training sessions were used for exactly four
|
||
epochs with no development set. On the causal-valid test metric, no objective
|
||
is a decisive winner: BCE has the highest mean mAP (`60.91733`), while
|
||
four-state CE has the highest 0.2-second wearer AP (`78.72040`). Relative to
|
||
LSTR-MCE, both alternatives slightly improve the two speech classes while
|
||
reducing background AP. The full protocol, per-seed summary, seed-0 horizon
|
||
curves, and machine-readable data are archived in
|
||
[`results/easycom_loss_structure_4ep_2026-08-26/`](results/easycom_loss_structure_4ep_2026-08-26/README.md).
|
||
|
||
#### R1/R2 transition-aware head ablation
|
||
|
||
The transition target for current state t and future horizon h is one exactly
|
||
when the complete three-label activity row changes between t+h-1 and t+h.
|
||
This includes onset, offset, speaker handover, and entering or leaving overlap.
|
||
It never uses media or labels after the prediction target t+h.
|
||
|
||
Both variants share the activity MLP trunk and add a small binary 10-horizon
|
||
transition readout:
|
||
|
||
- R1 supervised uses
|
||
L = L_activity + 0.2 L_transition,
|
||
where L_transition is BCEWithLogits with positive weight 3.
|
||
- R2 expected reward uses
|
||
L = L_activity + 0.2 L_transition - 0.1 E[R_transition].
|
||
For transition probability p and target z, correctness reward is
|
||
z p + (1-z)(1-p). Its expectation is computed exactly, weighted with the
|
||
same positive weight 3, and normalized by total weight. There is no sampled
|
||
policy gradient, so this is the low-variance control rather than a claim of
|
||
interactive reinforcement learning.
|
||
|
||
Run the fixed four-epoch, no-development-set comparison with the same schedule
|
||
as the loss-structure ablation:
|
||
|
||
~~~bash
|
||
FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
|
||
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
|
||
|
||
for seed in 0 1 2
|
||
do
|
||
CUDA_VISIBLE_DEVICES=0 \
|
||
OMP_NUM_THREADS=4 \
|
||
MKL_NUM_THREADS=4 \
|
||
OPENBLAS_NUM_THREADS=4 \
|
||
python train_turn_head.py \
|
||
--feature-root "$FEATURE_ROOT" \
|
||
--target-root "$TARGET_ROOT" \
|
||
--output "checkpoints/easycom-transition-r1-4ep-seed-$seed.pth" \
|
||
--sessions 4 5 6 7 8 9 10 11 12 \
|
||
--epochs 4 \
|
||
--batch-size 512 \
|
||
--learning-rate 7e-5 \
|
||
--weight-decay 5e-5 \
|
||
--warmup-epochs 1.6 \
|
||
--warmup-factor 0.3 \
|
||
--dropout 0.2 \
|
||
--loss lstr-mce \
|
||
--transition-objective r1-supervised \
|
||
--transition-loss-weight 0.2 \
|
||
--transition-pos-weight 3 \
|
||
--layer-fusion concat-mlp \
|
||
--expected-mode open-stream-replay \
|
||
--seed "$seed" \
|
||
--device cuda:0
|
||
|
||
CUDA_VISIBLE_DEVICES=0 \
|
||
OMP_NUM_THREADS=4 \
|
||
MKL_NUM_THREADS=4 \
|
||
OPENBLAS_NUM_THREADS=4 \
|
||
python train_turn_head.py \
|
||
--feature-root "$FEATURE_ROOT" \
|
||
--target-root "$TARGET_ROOT" \
|
||
--output "checkpoints/easycom-transition-r2-4ep-seed-$seed.pth" \
|
||
--sessions 4 5 6 7 8 9 10 11 12 \
|
||
--epochs 4 \
|
||
--batch-size 512 \
|
||
--learning-rate 7e-5 \
|
||
--weight-decay 5e-5 \
|
||
--warmup-epochs 1.6 \
|
||
--warmup-factor 0.3 \
|
||
--dropout 0.2 \
|
||
--loss lstr-mce \
|
||
--transition-objective r2-expected-reward \
|
||
--transition-loss-weight 0.2 \
|
||
--transition-pos-weight 3 \
|
||
--transition-reward-weight 0.1 \
|
||
--layer-fusion concat-mlp \
|
||
--expected-mode open-stream-replay \
|
||
--seed "$seed" \
|
||
--device cuda:0
|
||
done
|
||
~~~
|
||
|
||
Evaluate all six checkpoints. In addition to the unchanged EgoSpeak activity
|
||
metrics, inference writes transition_pred arrays and prints transition AP,
|
||
precision, recall, and F1 at threshold 0.5:
|
||
|
||
~~~bash
|
||
for variant in r1 r2
|
||
do
|
||
for seed in 0 1 2
|
||
do
|
||
CUDA_VISIBLE_DEVICES=0 python infer_turn_head.py \
|
||
--feature-root "$FEATURE_ROOT" \
|
||
--target-root "$TARGET_ROOT" \
|
||
--checkpoint "checkpoints/easycom-transition-$variant-4ep-seed-$seed.pth" \
|
||
--output-dir "results/easycom-transition-$variant-4ep-seed-$seed" \
|
||
--expected-mode open-stream-replay \
|
||
--batch-size 1024 \
|
||
--device cuda:0
|
||
done
|
||
done
|
||
~~~
|
||
|
||
Compare R1 and R2 against the existing three-seed LSTR-MCE baseline under the
|
||
same four-epoch budget. The primary paper outcome remains activity mAP; the
|
||
transition metrics diagnose whether an activity improvement is genuinely
|
||
associated with better boundary prediction.
|
||
|
||
The preliminary seed-0/1 results are archived in
|
||
[results/easycom_transition_auxiliary_4ep_2026-08-27/](results/easycom_transition_auxiliary_4ep_2026-08-27/README.md).
|
||
Neither R1 nor R2 establishes a meaningful activity-mAP gain. At 0.2 seconds,
|
||
transition prevalence is 11.21718%, while transition AP is approximately
|
||
15.3%, indicating weak but non-random ranking. R2 improves activity mean mAP
|
||
over R1 by only 0.01162 points and transition AP by 0.16660 points. Seed 2 is
|
||
still required before freezing this as a complete null ablation.
|
||
|
||
## Ego4D local-feature ablation family
|
||
|
||
To test whether the recent-audio / per-person-visual gains transfer from EasyCom,
|
||
see [the Ego4D local-feature experiment guide](docs/ego4d_local_baselines.md).
|
||
It reuses existing global Omni features and the EgoSpeak manifest, extracts local
|
||
A/V directly from raw clips, and runs **G / G+A / G+V / G+A+V** for four epochs,
|
||
without a dev set. `run_ego4d_local_baselines.py` preflights all train/test files
|
||
and encoder contracts before training; the summary reports matched-seed deltas,
|
||
including AV minus audio. No new accuracy claim is implied by adding the code.
|
||
|
||
The global `train/<uid>.pt` and `test/<uid>.pt` layout is now read directly by
|
||
extraction, training and inference; no symlink index is needed.
|
||
`extract_ego4d_local_multigpu.py` distributes full clips across eight independent
|
||
GPU workers, with per-GPU logs, validated resume, and foreground supervision.
|
||
See the guide's L20 commands for the `/data1/yfzx/wangx` paths.
|
||
|
||
For **encoder-only A / V / AV**, without Thinker states, see
|
||
[the Ego4D guide](docs/ego4d_encoder_only.md) or
|
||
[the EasyCom-first guide](docs/easycom_encoder_only.md).
|
||
`prepare_encoder_only_features.py` packs complete local vectors and validity
|
||
bits on CPU (pass `--dataset easycom` for EasyCom; default remains Ego4D).
|
||
`run_ego4d_encoder_only.py` / `run_easycom_encoder_only.py` train/evaluate
|
||
four-epoch heads with no dev set. This native-concatenation baseline is not
|
||
capacity- or context-matched to the Thinker baseline.
|
||
|
||
## Current Ego4D context-length results
|
||
|
||
The following single-seed snapshot was recorded on 2026-08-22. Both variants
|
||
use open-stream replay, 2 FPS Decord video, 5 Hz predictions, the same frozen
|
||
Qwen2.5-Omni-3B Thinker and independently trained `10 x 3` heads. They differ
|
||
only in the rolling causal-memory budget: 8 seconds versus 20 seconds. Both
|
||
runs use the same EgoSpeak Ego4D train/test manifest and test timeline.
|
||
|
||
| Context | EgoSpeak full-timeline mean mAP | Strict causal mean mAP |
|
||
|---|---:|---:|
|
||
| 8 seconds | **65.5943** | **65.6920** |
|
||
| 20 seconds | 63.1408 | 63.2298 |
|
||
| 20s minus 8s | -2.4535 | -2.4622 |
|
||
|
||
The strict-causal per-horizon comparison is:
|
||
|
||
| Horizon | 8s mAP | 20s mAP | 20s minus 8s |
|
||
|---:|---:|---:|---:|
|
||
| 0.2s | **74.2213** | 71.7403 | -2.4810 |
|
||
| 0.4s | **71.0180** | 68.5945 | -2.4234 |
|
||
| 0.6s | **68.3746** | 65.9287 | -2.4459 |
|
||
| 0.8s | **66.3190** | 63.8606 | -2.4584 |
|
||
| 1.0s | **64.7470** | 62.3329 | -2.4140 |
|
||
| 1.2s | **63.6416** | 61.2290 | -2.4126 |
|
||
| 1.4s | **62.8697** | 60.4035 | -2.4662 |
|
||
| 1.6s | **62.3142** | 59.7870 | -2.5272 |
|
||
| 1.8s | **61.8705** | 59.3678 | -2.5027 |
|
||
| 2.0s | **61.5444** | 59.0541 | -2.4903 |
|
||
| Mean | **65.6920** | 63.2298 | -2.4622 |
|
||
|
||
For the exact full-timeline protocol, averaging class AP over the ten
|
||
horizons gives:
|
||
|
||
| Class | 8s mean AP | 20s mean AP | 20s minus 8s |
|
||
|---|---:|---:|---:|
|
||
| Background | **69.5063** | 67.2409 | -2.2653 |
|
||
| Camera wearer speaking | **59.4005** | 56.5784 | -2.8222 |
|
||
| Other people speaking | **67.8760** | 65.6030 | -2.2730 |
|
||
|
||
The 8-second representation is better at every horizon and for every class.
|
||
The approximately 2.4--2.5 point gap is nearly constant across anticipation
|
||
horizons, while the largest class degradation is for camera-wearer speech.
|
||
This pattern does not support a long-horizon benefit from the additional 12
|
||
seconds of context. A plausible hypothesis is that excessive history dilutes
|
||
locally predictive turn-taking cues in the final frozen Thinker state, but the
|
||
single-seed comparison cannot establish that mechanism. Both context settings
|
||
must be repeated with the same additional seeds before treating the difference
|
||
as a final paper result.
|
||
|
||
## Why Omni-only instead of Omni + RNN
|
||
|
||
EgoSpeak separately extracts visual and audio features and then uses GRU,
|
||
LSTR, or Mamba for temporal modeling:
|
||
|
||
```text
|
||
ResNet50 video feature + wav2vec2 audio feature -> temporal model -> prediction
|
||
```
|
||
|
||
Qwen2.5-Omni already contains:
|
||
|
||
- audio and visual encoders;
|
||
- joint audio-video temporal positions;
|
||
- a causal decoder-only Thinker with KV cache;
|
||
- pretrained acoustic, visual, and semantic knowledge.
|
||
|
||
Adding a GRU after a continuously processed Omni sequence would model history
|
||
twice and would make it harder to attribute an improvement to Omni. The primary
|
||
comparison is therefore:
|
||
|
||
```text
|
||
EgoSpeak: ResNet50 + wav2vec2 + GRU
|
||
Ours: Qwen2.5-Omni Thinker + small prediction head
|
||
```
|
||
|
||
An Omni-feature-plus-GRU model can be kept as an ablation, but it is not the
|
||
main architecture.
|
||
|
||
## EgoSpeak contract that this project reproduces
|
||
|
||
The inspected reference repository is `/Volumes/760P/Code/EgoSpeak`.
|
||
It contains no Qwen or Omni implementation and no raw-media feature extraction
|
||
scripts. It distributes precomputed arrays instead:
|
||
|
||
```text
|
||
DATASET/
|
||
|- rgb_kinetics_resnet50/ # [T, 2048]
|
||
|- audio_wav2vec2/ # [T, 5120]
|
||
`- target_perframe/ # [T, 3]
|
||
```
|
||
|
||
The useful part for this project is the exact task interface.
|
||
|
||
### Time and classes
|
||
|
||
Both EasyCom and the EgoSpeak `EGO4DSHUFFLE` split use **5 FPS**, so one task
|
||
timestep is **0.2 seconds**. The three output columns are:
|
||
|
||
```text
|
||
0: background
|
||
1: camera wearer speaking
|
||
2: other people speaking
|
||
```
|
||
|
||
They are not strictly one-hot. Background is exclusive, but simultaneous
|
||
camera-wearer and other-person speech is stored as `[0, 1, 1]`. The inspected
|
||
EasyCom features contain 7,711 such overlap steps. Preserve the full
|
||
multi-positive target rather than applying `argmax`.
|
||
|
||
Relevant EgoSpeak files:
|
||
|
||
- `LSTR/data/data_info.json`: FPS, classes, and session splits;
|
||
- `MiniROAD/configs/EasyCom/miniroad_aa_easycom.yaml`: 128-step training
|
||
windows, stride 4, three classes, and anticipation length 10;
|
||
- `MiniROAD/datasets/dataset.py`: feature loading and future-label indexing;
|
||
- `MiniROAD/criterions/loss.py`: anticipation loss;
|
||
- `MiniROAD/trainer/eval.py`: per-horizon AP and mean anticipation mAP.
|
||
|
||
### Future target alignment
|
||
|
||
For current task index `i`, the prediction target is:
|
||
|
||
```python
|
||
target[i + 1 : i + 11]
|
||
```
|
||
|
||
Thus the ten output rows represent:
|
||
|
||
```text
|
||
t+0.2, t+0.4, t+0.6, ..., t+2.0 seconds
|
||
```
|
||
|
||
If `states[i]` is verified to correspond to EgoSpeak task index `i`, the
|
||
offline head-training tensors should be constructed as follows:
|
||
|
||
```python
|
||
target = target_perframe # [T, 3], multi-positive
|
||
usable = min(len(states), len(target)) - 10
|
||
|
||
x = states[:usable] # [N, 2048]
|
||
y = stack(
|
||
[target[h : h + usable] for h in range(1, 11)],
|
||
axis=1,
|
||
) # [N, 10, 3]
|
||
```
|
||
|
||
Do not silently assume that the first saved Omni state matches label index
|
||
zero. EgoSpeak does not publish the script that generated its original
|
||
ResNet/wav2vec2 features, so the exact zero-time convention is not documented.
|
||
Before training, inspect a short clip with known speech boundaries and test the
|
||
candidate mappings `states[i] -> target[i]` and `states[i] -> target[i+1]`.
|
||
Store the selected offset in every feature file/checkpoint.
|
||
|
||
### Splits
|
||
|
||
Use the published split, not a random frame split.
|
||
|
||
EasyCom:
|
||
|
||
- train, preserving EgoSpeak's list: `Session_7`, `Session_11`, `Session_10`,
|
||
`Session_6`, `Session_9`, `Session_4`, `Session_8`, `Session_5`, `Session_12`;
|
||
- test, preserving EgoSpeak's list: `Session_3`, `Session_2`, `Session_1`.
|
||
|
||
This is the **EgoSpeak comparison split**. The EasyCom dataset repository
|
||
itself recommends a different train/validation/test split; do not substitute
|
||
that split when reproducing EgoSpeak Table 2.
|
||
|
||
Ego4D Audio-Visual Diarization:
|
||
|
||
- use the `EGO4DSHUFFLE` key;
|
||
- 346 training clips;
|
||
- 87 test clips;
|
||
- each clip is approximately five minutes.
|
||
|
||
Never mix timesteps from the same session across training and validation.
|
||
|
||
## EasyCom raw-media layout and manifest
|
||
|
||
The extracted release used here has this relevant structure:
|
||
|
||
```text
|
||
extracted_easycom/
|
||
`- Main/
|
||
|- Video_Compressed/
|
||
| |- Session_1/
|
||
| | |- 00-00-000.mp4
|
||
| | |- 01-00-000.mp4
|
||
| | `- ...
|
||
| `- Session_2 ... Session_12/
|
||
|- Glasses_Microphone_Array_Audio/
|
||
|- Close_Microphone_Audio/
|
||
|- Speech_Transcriptions/
|
||
`- Metadata/
|
||
```
|
||
|
||
`Video_Compressed` is the AR-glasses wearer's egocentric stream with embedded
|
||
stereo AAC audio. Each session is divided into approximately one-minute clips;
|
||
the `mm-ss-SSS` filename is the clip's start time on the original recording
|
||
timeline. Some intervals are absent because EasyCom redacted one-minute
|
||
segments. These raw time gaps are expected.
|
||
|
||
EgoSpeak does not preserve those empty intervals. Its appendix states that it
|
||
crops every EasyCom clip to retain only the 1920-pixel-wide video region and
|
||
then merges the retained clips into one dense video per session. Therefore the
|
||
EgoSpeak-compatible label time for a clip is its cumulative position in the
|
||
concatenation, not the `mm-ss-SSS` source time. The manifest stores both.
|
||
|
||
Build a raw-video manifest and, when the EgoSpeak labels are available, check
|
||
their duration at 5 Hz:
|
||
|
||
```bash
|
||
python easycom_manifest.py \
|
||
--dataset-root /data1/yfzx/wangx/Datasets/easycom/extracted_easycom \
|
||
--target-root /path/to/EasyCom/target_perframe \
|
||
--output manifests/easycom.json
|
||
```
|
||
|
||
Omit `--target-root` for the first media-only inventory. The manifest reports
|
||
raw-source discontinuities as informational and computes each clip's dense
|
||
EgoSpeak concatenation offset and corresponding 5 Hz target index. For the
|
||
audio-only ablation, use the same embedded glasses audio as A+V (extract it to
|
||
mono WAV if needed); do not switch to close microphones, because that changes
|
||
the sensor and makes the modality ablation unfair.
|
||
|
||
Audit downloaded EgoSpeak arrays, and optionally compare every session length
|
||
against the raw manifest:
|
||
|
||
```bash
|
||
python audit_egospeak_features.py \
|
||
--feature-root /path/to/EgoSpeak/dataset/EasyCom \
|
||
--raw-manifest manifests/easycom.json \
|
||
--output reports/egospeak_easycom_audit.json
|
||
```
|
||
|
||
The inspected release contains 95,453 synchronized steps across all 12
|
||
sessions: 5.302944 hours at 5 Hz. Every session has identical RGB, flow, audio,
|
||
and target length. Shapes are `[T,2048]`, `[T,2048]`, `[T,5120]`, and `[T,3]`.
|
||
|
||
## Recommended comparison matrix
|
||
|
||
| Experiment | Visual sampling | Audio | Output rate |
|
||
|---|---:|---|---:|
|
||
| EgoSpeak official A+V | 5 Hz ResNet features | 5 Hz wav2vec2 features | 5 Hz |
|
||
| Omni audio-only | none | raw glasses-wearer waveform | 5 Hz |
|
||
| Omni A+V primary | 2 FPS | raw embedded waveform | 5 Hz |
|
||
| Omni A+V FPS ablation | 5 FPS | raw embedded waveform | 5 Hz |
|
||
|
||
Two FPS is the primary Omni setting because it matches Qwen's default video
|
||
sampling and permits longer context. Five FPS is the visual-rate ablation.
|
||
Input feature rates need not match EgoSpeak; the split, causal prediction
|
||
times, three classes, ten target horizons, and evaluator must match exactly.
|
||
|
||
## Prepare an EgoSpeak-compatible EasyCom session
|
||
|
||
Do not pass the raw one-minute files independently to Omni and do not preserve
|
||
the gaps implied by their filenames. EgoSpeak concatenates the retained clips
|
||
into a dense session timeline and crops the participant-ID banner on the left.
|
||
|
||
`prepare_easycom_session.py` reproduces this policy and creates videos whose
|
||
stored frames already lie on a fixed session-global 2 FPS or 5 FPS grid. This
|
||
prevents Qwen's per-window video loader from choosing a new adaptive set of
|
||
frames for every overlapping prefix.
|
||
|
||
Session 1 has been checked against the local release:
|
||
|
||
```text
|
||
24 retained clips
|
||
27,768 source frames at 20 FPS
|
||
/ 4 frames per task step
|
||
= 6,942 steps at 5 Hz
|
||
= target_perframe/Session_1.npy length exactly
|
||
```
|
||
|
||
Preview the commands and validate frames against the target without encoding:
|
||
|
||
```bash
|
||
python prepare_easycom_session.py \
|
||
--session-dir /Volumes/MX500B/Code/easycom/Main/Video_Compressed/Session_1 \
|
||
--output-dir prepared_easycom/Session_1 \
|
||
--target /Volumes/MX500B/Code/EgoSpeak/dataset/EasyCom/target_perframe/Session_1.npy \
|
||
--fps 2 5 \
|
||
--dry-run
|
||
```
|
||
|
||
Remove `--dry-run` to run FFmpeg. The outputs are:
|
||
|
||
```text
|
||
Session_1_2fps.mp4 primary Omni A+V input
|
||
Session_1_5fps.mp4 visual-rate ablation
|
||
Session_1.json clip order, exact T, crop and sampling provenance
|
||
Session_1.ffconcat reproducible dense concatenation order
|
||
```
|
||
|
||
The target length must equal the number of complete 5 Hz intervals,
|
||
`floor(sum(source_frames) / 4)`. If a session ends with one to three residual
|
||
20 Hz frames, the target-derived duration discards only that incomplete final
|
||
interval; the discarded frame count is recorded in the sidecar.
|
||
|
||
Each clip's decoded video and audio are trimmed to the exact duration implied
|
||
by its video frame count before concatenation. This prevents AAC priming/tail
|
||
padding from accumulating at the 24 clip boundaries and causing progressive
|
||
audio/video drift. The session-global visual grid is padded defensively at EOF
|
||
and then trimmed by frame index, so it contains exactly
|
||
`ceil(target_steps / 5 * video_fps)` frames on every FFmpeg version.
|
||
|
||
On the remote L20 machine, use its own dataset paths. For example:
|
||
|
||
```bash
|
||
python prepare_easycom_session.py \
|
||
--session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_1 \
|
||
--output-dir /data1/yfzx/wangx/Datasets/easycom/omni_prepared/Session_1 \
|
||
--target /path/to/EgoSpeak/dataset/EasyCom/target_perframe/Session_1.npy \
|
||
--fps 2 5
|
||
```
|
||
|
||
This is the command that performs the potentially long FFmpeg processing; it
|
||
has not been run by the code authoring workflow.
|
||
|
||
If encoding completed but a later validation/metadata step failed, update the
|
||
script and reuse the existing MP4 without encoding it again:
|
||
|
||
```bash
|
||
python prepare_easycom_session.py ... --fps 2 --reuse-existing
|
||
```
|
||
|
||
## What Qwen's native 2-second streaming design means
|
||
|
||
Qwen2.5-Omni was designed for **block-wise multimodal prefill**. Its
|
||
[technical report](https://arxiv.org/html/2503.20215) states that the audio
|
||
and visual encoders are modified for block-wise processing and that audio
|
||
self-attention is performed in independent two-second blocks. The released
|
||
[3B configuration](https://huggingface.co/Qwen/Qwen2.5-Omni-3B/raw/main/config.json)
|
||
uses `seconds_per_chunk=2`, `position_id_per_seconds=25`, and audio
|
||
`n_window=100`.
|
||
|
||
Conceptually, a native online input is organized as follows:
|
||
|
||
```text
|
||
open one user/media turn
|
||
encode/append AV [0,2] -> commit Thinker KV
|
||
encode/append AV [2,4] -> commit Thinker KV
|
||
encode/append AV [4,6] -> commit Thinker KV
|
||
...
|
||
close the media turn -> generate text and/or speech
|
||
```
|
||
|
||
Within each temporal group, video tokens precede the aligned audio tokens.
|
||
TMRoPE assigns 25 temporal positions per second, or 50 positions per native
|
||
two-second group, so audio and video share a physical-time coordinate system.
|
||
Completed blocks are appended to one causal Thinker sequence; the long-range
|
||
history is modeled by the Thinker rather than by cross-block full attention in
|
||
the modality encoders.
|
||
|
||
This architecture is block-causal, not frame-causal. Attention inside an
|
||
audio block is non-causal. It is therefore invalid to encode the complete
|
||
`[0,2]` block and claim that its representation at 0.2 seconds saw no future:
|
||
|
||
```text
|
||
encode complete [0,2] AV -> read state at 0.2 s # future leakage
|
||
```
|
||
|
||
For this project's 5 Hz targets, the current incomplete block must instead be
|
||
encoded only through the prediction time. These 0.2-second partial-block
|
||
readouts are an extension made for EgoSpeak; the Qwen paper does not define a
|
||
canonical hidden state every 200 ms.
|
||
|
||
The public Transformers and local Qwen web-demo APIs normally receive a
|
||
completed media object and call the processor/model once. They retain the
|
||
model's block-wise masks and token organization, but do not expose the full
|
||
persistent live-session implementation used by the hosted real-time product.
|
||
|
||
## Native Qwen streaming versus `causal-prefix`
|
||
|
||
The two extraction modes are both causal with respect to raw observations,
|
||
but they define different model inputs. `causal-prefix` is a sliding-window
|
||
reference invented for this experiment; it is not Qwen's native streaming
|
||
execution model.
|
||
|
||
| Property | `causal-prefix` | Native Qwen streaming idea |
|
||
|---|---|---|
|
||
| Session | New inference at every 0.2 s | One open media turn |
|
||
| Media organization | One complete `[max(0,t-M),t]` item | Fixed source-aligned 2 s blocks |
|
||
| Modality encoding | Re-encode the complete window | Encode/commit each completed block once |
|
||
| Thinker KV | Start empty for every state | Append to persistent KV |
|
||
| TMRoPE | Recomputed from each window origin | Advances with the stream |
|
||
| History | Exact rolling `M` seconds | Accumulates until a serving/context policy intervenes |
|
||
| 5 Hz readout | Close/read a new turn each time | Not specified by the original model |
|
||
|
||
The difference becomes especially important after a sliding window fills. At
|
||
`t=20.8` with `M=20`, `causal-prefix` constructs one new media item covering:
|
||
|
||
```text
|
||
[0.8,20.8]
|
||
-> encoder blocks [0.8,2.8], [2.8,4.8], ..., [18.8,20.8]
|
||
```
|
||
|
||
At `t=21.0`, every encoder boundary moves again:
|
||
|
||
```text
|
||
[1.0,21.0]
|
||
-> encoder blocks [1.0,3.0], [3.0,5.0], ..., [19.0,21.0]
|
||
```
|
||
|
||
Native streaming keeps the source-aligned partition instead:
|
||
|
||
```text
|
||
[0,2], [2,4], ..., [18,20], current partial [20,20.8]
|
||
```
|
||
|
||
Because Qwen's audio encoder uses block-local non-causal attention, moving a
|
||
block boundary changes the representation even when most raw samples are the
|
||
same. Window-relative TMRoPE, convolution/padding boundaries, chat closure,
|
||
and BF16/attention execution add further differences. This is why exact KV
|
||
replay can be numerically identical to its own from-scratch replay while still
|
||
differing from `causal-prefix`; the modes are evaluating different token and
|
||
encoder histories.
|
||
|
||
Use the modes according to the scientific question:
|
||
|
||
- use `open-stream-replay` for the primary online experiment, training, and
|
||
deployment-aligned evaluation;
|
||
- use `causal-prefix` as an expensive continuous-window reference and
|
||
short-clip ablation;
|
||
- never train a head on one representation and deploy it on the other without
|
||
retraining, because their feature distributions are not interchangeable.
|
||
|
||
## Causal 5 Hz extraction modes
|
||
|
||
`streaming_turn_model.py` implements two modes.
|
||
|
||
### 1. `causal-prefix`: continuous-window reference
|
||
|
||
At every 200 ms, independently process the causal context ending at the
|
||
current time:
|
||
|
||
```text
|
||
[max(0, t-20 s), t] AV -> Omni -> q_t
|
||
```
|
||
|
||
Properties:
|
||
|
||
- strict no-future input;
|
||
- one continuous media window per prediction;
|
||
- no cross-forward cache;
|
||
- very expensive because overlapping context is recomputed;
|
||
- useful for checking continuous sliding-window semantics and causality;
|
||
- not the native Qwen block-streaming execution model.
|
||
|
||
Example:
|
||
|
||
```bash
|
||
python streaming_turn_model.py \
|
||
--video /path/to/easycom_session.mp4 \
|
||
--use-audio-in-video \
|
||
--mode causal-prefix \
|
||
--start 0 \
|
||
--end 20 \
|
||
--memory-seconds 20 \
|
||
--output features/easycom_20s_prefix.pt
|
||
```
|
||
|
||
### 2. `open-stream-replay`: recommended efficient mode
|
||
|
||
This mode keeps one user/media turn open across the whole bounded-memory
|
||
segment. The first block contributes the chat and AV opening markers once;
|
||
later completed two-second blocks contribute only interleaved audio/video
|
||
payload tokens:
|
||
|
||
```text
|
||
open AV prefix
|
||
completed [0,2] payload -> commit KV
|
||
current [2,2.2] payload -> temporary branch -> close/read q_t
|
||
current [2,2.4] payload -> temporary branch -> close/read q_t
|
||
...
|
||
completed [2,4] payload -> commit KV
|
||
AV/chat closure -> never committed; readout branch only
|
||
```
|
||
|
||
This matches the core Qwen design choices: fixed two-second AV groups, one
|
||
open media turn, completed-block commits, and incremental Thinker KV. It makes
|
||
two practical changes for this dataset:
|
||
|
||
1. the current 0--2 second partial block is evaluated speculatively every
|
||
0.2 seconds and discarded unless it reaches a complete block boundary;
|
||
2. the KV history is bounded and rebuilt from recent complete blocks rather
|
||
than accumulating for an entire multi-minute session. Complete-block AV
|
||
embeddings are retained in a bounded ring, so rolling rebuilds rerun only
|
||
the Thinker and do not rerun the processor, audio tower, or video tower.
|
||
|
||
It should therefore be described as **bounded Qwen-style open streaming**, not
|
||
as bit-for-bit reproduction of an unbounded official serving session.
|
||
|
||
The audio encoder accepts a shorter final chunk with padding and an
|
||
actual-length mask. For the two-frame visual patch requirement, strict-grid
|
||
mode duplicates only the most recent past frame when necessary.
|
||
|
||
Run a four-second validation before using it for extraction:
|
||
|
||
```bash
|
||
python streaming_turn_model.py \
|
||
--model-id /path/to/Qwen2.5-Omni-3B \
|
||
--video prepared_easycom/Session_1/Session_1_2fps.mp4 \
|
||
--use-audio-in-video \
|
||
--mode open-stream-replay \
|
||
--start 0 --end 4 \
|
||
--video-fps 2 \
|
||
--strict-fixed-video-grid \
|
||
--verify-every 1 \
|
||
--verify-continuous-every 1 \
|
||
--flash-attention-2 \
|
||
--output features/Session_1_open_first_4s.pt
|
||
```
|
||
|
||
`cache_replay` must be identical to replaying the same open stream from an
|
||
empty cache. `continuous_window_semantics` compares it with one normal
|
||
continuous Qwen AV input. Do not assume equivalence: the modality encoders are
|
||
still invoked per tail block in the efficient path, so this numerical check is
|
||
required even though artificial user-turn boundaries have been removed.
|
||
|
||
## Whole-session prediction and the 20-second limit
|
||
|
||
Twenty seconds is a practical memory/context setting for the 3090, not the
|
||
maximum video duration and not a limit specified by Qwen. The original
|
||
append-only idea eventually reaches both linear KV-memory growth and the
|
||
Thinker's 32,768-token context limit, so an unbounded multi-minute stream is
|
||
not practical. The complete session is processed sequentially and a
|
||
prediction is emitted every 0.2 seconds.
|
||
|
||
`open-stream-replay` uses blockwise rolling memory. With a 20-second
|
||
budget it rebuilds the open cache from the latest nine completed blocks
|
||
(18 seconds) when a new block starts, then adds the current 0--2 second tail.
|
||
Its effective history therefore varies from 18 to 20 seconds and never drops
|
||
to an empty 0.2-second context at a 20-second boundary. The nine blocks'
|
||
prepared AV embeddings and local TMRoPE coordinates are cached once; strict
|
||
window replay still recomputes the Thinker KV so retained blocks no longer
|
||
carry indirect influence from observations that fell out of the window.
|
||
|
||
For a short native-stream fidelity check, set `--memory-seconds` to at least
|
||
the tested clip duration. For example, the options
|
||
`--start 0 --end 40 --memory-seconds 40` prevent a rolling rebuild during that
|
||
40-second clip.
|
||
Do not extend this policy to complete EasyCom/Ego4D sessions: KV memory grows
|
||
linearly, and this implementation clones the cache for speculative partial
|
||
blocks and readout branches.
|
||
|
||
For the paper experiment, drive the output length from the preparation sidecar
|
||
rather than a floating MP4 duration:
|
||
|
||
```bash
|
||
python streaming_turn_model.py \
|
||
--session-metadata prepared_easycom/Session_1/Session_1.json \
|
||
--use-audio-in-video \
|
||
--mode open-stream-replay \
|
||
--memory-seconds 20 \
|
||
--video-fps 2 \
|
||
--strict-fixed-video-grid \
|
||
--verify-every 25 \
|
||
--verify-continuous-every 25 \
|
||
--output features/Session_1_omni_2fps.pt
|
||
```
|
||
|
||
The sidecar selects the matching fixed-FPS MP4 and sets `N` to its exact
|
||
`num_steps`. Alternatively pass `--video ... --target ...` explicitly. A raw
|
||
20 FPS MP4 with `--video-fps 2` is rejected when
|
||
`--strict-fixed-video-grid` is enabled.
|
||
|
||
In strict A+V mode the extractor does not delegate temporal frame selection to
|
||
the generic video utility. It reads the already-fixed MP4 grid explicitly and
|
||
assigns both video frames and audio samples with half-open intervals
|
||
`[start, end)`. Thus a frame exactly on a two-second boundary belongs only to
|
||
the following block. It duplicates the most recent available frame when the
|
||
visual tower needs an even frame count. Audio is loaded once at 16 kHz and
|
||
sliced at the same causal boundary. The processor still receives
|
||
`use_audio_in_video=True`, so Qwen's audio/video token interleaving is retained.
|
||
Long runs use `tqdm` and update the progress bar after every completed state,
|
||
showing percentage, media time, elapsed time, ETA, and throughput. Pass
|
||
`--no-progress` to disable it. The Thinker-only path also suppresses Qwen's
|
||
irrelevant speech-output system prompt warning.
|
||
|
||
Before a full session, run only the first four seconds and inspect both check
|
||
types:
|
||
|
||
```bash
|
||
python streaming_turn_model.py \
|
||
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
||
--video /data1/yfzx/wangx/Datasets/easycom/omni_prepared/Session_1/Session_1_2fps.mp4 \
|
||
--use-audio-in-video \
|
||
--mode open-stream-replay \
|
||
--start 0 --end 4 \
|
||
--video-fps 2 \
|
||
--strict-fixed-video-grid \
|
||
--verify-every 1 \
|
||
--verify-continuous-every 1 \
|
||
--flash-attention-2 \
|
||
--output features/Session_1_first_4s_check.pt
|
||
```
|
||
|
||
Do not launch all sessions unless `cache_replay` cosine is nearly 1. For
|
||
`open-stream-replay`, a materially lower `continuous_window_semantics` cosine
|
||
means that the custom payload stitching is not equivalent to one continuous
|
||
Qwen input; it must not be dismissed as a media-turn-boundary effect because
|
||
this mode intentionally keeps a single media turn open.
|
||
|
||
Video can be sampled at 2 FPS while predictions remain at 5 Hz. Audio and the
|
||
causal boundary update every 200 ms; consecutive prediction times may use the
|
||
same most-recent visual frame. A frame with timestamp later than `t` must never
|
||
be included.
|
||
|
||
## Direct extraction without a prepared 2 FPS MP4
|
||
|
||
`streaming_turn_model_directly.py` removes the offline video-transcoding
|
||
requirement. It reads original frame PTS with Decord, constructs a global
|
||
2 FPS lattice, maps every lattice point to the latest source frame whose PTS
|
||
is not later than that point, and passes explicit frame lists to Qwen. Audio is
|
||
decoded once to 16-kHz mono and sliced at exact causal boundaries. Therefore
|
||
no FFmpeg `fps` filter or second H.264/AAC encode can alter the experiment.
|
||
|
||
Validate one Ego4D clip without loading Qwen:
|
||
|
||
```bash
|
||
python streaming_turn_model_directly.py \
|
||
--video /data/wangx/Datasets/ego4d/v1/clips/CLIP_UID.mp4 \
|
||
--target /data/wangx/Datasets/ego4d/ego4d_target_frame/CLIP_UID.npy \
|
||
--video-fps 2 \
|
||
--media-only
|
||
```
|
||
|
||
Run a short native persistent-KV check:
|
||
|
||
```bash
|
||
python streaming_turn_model_directly.py \
|
||
--model-id /data/wangx/Models/Qwen2.5-Omni-3B \
|
||
--video /data/wangx/Datasets/ego4d/v1/clips/CLIP_UID.mp4 \
|
||
--end 40 \
|
||
--mode open-stream-replay \
|
||
--memory-seconds 0 \
|
||
--video-fps 2 \
|
||
--flash-attention-2 \
|
||
--device cuda:0 \
|
||
--output features/direct/CLIP_UID_first_40s.pt
|
||
```
|
||
|
||
Here `--memory-seconds 0` means that completed-block Thinker KV persists for
|
||
the entire requested interval. Do not use this setting blindly for complete
|
||
multi-minute recordings: it has linear KV growth and remains subject to the
|
||
Thinker's context limit. A full bounded-memory dataset run should explicitly
|
||
use the experimental budget, for example `--memory-seconds 20`.
|
||
|
||
EasyCom can be read directly as the dense concatenation of retained clips;
|
||
filename timecode gaps are ignored and the rightmost 1920x1080 region is kept:
|
||
|
||
```bash
|
||
python streaming_turn_model_directly.py \
|
||
--model-id /path/to/Qwen2.5-Omni-3B \
|
||
--easycom-session-dir /path/to/Main/Video_Compressed/Session_1 \
|
||
--target target_frame/Session_1.npy \
|
||
--mode open-stream-replay \
|
||
--memory-seconds 20 \
|
||
--video-fps 2 \
|
||
--flash-attention-2 \
|
||
--device cuda:0 \
|
||
--output features/direct/Session_1.pt
|
||
```
|
||
|
||
The direct reader defaults to Decord. On NVIDIA systems, install
|
||
`PyNvVideoCodec` and select the NVDEC backend explicitly:
|
||
|
||
```bash
|
||
python -m pip install PyNvVideoCodec
|
||
|
||
CUDA_VISIBLE_DEVICES=0 python streaming_turn_model_directly.py \
|
||
--model-id /path/to/Qwen2.5-Omni-3B \
|
||
--easycom-session-dir /path/to/Main/Video_Compressed/Session_1 \
|
||
--end 24 \
|
||
--mode open-stream-replay \
|
||
--memory-seconds 20 \
|
||
--video-fps 2 \
|
||
--frame-cache-size 64 \
|
||
--decoder pynvcodec \
|
||
--decoder-gpu-id 0 \
|
||
--flash-attention-2 \
|
||
--device cuda:0 \
|
||
--output features/direct/Session_1_24s_pynvcodec.pt
|
||
```
|
||
|
||
The NVDEC backend uses ffprobe for exact display-order PTS indexing, one
|
||
reconfigurable `SimpleDecoder` for pixel decode, and the existing ffmpeg PCM
|
||
audio path. `--decoder-gpu-id` is relative to `CUDA_VISIBLE_DEVICES`, so it is
|
||
normally `0` even when the worker is assigned a nonzero host GPU. Decoder RGB
|
||
conversion can differ slightly; validate a short Decord/PyNvVideoCodec pair
|
||
before extracting every session, and do not mix decoder backends across
|
||
train/test splits.
|
||
|
||
### Persistent-model Ego4D batch extraction
|
||
|
||
`extract_ego4d_batch.py` loads Qwen and its processor once, then extracts the
|
||
selected EgoSpeak manifest rows sequentially. Each clip receives a new media
|
||
reader, Thinker KV cache, TMRoPE origin, and rolling-memory state; only the
|
||
immutable model weights and processor are reused. Manifest indices are
|
||
one-based data-row indices and do not count the TSV header.
|
||
|
||
For example, process manifest samples 81 through the end on visible GPU 0:
|
||
|
||
```bash
|
||
CUDA_VISIBLE_DEVICES=0 \
|
||
OMP_NUM_THREADS=4 \
|
||
MKL_NUM_THREADS=4 \
|
||
OPENBLAS_NUM_THREADS=4 \
|
||
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||
python extract_ego4d_batch.py \
|
||
--data-root /data1/yfzx/wangx/Datasets/ego4d \
|
||
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
||
--start-index 81 \
|
||
--memory-seconds 20 \
|
||
--video-fps 2 \
|
||
--decoder decord \
|
||
--dtype bfloat16 \
|
||
--flash-attention-2 \
|
||
--device cuda:0
|
||
```
|
||
|
||
Completed non-empty `.pt` files are skipped. Each new output is first written
|
||
to a process-specific `.partial` path and atomically renamed after extraction,
|
||
so rerunning the same range resumes safely.
|
||
|
||
For CPU Decord, the direct reader intentionally indexes each clip and releases
|
||
that temporary reader, then keeps an LRU of at most two live readers during
|
||
extraction. Each reader defaults to one decode thread. These limits avoid one
|
||
worker retaining a thread pool for every one-minute EasyCom clip and can be
|
||
overridden with `--decoder-threads` and `--max-open-readers`. The recommended
|
||
full-run settings are:
|
||
|
||
```text
|
||
--decoder decord --decoder-threads 1 --max-open-readers 2
|
||
--frame-cache-size 64
|
||
```
|
||
|
||
Matching `states.shape[0]` to the target is necessary but not sufficient.
|
||
The direct reader also validates EasyCom's 20-Hz source inventory, source PTS
|
||
monotonicity, dense duration, crop bounds, audio sample count, and causal
|
||
frame-to-grid mapping. The original prepare scripts remain useful only as an
|
||
optional decode-speed/reproducibility cache.
|
||
|
||
## Saved streaming tensors
|
||
|
||
Each `.pt` file contains:
|
||
|
||
- `times`: `[N]`, prediction times in seconds;
|
||
- `states`: `[N, 2048]`, causal fused Thinker states;
|
||
- `label_indices`: `[0, ..., N-1]`, direct `target_perframe` row alignment;
|
||
- `valid_10_horizon_steps`: `N-10`, because the last ten current rows do not
|
||
have all future labels;
|
||
- `mode`: `causal-prefix` or `open-stream-replay`;
|
||
- `stream_representation`: explicit input-organization metadata;
|
||
- `config`: step, native block, memory, FPS, and pixel settings;
|
||
- `verification`: cached-versus-exact numerical checks;
|
||
- `input_video_fps` and `video_sampling_grid`: sampling provenance;
|
||
- `media_interval_policy`: fixed-grid AV uses half-open `[start, end)` ownership;
|
||
- `logits`: optional `[N, 10, 3]` output if a trained head is supplied;
|
||
- `probabilities`: optional softmax probabilities.
|
||
|
||
## Training the `10 x 3` head
|
||
|
||
The head must be trained. It is not part of pretrained Qwen.
|
||
|
||
The first experiment should freeze all Qwen parameters and train only:
|
||
|
||
```text
|
||
LayerNorm(2048)
|
||
-> Linear(2048, 1024)
|
||
-> GELU
|
||
-> Dropout
|
||
-> Linear(1024, 30)
|
||
-> reshape [10, 3]
|
||
```
|
||
|
||
`easycom_head.py` defines the head and loss; they remain re-exported by
|
||
`streaming_turn_model.py` for compatibility:
|
||
|
||
```python
|
||
from streaming_turn_model import (
|
||
TurnPredictionHead,
|
||
egospeak_multihorizon_loss,
|
||
)
|
||
```
|
||
|
||
The supplied LSTR config uses Adam, 50 epochs, learning rate `7e-5`, weight
|
||
decay `5e-5`, dropout `0.2`, and 10 warmup epochs followed by cosine decay.
|
||
`train_turn_head.py` uses those defaults. Its batch size is 512 causal states,
|
||
not LSTR's 16 temporal windows, because the two model inputs have different
|
||
shapes.
|
||
|
||
Train once on the official nine sessions with a fixed 50-epoch protocol:
|
||
|
||
```bash
|
||
python train_turn_head.py \
|
||
--feature-root features/open_2fps \
|
||
--target-root target_frame \
|
||
--output checkpoints/open_2fps_head.pth \
|
||
--device cuda:0
|
||
```
|
||
|
||
The default loss is `lstr-mce`, exactly matching LSTR's MCE treatment of a
|
||
multi-positive `[0,1,1]` row: it sums both active negative-log-softmax terms
|
||
without normalizing the target. The previous L2-normalized objective belongs
|
||
to MiniROAD and is available only as the explicit
|
||
`--loss miniroad-normalized` ablation.
|
||
|
||
Two label-structure ablations are also available without changing the saved
|
||
three-column target files:
|
||
|
||
- `--loss independent-bce` keeps a `10 x 3` head, trains the three labels with
|
||
independent sigmoid/BCE, and evaluates the sigmoid marginals directly;
|
||
- `--loss four-state-ce` changes the head to `10 x 4` and maps each target to
|
||
`silence`, `wearer_only`, `others_only`, or `overlap`. At evaluation time it
|
||
returns the original three marginal scores as `P(silence)`,
|
||
`P(wearer_only)+P(overlap)`, and `P(others_only)+P(overlap)`.
|
||
|
||
The four-state marginals are intentionally not renormalized. Keep features,
|
||
split, fusion, optimizer, schedule, and seeds fixed when comparing these losses.
|
||
For checkpoint selection, reserve one official training session such as
|
||
Session 12 with `--dev-sessions 12`; Sessions 1--3 remain test-only.
|
||
|
||
Recommended experimental procedure:
|
||
|
||
1. Freeze Qwen and extract all training-session states offline.
|
||
2. Verify state/label time offset and truncate only the final ten invalid
|
||
current positions.
|
||
3. Keep EgoSpeak targets as float multi-positive tensors `[N, 10, 3]`; never
|
||
convert them with `argmax`.
|
||
4. For comparison with the supplied LSTR shell, train with the default
|
||
`egospeak_multihorizon_loss`/`lstr-mce` objective.
|
||
5. Select hyperparameters using only training sessions. For EasyCom, reserve a
|
||
training session for development or use session-level cross-validation;
|
||
never tune on Sessions 1--3.
|
||
6. After choosing the epoch/hyperparameters, optionally retrain on all nine
|
||
EasyCom training sessions and evaluate the fixed model once on the test set.
|
||
|
||
Class-balanced/focal loss, a learned turn-query token, and LoRA are later
|
||
ablations. They should not replace the frozen-Omni linear/MLP-head baseline.
|
||
|
||
### Offline frozen states with transition-sparse supervision
|
||
|
||
This is the clean sampling-only control. It does not reconstruct cached AV
|
||
tokens and does not run the Thinker during training. It loads the same dense
|
||
5 Hz frozen states used by the standard head baseline, then retains only the
|
||
`target_index` values listed by the existing transition-sparse cache
|
||
manifests. The representation, head architecture, optimizer, 50 epochs, loss,
|
||
targets, and final dense test evaluator are otherwise unchanged.
|
||
|
||
Set `FROZEN_FEATURE_ROOT` to the exact feature directory used by the matching
|
||
dense-supervision baseline. Do not compare sparse 8-second states with a
|
||
dense 20-second baseline (or vice versa):
|
||
|
||
```bash
|
||
FROZEN_FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_20s_decord
|
||
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
|
||
SPARSE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s
|
||
|
||
CUDA_VISIBLE_DEVICES=3 \
|
||
python train_turn_head.py \
|
||
--feature-root "$FROZEN_FEATURE_ROOT" \
|
||
--target-root "$TARGET_ROOT" \
|
||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_7" \
|
||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_11" \
|
||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_10" \
|
||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_6" \
|
||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_9" \
|
||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_4" \
|
||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_8" \
|
||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_5" \
|
||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_12" \
|
||
--output checkpoints/easycom_frozen_states_sparse_supervision_50ep.pth \
|
||
--epochs 50 \
|
||
--batch-size 512 \
|
||
--expected-mode any \
|
||
--device cuda:0
|
||
```
|
||
|
||
Evaluation remains dense 5 Hz on the official test sessions:
|
||
|
||
```bash
|
||
CUDA_VISIBLE_DEVICES=3 \
|
||
python infer_turn_head.py \
|
||
--feature-root "$FROZEN_FEATURE_ROOT" \
|
||
--target-root "$TARGET_ROOT" \
|
||
--checkpoint checkpoints/easycom_frozen_states_sparse_supervision_50ep.pth \
|
||
--output-dir results/easycom_frozen_states_sparse_supervision_50ep \
|
||
--expected-mode any \
|
||
--device cuda:0
|
||
```
|
||
|
||
Compare this result against a head trained for the same 50 epochs on every
|
||
dense 5 Hz anchor from the same `FROZEN_FEATURE_ROOT`. A large drop isolates
|
||
the problem to sparse supervision; a close result means sparse supervision is
|
||
not sufficient to explain the LoRA regression.
|
||
|
||
## Thinker-LoRA sampling and cache policy
|
||
|
||
This is an implemented second-stage experiment, not yet a reported benchmark
|
||
result.
|
||
Dense 5 Hz partial-prefix caching is fast during training but does not fit the
|
||
combined EasyCom and Ego4D storage budget. The 24-second Session 7 pilot used
|
||
0.115 GiB, which extrapolates to about 17.25 GiB per media hour. In the latest
|
||
controlled single-L20 test without gradient checkpointing, 200 updates took
|
||
57.718 s (3.465 update/s) with every partial cached, 130.151 s (1.537
|
||
update/s) with current partials recomputed using processor defaults, and
|
||
111.146 s (1.799 update/s) after CUDA log-Mel and bounded-audio-padding
|
||
optimization. The optimized online run spent 53.408 s in 163 frozen-AV calls;
|
||
the cached run therefore remains the preferred training path when storage is
|
||
available.
|
||
|
||
The proposed training sampler is therefore:
|
||
|
||
1. retain a uniform low-rate set of ordinary anchors, initially 1 Hz;
|
||
2. retain every 5 Hz anchor within a two-second band around annotated speaker
|
||
onset, offset, handover, and overlap-state transitions;
|
||
3. optionally add high-loss or high-uncertainty anchors after a first training
|
||
pass, while selecting them only from the training split;
|
||
4. cache exact partial AV embeddings only for these selected anchors, plus one
|
||
reusable complete embedding for every required 2-second history block;
|
||
5. enforce a 240 GiB cache target and a hard 300 GB upper bound, leaving room
|
||
for manifests, checkpoints, and extracted evaluation states.
|
||
|
||
Transition-centered selection is training-example sampling, not inference
|
||
look-ahead: model inputs must still end at the current time. Final extraction,
|
||
head training, and Table 2 evaluation remain on the complete 5 Hz timeline.
|
||
The paper must report the sampling rule and compare uniform-only sampling with
|
||
transition-enhanced sampling so that any gain is not attributed solely to
|
||
additional transition examples.
|
||
|
||
`cache_omni_av_pieces.py --anchor-policy transition-sparse` implements this
|
||
policy without changing the trainer cache schema. A transition is any change
|
||
in the three-class EasyCom target vector. The default two-second dense band is
|
||
defined explicitly as one second before through one second after that target
|
||
row. Every complete two-second source block is still cached once as reusable
|
||
history. Ordinary selected anchors and transition-band anchors cache only
|
||
their exact current causal phase. Label information selects training examples
|
||
offline but never changes the media endpoint supplied to Qwen.
|
||
|
||
Inspect the selection counts for the 24-second pilot without loading media or
|
||
the model:
|
||
|
||
```bash
|
||
python cache_omni_av_pieces.py \
|
||
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
||
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_7 \
|
||
--target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_7.npy \
|
||
--output-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_7_24s \
|
||
--end 24 \
|
||
--anchor-policy transition-sparse \
|
||
--ordinary-anchor-hz 1 \
|
||
--transition-before-seconds 1 \
|
||
--transition-after-seconds 1 \
|
||
--selection-only
|
||
```
|
||
|
||
After auditing the counts, remove `--selection-only` and add the normal AV
|
||
extraction options. Use `--end 0` only after the 24-second cache and replay
|
||
verification pass; zero selects the complete target timeline. The resulting
|
||
manifest contains only selected training anchors, so `train_omni_turn_lora.py`
|
||
uses the sparse dataset automatically and must be run without
|
||
`--online-current-tail`.
|
||
|
||
### Multi-GPU Thinker-LoRA training
|
||
|
||
Fully cached AV training supports native single-node DDP through `torchrun`.
|
||
Each rank holds one complete 3B Thinker replica, reads a disjoint shard of the
|
||
selected cache anchors, and synchronizes only the trainable LoRA and turn-head
|
||
gradients. Model checkpoint loading is deliberately staggered by rank so four
|
||
processes do not read all checkpoint shards simultaneously. During gradient
|
||
accumulation, DDP synchronization occurs only on the final microbatch. Only
|
||
rank zero writes the adapter, head, summary, and progress log.
|
||
|
||
For four GPUs, use per-GPU batch 2 and accumulation 2 for global batch 16:
|
||
|
||
```bash
|
||
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||
OMP_NUM_THREADS=2 \
|
||
MKL_NUM_THREADS=2 \
|
||
OPENBLAS_NUM_THREADS=2 \
|
||
PYTORCH_ALLOC_CONF=expandable_segments:True \
|
||
torchrun --standalone --nproc_per_node=4 train_omni_turn_lora.py \
|
||
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_7 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_11 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_10 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_6 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_9 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_4 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_8 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_5 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_12 \
|
||
--output-dir checkpoints/omni_turn_lora_easycom_sparse_8s_ddp4 \
|
||
--max-steps 10170 \
|
||
--batch-size 2 \
|
||
--gradient-accumulation-steps 2 \
|
||
--num-workers 0 \
|
||
--block-cache-size 32 \
|
||
--learning-rate 2e-5 \
|
||
--head-learning-rate 1e-4 \
|
||
--warmup-steps 500 \
|
||
--dtype bfloat16 \
|
||
--flash-attention-2 \
|
||
--log-every 20
|
||
```
|
||
|
||
`10170` is only the current three-epoch estimate for roughly 54,000 selected
|
||
anchors. Recompute it from the completed manifests as
|
||
`ceil(num_examples / 16) * 3`. Do not pass `--device`: under `torchrun`, each
|
||
process uses its `LOCAL_RANK`. DDP intentionally rejects
|
||
`--online-current-tail`; the online producer-consumer path remains a separate
|
||
benchmark.
|
||
|
||
### Dense low-drift LoRA probe
|
||
|
||
The first follow-up to the sparse-LoRA regression should remove sparse
|
||
supervision as a variable and constrain adaptation near the Thinker output.
|
||
Use dense 5-Hz caches, reserve `Session_12` as development data, and train on
|
||
`Session_4` through `Session_11`. `--lora-last-n-layers 8` discovers the
|
||
language-layer indices and passes the 24 selected full module paths directly
|
||
to PEFT for the 36-layer Qwen2.5-Omni-3B Thinker. Explicit paths are required
|
||
because the extracted Thinker exposes names beginning with `layers.0...`,
|
||
which some PEFT `layers_pattern` versions fail to recognize. Combining it with
|
||
`--lora-target-modules q_proj,v_proj,o_proj` reduces the adapter from all 252
|
||
attention/MLP projections to 24 output-side attention projections.
|
||
|
||
Start with the successful frozen 8-second head fixed. `--freeze-head`
|
||
requires `--head-checkpoint`, excludes the head from DDP and AdamW, but still
|
||
backpropagates through it into the Thinker LoRA. This isolates representation
|
||
adaptation from head decision-boundary drift. Do not continue from a regressed
|
||
adapter: create this adapter from the original Thinker.
|
||
|
||
For four GPUs with global batch 16, compute exactly one epoch from the dense
|
||
manifest anchor counts and run:
|
||
|
||
```bash
|
||
cd /data1/yfzx/wangx/Code/icassp2027-experiment
|
||
|
||
MAX_STEPS=$(python - <<'PY'
|
||
import json
|
||
import math
|
||
from pathlib import Path
|
||
|
||
root = Path("/data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s")
|
||
sessions = (4, 5, 6, 7, 8, 9, 10, 11)
|
||
examples = sum(len(json.loads((root / f"Session_{s}" / "manifest.json").read_text())["anchors"])
|
||
for s in sessions)
|
||
print(math.ceil(examples / 16))
|
||
PY
|
||
)
|
||
WARMUP_STEPS=$(( (MAX_STEPS + 19) / 20 ))
|
||
|
||
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||
OMP_NUM_THREADS=2 \
|
||
MKL_NUM_THREADS=2 \
|
||
OPENBLAS_NUM_THREADS=2 \
|
||
PYTORCH_ALLOC_CONF=expandable_segments:True \
|
||
torchrun --standalone --nproc_per_node=4 train_omni_turn_lora.py \
|
||
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_4 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_5 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_6 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_7 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_8 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_9 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_10 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_11 \
|
||
--head-checkpoint checkpoints/easycom_open_stream_2fps_8s_head_seed0.pth \
|
||
--freeze-head \
|
||
--lora-target-modules q_proj,v_proj,o_proj \
|
||
--lora-last-n-layers 8 \
|
||
--lora-rank 8 \
|
||
--lora-alpha 16 \
|
||
--learning-rate 5e-6 \
|
||
--output-dir checkpoints/omni_turn_lora_dense_8s_last8_qvo_frozen_head_epoch1 \
|
||
--max-steps "${MAX_STEPS}" \
|
||
--batch-size 2 \
|
||
--gradient-accumulation-steps 2 \
|
||
--num-workers 0 \
|
||
--block-cache-size 32 \
|
||
--warmup-steps "${WARMUP_STEPS}" \
|
||
--dtype bfloat16 \
|
||
--flash-attention-2 \
|
||
--log-every 20
|
||
```
|
||
|
||
Select this recipe on dense `Session_12` using 0.2-second mAP as the primary
|
||
metric and mean mAP as the tie-breaker. Only after the recipe is fixed should
|
||
it be evaluated on official test `Sessions_1--3`. A second controlled run may
|
||
omit `--freeze-head` and use `--head-learning-rate 1e-5`; all other settings
|
||
must remain identical.
|
||
|
||
For a continuous five-epoch run, set
|
||
`TOTAL_STEPS=$((5 * MAX_STEPS))` and pass
|
||
`--max-steps "${TOTAL_STEPS}" --save-every-steps "${MAX_STEPS}"`. The trainer
|
||
keeps one AdamW/cosine-scheduler trajectory and writes evaluation-ready
|
||
`adapter/` plus `turn_head.pt` pairs under `checkpoint-step-MAX_STEPS`,
|
||
`checkpoint-step-2*MAX_STEPS`, through the fifth epoch. The top-level adapter
|
||
and head duplicate the final fifth checkpoint. Evaluate all five intermediate
|
||
checkpoints only on held-out `Session_12`, select one, then run that checkpoint
|
||
once on `Sessions_1--3`.
|
||
|
||
### Frozen-Thinker sparse head-only control
|
||
|
||
To determine whether the regression comes from transition-sparse sampling or
|
||
from updating the Thinker representation, rerun the same sparse-anchor
|
||
training with `--freeze-thinker`. This still forwards every reconstructed
|
||
8-second sequence through the original Thinker, but runs that forward under
|
||
`no_grad` and updates only the turn head initialized from the frozen 8-second
|
||
checkpoint. It does not create an adapter directory.
|
||
|
||
For a schedule-matched comparison with a one-epoch LoRA run at global batch
|
||
16, use the same cache directories, head learning rate, warmup, seed, and
|
||
approximately `ceil(num_examples / 16)` updates:
|
||
|
||
```bash
|
||
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||
OMP_NUM_THREADS=2 \
|
||
MKL_NUM_THREADS=2 \
|
||
OPENBLAS_NUM_THREADS=2 \
|
||
PYTORCH_ALLOC_CONF=expandable_segments:True \
|
||
torchrun --standalone --nproc_per_node=4 train_omni_turn_lora.py \
|
||
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_7 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_11 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_10 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_6 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_9 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_4 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_8 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_5 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_12 \
|
||
--freeze-thinker \
|
||
--head-checkpoint checkpoints/easycom_open_stream_2fps_8s_head_seed0.pth \
|
||
--output-dir checkpoints/easycom_sparse_head_only_from_frozen_8s_ddp4_epoch1 \
|
||
--max-steps 3390 \
|
||
--batch-size 2 \
|
||
--gradient-accumulation-steps 2 \
|
||
--num-workers 0 \
|
||
--block-cache-size 32 \
|
||
--head-learning-rate 1e-4 \
|
||
--warmup-steps 500 \
|
||
--dtype bfloat16 \
|
||
--flash-attention-2 \
|
||
--log-every 20
|
||
```
|
||
|
||
Evaluate the resulting head on the same dense test caches without an adapter:
|
||
|
||
```bash
|
||
CUDA_VISIBLE_DEVICES=0,1,2 \
|
||
OMP_NUM_THREADS=1 \
|
||
MKL_NUM_THREADS=1 \
|
||
OPENBLAS_NUM_THREADS=1 \
|
||
PYTORCH_ALLOC_CONF=expandable_segments:True \
|
||
TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
|
||
torchrun --standalone --nproc_per_node=3 eval_omni_turn_lora.py \
|
||
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_3 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_2 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_1 \
|
||
--head-checkpoint checkpoints/easycom_sparse_head_only_from_frozen_8s_ddp4_epoch1/turn_head.pt \
|
||
--output-dir results/easycom_sparse_head_only_dense_eval_epoch1 \
|
||
--batch-size 2 \
|
||
--block-cache-size 32 \
|
||
--dtype bfloat16 \
|
||
--flash-attention-2 \
|
||
--ddp-timeout-minutes 60
|
||
```
|
||
|
||
If this head-only control also falls well below the frozen `58.16` dense-cache
|
||
control, sparse sampling/head optimization is sufficient to explain the
|
||
regression. If it stays near `58.16` while LoRA falls, the main failure is
|
||
Thinker representation drift.
|
||
|
||
To continue from a completed run, pass both its `adapter/` directory through
|
||
`--adapter-checkpoint` and its `turn_head.pt` through `--head-checkpoint` while
|
||
writing to a new output directory. This is weights-only continuation: the
|
||
adapter and head weights resume, but AdamW moments and the cosine scheduler are
|
||
reinitialized. Preserve the previous global batch when changing GPU count;
|
||
for example, change 8 GPUs with batch 2 and accumulation 1 to 4 GPUs with batch
|
||
2 and accumulation 2. Use zero warmup and a reduced learning rate for a short
|
||
post-hoc continuation because a one-epoch cosine schedule has already reached
|
||
zero.
|
||
|
||
### Evaluate a Thinker-LoRA checkpoint
|
||
|
||
Use `eval_omni_turn_lora.py` for the formal EasyCom Table 2 evaluation. It
|
||
loads the trained Thinker LoRA and `10 x 3` turn head, reconstructs every causal
|
||
state from dense AV caches, and reports both the EgoSpeak full-timeline and
|
||
strict causal-valid protocols. Sparse training caches are rejected because
|
||
they do not contain predictions at every 200 ms timestamp.
|
||
|
||
The following command evaluates the official EasyCom test sessions on three
|
||
GPUs, with one complete session assigned to each rank:
|
||
|
||
```bash
|
||
CHECKPOINT=/data1/yfzx/wangx/Code/icassp2027-experiment/checkpoints/omni_turn_lora_easycom_sparse_8s_ddp8_epoch1
|
||
|
||
CUDA_VISIBLE_DEVICES=0,1,2 \
|
||
OMP_NUM_THREADS=1 \
|
||
MKL_NUM_THREADS=1 \
|
||
OPENBLAS_NUM_THREADS=1 \
|
||
PYTORCH_ALLOC_CONF=expandable_segments:True \
|
||
TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
|
||
torchrun --standalone --nproc_per_node=3 eval_omni_turn_lora.py \
|
||
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_3 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_2 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_1 \
|
||
--adapter-checkpoint "${CHECKPOINT}/adapter" \
|
||
--head-checkpoint "${CHECKPOINT}/turn_head.pt" \
|
||
--output-dir results/omni_turn_lora_easycom_sparse_8s_ddp8_epoch1 \
|
||
--batch-size 2 \
|
||
--block-cache-size 32 \
|
||
--dtype bfloat16 \
|
||
--flash-attention-2 \
|
||
--ddp-timeout-minutes 60
|
||
```
|
||
|
||
The evaluator saves per-session probabilities and valid masks under `pred/`
|
||
and `valid/`, and writes the aggregate metrics to `metrics.json`. Add
|
||
`--overwrite` only when intentionally replacing an existing result directory.
|
||
Treat Sessions 1, 2, and 3 as the official test set: inspect their metrics only
|
||
after the training recipe and hyperparameters have been fixed.
|
||
|
||
To isolate whether a score change comes from LoRA or from the dense-cache
|
||
reconstruction path, run the same evaluator without `--adapter-checkpoint` and
|
||
load the original frozen 8-second head:
|
||
|
||
```bash
|
||
FROZEN_HEAD=checkpoints/easycom_open_stream_2fps_8s_head_seed0.pth
|
||
|
||
CUDA_VISIBLE_DEVICES=0,1,2 \
|
||
OMP_NUM_THREADS=1 \
|
||
MKL_NUM_THREADS=1 \
|
||
OPENBLAS_NUM_THREADS=1 \
|
||
PYTORCH_ALLOC_CONF=expandable_segments:True \
|
||
TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
|
||
torchrun --standalone --nproc_per_node=3 eval_omni_turn_lora.py \
|
||
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_3 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_2 \
|
||
--cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_1 \
|
||
--head-checkpoint "${FROZEN_HEAD}" \
|
||
--output-dir results/easycom_frozen_8s_dense_cache_control \
|
||
--batch-size 2 \
|
||
--block-cache-size 32 \
|
||
--dtype bfloat16 \
|
||
--flash-attention-2 \
|
||
--ddp-timeout-minutes 60
|
||
```
|
||
|
||
This control should approach the previously reported frozen 8-second score of
|
||
58.51937 full-timeline mean mAP. A material mismatch indicates a representation
|
||
difference between dense-cache reconstruction and the earlier open-stream
|
||
feature files; a close match isolates the regression to LoRA/head training.
|
||
|
||
### Recorded sparse-LoRA results
|
||
|
||
The current single-run EasyCom results confirm negative transfer from this
|
||
sparse LoRA recipe:
|
||
|
||
| Checkpoint | Full-timeline mean mAP | Strict causal mean mAP |
|
||
|---|---:|---:|
|
||
| Frozen Thinker dense-cache control | 58.16091 | 58.17146 |
|
||
| Frozen states + sparse supervision, 50 epochs | 57.20379 | 57.21367 |
|
||
| Sparse LoRA epoch1 | 56.52480 | 56.53450 |
|
||
| Frozen-head initialized LoRA, 3 epochs | 56.19608 | 56.20535 |
|
||
| Sparse LoRA weights-only epoch2 | 54.19850 | 54.20711 |
|
||
|
||
Epoch2 is a weights-only continuation: adapter/head weights resume, but AdamW
|
||
and the scheduler restart. It is 2.32630 mAP points below epoch1 and 3.96241
|
||
below the frozen dense-cache control in the full-timeline protocol. The largest
|
||
regression is camera-wearer speaking. Do not continue the same recipe to a
|
||
third run. The complete snapshot and per-class metrics are archived in
|
||
[`results/easycom_omni_lora_sparse_epoch2_2026-08-24/`](results/easycom_omni_lora_sparse_epoch2_2026-08-24/).
|
||
|
||
The frozen-state sparse-supervision and frozen-head initialized three-epoch
|
||
LoRA tables are archived in
|
||
[`results/easycom_sparse_supervision_vs_lora_2026-08-24/`](results/easycom_sparse_supervision_vs_lora_2026-08-24/README.md).
|
||
At 0.2 seconds, sparse supervision alone changes mAP from `71.69695` to
|
||
`71.69194` (effectively unchanged), while the three-epoch LoRA result is
|
||
`71.08589`. This isolates the short-horizon regression primarily to Thinker
|
||
updating rather than sparse target selection.
|
||
|
||
For a speed-only alternative, `train_omni_turn_lora.py` supports a two-GPU
|
||
producer-consumer benchmark. The producer GPU runs the frozen AV towers and
|
||
places current-tail embeddings in a bounded CPU queue while the consumer GPU
|
||
runs Thinker LoRA. This preserves exact online partial semantics; it changes
|
||
only scheduling. Compare its reported update rate and consumer queue-wait
|
||
time with the synchronous 0.787 update/s pilot before adopting it.
|
||
|
||
## EgoSpeak-compatible evaluation
|
||
|
||
Run inference on the official test sessions (`Session_3`, `Session_2`, and
|
||
`Session_1`):
|
||
|
||
```bash
|
||
python infer_turn_head.py \
|
||
--feature-root features/open_2fps \
|
||
--target-root target_frame \
|
||
--checkpoint checkpoints/open_2fps_head.pth \
|
||
--output-dir results/open_2fps_head \
|
||
--device cuda:0
|
||
```
|
||
|
||
This writes `pred/Session_X.npy` with the exact LSTR/EgoSpeak layout
|
||
`[target_time, class, future_horizon]`, i.e. `[T,3,10]`, plus `metrics.json`.
|
||
For every valid current timestep, the head itself produces probabilities
|
||
shaped:
|
||
|
||
```text
|
||
[number of timesteps, 10 future horizons, 3 classes]
|
||
```
|
||
|
||
Then reproduce EgoSpeak's Table 2 procedure:
|
||
|
||
1. for each horizon independently, concatenate test-session predictions and
|
||
ground truth;
|
||
2. compute per-class frame-level average precision;
|
||
3. average the three class AP values to obtain horizon mAP;
|
||
4. report all ten horizon mAP values;
|
||
5. average the ten horizon mAP values for the final score.
|
||
|
||
With EgoSpeak's `ignore_index=-100`, the released evaluator does not remove
|
||
class index zero, so background is included. The script reports two protocols:
|
||
|
||
- `egospeak_full_timeline`: the same full `[T,3,10]` array and evaluator as
|
||
LSTR, with zeros where a prediction would require pre-session context;
|
||
- `causal_valid_only`: excludes only those unavailable first `h` rows for
|
||
horizon `h`. This is the primary strictly causal score.
|
||
|
||
The difference involves at most the first ten rows of each session, but both
|
||
are saved so the comparison is explicit.
|
||
|
||
The convenience wrapper mirrors EgoSpeak's train/inference shells:
|
||
|
||
```bash
|
||
chmod +x run_easycom_head_experiment.sh
|
||
CUDA_VISIBLE_DEVICES=0 ./run_easycom_head_experiment.sh all
|
||
```
|
||
|
||
Reference Table 2 targets from the EgoSpeak paper:
|
||
|
||
- EasyCom GRU A+V: **60.6 mean mAP**;
|
||
- EasyCom Transformer A+V: **58.7 mean mAP**;
|
||
- Ego4D Transformer audio: **69.2 mean mAP**;
|
||
- Ego4D Transformer A+V: **69.0 mean mAP**;
|
||
- Ego4D GRU audio: **69.2 mean mAP**;
|
||
- Ego4D GRU A+V: **68.2 mean mAP**.
|
||
|
||
Do not compare these numbers with EgoSpeak Table 3 target-speaker-only AP;
|
||
Table 3 uses a different metric.
|
||
|
||
## Experiments that demonstrate Omni's semantic advantage
|
||
|
||
The headline comparison alone cannot establish that improvements come from
|
||
semantics. Run the following controlled experiments:
|
||
|
||
1. **Omni audio vs Omni audio+video**: measure whether visual grounding helps.
|
||
2. **Per-horizon curve**: semantic completion should be especially visible at
|
||
longer horizons such as 1.2--2.0 seconds.
|
||
3. **Context length**: compare 4, 10, 15, and 20 seconds without adding an RNN.
|
||
4. **Layer ablation**: compare audio-encoder output and several Thinker layers,
|
||
because an intermediate layer may be more discriminative than the final
|
||
generation-oriented layer.
|
||
5. **Semantic-completion subset**: contrast completed utterances with
|
||
hesitation/incomplete pauses while matching pause duration and basic
|
||
prosody as closely as possible.
|
||
6. **Question/statement/backchannel subsets**: report target-speaker AP and
|
||
premature-interruption rate on linguistically meaningful events.
|
||
7. **Causal transcript baseline**: compare Omni audio with causal ASR text plus
|
||
a language model to isolate lexical/semantic contribution.
|
||
|
||
The frozen-Omni head is the cleanest representation test. LoRA can be reported
|
||
as a second-stage ceiling, not as the only result.
|
||
|
||
## Transcript-derived interaction-event proxies
|
||
|
||
EasyCom's released human transcriptions contain inclusive 20 FPS
|
||
`Start_Frame`/`End_Frame`, `Participant_ID`, `Transcription`, and
|
||
`Target_of_Speech`. They do **not** contain a semantic backchannel class.
|
||
[`derive_easycom_interaction_events.py`](derive_easycom_interaction_events.py)
|
||
therefore separates observed timing relations from weak semantic proxies:
|
||
|
||
- `ordinary_switch`: a non-overlapping recent speaker change followed by a
|
||
sustained incoming turn;
|
||
- `overlap_takeover`: an observed overlapping onset followed by sustained
|
||
takeover, without claiming interruption intent;
|
||
- `overlap_hold`: overlap without enough evidence for takeover;
|
||
- `backchannel_proxy_high`: a conservative short lexical response while the
|
||
same other participant dominates immediately before and after it;
|
||
- `interruption_proxy_high`: the previous transcript has a non-boundary
|
||
trailing cutoff and the incoming participant takes over;
|
||
- `nonverbal_feedback`: laughter-only feedback around another participant's
|
||
floor; pure `[H]`, `[C]`, `[T]`, and `[U]` records are ambiguous candidates.
|
||
|
||
`backchannel_candidate`, `interruption_candidate`, and
|
||
`nonverbal_candidate` are retained for auditing but masked from auxiliary
|
||
training. `[X]` continuations across adjacent one-minute files are not new
|
||
events. Source-time gaps larger than one second create separate segments, so
|
||
the dense EgoSpeak concatenation does not create a false switch across a
|
||
redacted interval. A one- or two-frame annotation/media endpoint discrepancy
|
||
is clamped and reported; larger disagreement is fatal.
|
||
|
||
A read-only audit on the raw-data machine is:
|
||
|
||
```bash
|
||
cd /mnt/58462F4C462F29E8/Code/icassp2027-experiment
|
||
|
||
python3 derive_easycom_interaction_events.py \
|
||
--dataset-root /mnt/58462F4C462F29E8/Datasets/easycom \
|
||
--target-root /mnt/58462F4C462F29E8/Code/EgoSpeak/dataset/EasyCom/target_perframe \
|
||
--examples-per-event 10
|
||
```
|
||
|
||
To persist JSONL/CSV reports outside the dataset, add:
|
||
|
||
```bash
|
||
--output-dir reports/easycom_interaction_events_v1 \
|
||
--overwrite
|
||
```
|
||
|
||
In an environment with NumPy, add `--write-frame-arrays` to also write one
|
||
`Session_*.aux_targets.npz` per session. It contains 5 Hz `event_onsets`,
|
||
`event_roles`, collision flags, and event-name metadata aligned to
|
||
`target_perframe`. The raw EasyCom tree is never modified.
|
||
|
||
The initial all-session audit found 6,981 utterances, including 530
|
||
high-confidence backchannel proxies, 78 high-confidence interruption proxies,
|
||
937 overlap-takeovers, 686 overlap-holds, and 349 laughter-feedback events.
|
||
These counts establish usable weak-supervision coverage, not label accuracy.
|
||
Before training, inspect a stratified sample from Sessions 4--12 and freeze the
|
||
rules; do not tune thresholds on test Sessions 1--3. The primary evaluator
|
||
remains the original EgoSpeak three-label AP/mAP protocol.
|
||
|
||
### Optional DeepSeek adjudication of ambiguous candidates
|
||
|
||
DeepSeek is not required for the deterministic timing labels above. It is an
|
||
optional second-stage weak-label adjudicator for only
|
||
`backchannel_candidate` and `interruption_candidate`; sending all 6,981
|
||
utterances to an LLM would add cost and noise without adding observable
|
||
prosody, gaze, or intent. The current audit contains 614 such candidates over
|
||
all sessions and 497 in the training split (Sessions 4--12).
|
||
|
||
[`adjudicate_easycom_interaction_events.py`](adjudicate_easycom_interaction_events.py)
|
||
uses DeepSeek JSON Output through the standard-library HTTP client, so no
|
||
additional Python package is required. It makes two differently instructed
|
||
conservative passes. A candidate is promoted only if both passes return the candidate's
|
||
matching semantic label with high confidence; disagreements, medium/low
|
||
confidence, short answers, cooperative overlaps, and uncertain cases remain
|
||
masked. The result is still a weak proxy and must not be presented as human
|
||
ground truth. The append-only JSONL records prompt/model metadata and resumes
|
||
completed candidates, while rejecting a mixed model or prompt configuration.
|
||
|
||
First inspect exactly what would leave the machine. This makes no API call and
|
||
does not require a key:
|
||
|
||
```bash
|
||
python3 adjudicate_easycom_interaction_events.py \
|
||
--input-dir reports/easycom_interaction_events_v1 \
|
||
--output reports/easycom_interaction_llm_v1.jsonl \
|
||
--max-events 16 \
|
||
--dry-run
|
||
```
|
||
|
||
If external transcript upload is authorized, enter the key without placing it
|
||
in shell history and run a 32-candidate pilot:
|
||
|
||
```bash
|
||
read -s -p "DeepSeek API key: " DEEPSEEK_API_KEY; echo
|
||
export DEEPSEEK_API_KEY
|
||
|
||
python3 adjudicate_easycom_interaction_events.py \
|
||
--input-dir reports/easycom_interaction_events_v1 \
|
||
--output reports/easycom_interaction_llm_pilot_v1.jsonl \
|
||
--max-events 32 \
|
||
--selection-seed 2027 \
|
||
--batch-size 8 \
|
||
--passes 2 \
|
||
--allow-external-transcript-upload
|
||
|
||
unset DEEPSEEK_API_KEY
|
||
```
|
||
|
||
Audit the pilot rationales and precision manually before running the full
|
||
training split. For the full pass, use a new output path, remove
|
||
`--max-events`, and keep the default `--splits train`. The default model is
|
||
`deepseek-v4-flash`; it can be changed with `--model` or `DEEPSEEK_MODEL` as
|
||
the provider's available model names evolve. Never tune the prompt or
|
||
acceptance rule using Sessions 1--3. LLM-adjudicated candidates are deliberately
|
||
kept in a separate JSONL until pilot precision is accepted; they are not yet
|
||
silently merged into the training arrays.
|
||
|
||
## Candidate improvement: explicit working and long memory
|
||
|
||
Motivated by EgoSpeak's separation of long-term and working memory, test an
|
||
explicit two-timescale representation instead of compressing the entire causal
|
||
window into one undifferentiated final Thinker state. At prediction time `t`,
|
||
align long memory to fixed global 2-second boundaries and let working memory
|
||
slide with the current prediction time:
|
||
|
||
```text
|
||
b = floor(t / 2 s) * 2 s
|
||
long memory: complete 2-second chunks ending at b, ..., [b-4, b-2], [b-2, b]
|
||
working memory: the sliding recent interval [t-2, t], updated every 0.2 s
|
||
```
|
||
|
||
For example, at `t = 20.4 s`, long memory contains the fixed chunks
|
||
`[16,18]`, `[18,20]`, and any earlier retained chunks, while the working
|
||
partial is `[18.4,20.4]`. The overlap between `[18,20]` and `[18.4,20.4]` is
|
||
intentional: long memory provides a stable completed-chunk representation,
|
||
whereas working memory preserves the exact high-resolution context ending at
|
||
the current prediction time. Cache one representation per completed
|
||
long-memory chunk, recompute the working-memory representation at each 5 Hz
|
||
prediction step, and fuse the two branches with a small gated/attention
|
||
pooling module before the same `10 x 3` prediction head. Keep the Omni
|
||
backbone frozen in the first test so that any gain can be attributed to memory
|
||
organization rather than additional backbone capacity.
|
||
|
||
This is a candidate score-improvement idea, not yet an experimental result.
|
||
The key hypothesis is that a dedicated recent branch preserves local prosody,
|
||
speaker-change, and utterance-completion cues while long memory supplies
|
||
conversation context without diluting them in one final state. Evaluate it
|
||
against matched-compute baselines: (1) the current single-state 8-second model,
|
||
(2) working-memory-only `[t-2,t]`, and (3) long+working memory with the same
|
||
total causal context. Report parameter count, throughput, and per-horizon and
|
||
per-class AP over identical seeds. A useful result requires the fused model to
|
||
beat both controls consistently; otherwise a gain could merely come from the
|
||
extra fusion parameters or the stronger recent-window shortcut.
|
||
|
||
### Frozen serialized-recap ablation (implemented)
|
||
|
||
Before adding a learned two-branch fusion module, test a stricter frozen-Omni
|
||
variant with exactly the same 2048-dimensional readout and prediction head as
|
||
the current 8-second baseline. Mode `overlap-working-memory` serializes the two
|
||
timescales inside one still-open Thinker media turn:
|
||
|
||
```text
|
||
total AV presentation budget = 8 s
|
||
fixed history budget = 6 s = three cached 2 s blocks
|
||
working recap = 2 s = [t-2,t]
|
||
|
||
t = 20.4 s:
|
||
fixed: [14,16] [16,18] [18,20]
|
||
working: [18.4,20.4]
|
||
-> one frozen Thinker state [2048]
|
||
```
|
||
|
||
The overlap is intentional. It presents recent evidence twice but does not add
|
||
AV-token duration beyond the matched 8-second budget. Thus this ablation trades
|
||
older unique context for stronger recent evidence instead of silently giving
|
||
the candidate more input tokens. Fixed blocks retain the global half-open 2 FPS
|
||
grid. The working recap reuses three causal global-grid history frames and
|
||
replaces the oldest visual slot with the current causal frame:
|
||
|
||
```text
|
||
fixed [18,20): 18.0 18.5 19.0 19.5
|
||
working [18.4,20.4]: 18.5 19.0 19.5 20.4
|
||
```
|
||
|
||
Keeping four frames means the oldest visual tick is replaced by the current
|
||
frame rather than adding a fifth frame; this isolates endpoint recency without
|
||
increasing visual tokens. Historical target times are causally floored to the
|
||
session-global 2 FPS grid. This lets them share the complete-block frame cache
|
||
and avoids making Decord seek backward through a long-GOP H.264 stream for four
|
||
new endpoint-shifted timestamps every 200 ms. Audio remains the full, exactly
|
||
half-open `[start,end)` slice. Consequently, call this a global-history plus
|
||
current-frame recent-video intervention, not an exact timestamp-preserving
|
||
resampling of both modalities. The recap is serialized as the next TMRoPE
|
||
chunk; it is deliberately a recent-evidence recap, not a claim that duplicated
|
||
samples occur later in physical time. This mode is an ablation, not native Qwen
|
||
streaming semantics, and the Omni backbone stays frozen.
|
||
|
||
One deliberate side effect is a 2-second phase cycle: overlap falls toward
|
||
0.2 seconds just before a fixed boundary and jumps to 2.0 seconds at the
|
||
boundary. Every output stores this value in
|
||
`overlap_working_memory_policy.layout`. If aggregate mAP changes, also inspect
|
||
performance by `step_index % 10`; a boundary-only gain or failure is a chunk
|
||
phase artifact rather than general evidence for better recent semantics.
|
||
|
||
Run a 24-second L20 smoke test first:
|
||
|
||
The sliding 2-second recap otherwise invokes Whisper's model-maximum audio
|
||
padding 120 times. Run log-Mel extraction on the model GPU and set explicit
|
||
right padding to zero. Every EasyCom 0.2-second increment contains 3200 samples
|
||
at 16 kHz and is already divisible by Whisper's 160-sample hop, so no external
|
||
silence is needed. This preserves the feature extractor's own STFT boundary
|
||
handling while avoiding repeated processing of tens of seconds of zeros.
|
||
|
||
```bash
|
||
mkdir -p features/easycom_overlap_working_8s_smoke
|
||
|
||
CUDA_VISIBLE_DEVICES=3 \
|
||
OMP_NUM_THREADS=4 \
|
||
MKL_NUM_THREADS=4 \
|
||
OPENBLAS_NUM_THREADS=4 \
|
||
PYTORCH_ALLOC_CONF=expandable_segments:True \
|
||
python streaming_turn_model_directly.py \
|
||
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
||
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_7 \
|
||
--end 24 \
|
||
--mode overlap-working-memory \
|
||
--memory-seconds 8 \
|
||
--working-memory-seconds 2 \
|
||
--video-fps 2 \
|
||
--frame-cache-size 128 \
|
||
--audio-feature-device model \
|
||
--audio-right-padding-samples 0 \
|
||
--decoder decord \
|
||
--decoder-threads 1 \
|
||
--max-open-readers 1 \
|
||
--dtype bfloat16 \
|
||
--flash-attention-2 \
|
||
--device cuda:0 \
|
||
--verify-every 0 \
|
||
--output features/easycom_overlap_working_8s_smoke/Session_7_24s_global_history_current.pt
|
||
```
|
||
|
||
Expected shape is `(120, 2048)`. This speed smoke test disables replay
|
||
verification; after its throughput is acceptable, rerun a short four-second
|
||
check with `--verify-every 5`, where `overlap_cache_replay` should be
|
||
numerically exact or effectively exact. Inspect the saved timing policy and
|
||
decoder cache counters with:
|
||
|
||
```bash
|
||
python - <<'PY'
|
||
import torch
|
||
|
||
p = torch.load(
|
||
"features/easycom_overlap_working_8s_smoke/Session_7_24s_global_history_current.pt",
|
||
map_location="cpu",
|
||
weights_only=False,
|
||
)
|
||
print(p["mode"], tuple(p["states"].shape))
|
||
print(p["overlap_working_memory_policy"]["working_media"])
|
||
print(p["direct_media"]["runtime_counters"])
|
||
for row in p["overlap_working_memory_policy"]["layout"][98:102]:
|
||
print(row)
|
||
PY
|
||
```
|
||
|
||
For one complete session, replace `--end 24` with its target and use a
|
||
session-level output filename:
|
||
|
||
```bash
|
||
mkdir -p /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s
|
||
|
||
CUDA_VISIBLE_DEVICES=3 \
|
||
OMP_NUM_THREADS=4 \
|
||
MKL_NUM_THREADS=4 \
|
||
OPENBLAS_NUM_THREADS=4 \
|
||
PYTORCH_ALLOC_CONF=expandable_segments:True \
|
||
python streaming_turn_model_directly.py \
|
||
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
||
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_7 \
|
||
--target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_7.npy \
|
||
--mode overlap-working-memory \
|
||
--memory-seconds 8 \
|
||
--working-memory-seconds 2 \
|
||
--video-fps 2 \
|
||
--frame-cache-size 128 \
|
||
--audio-feature-device model \
|
||
--audio-right-padding-samples 0 \
|
||
--decoder decord \
|
||
--decoder-threads 1 \
|
||
--max-open-readers 1 \
|
||
--dtype bfloat16 \
|
||
--flash-attention-2 \
|
||
--device cuda:0 \
|
||
--output /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s/Session_7.pt
|
||
```
|
||
|
||
After all train/test sessions have been extracted, train the unchanged frozen
|
||
head by setting only the expected feature mode:
|
||
|
||
```bash
|
||
python train_turn_head.py \
|
||
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s \
|
||
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
|
||
--output checkpoints/easycom_overlap_working_frozen_8s.pt \
|
||
--expected-mode overlap-working-memory \
|
||
--epochs 50 \
|
||
--batch-size 512 \
|
||
--device cuda:0
|
||
|
||
python infer_turn_head.py \
|
||
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s \
|
||
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
|
||
--checkpoint checkpoints/easycom_overlap_working_frozen_8s.pt \
|
||
--output-dir results/easycom_overlap_working_frozen_8s \
|
||
--expected-mode overlap-working-memory \
|
||
--device cuda:0
|
||
```
|
||
|
||
The decisive comparison is against the frozen single-state 8-second baseline
|
||
using the same split, head, optimization, and seeds. Report the full
|
||
per-horizon curve: the hypothesis predicts the clearest gain at 0.2--1.0
|
||
seconds; degradation at 1.2--2.0 seconds would indicate that repeated local
|
||
cues displaced too much useful long context. Only if this parameter-matched
|
||
test is positive should the learned two-branch fusion experiment be added.
|
||
|
||
## Single-window feature inspection
|
||
|
||
`qwen_omni_features.py` is useful for inspecting a short window or layer, but
|
||
it is not the main whole-session experiment.
|
||
|
||
If audio is embedded in the video:
|
||
|
||
```bash
|
||
python qwen_omni_features.py \
|
||
--video /path/to/window.mp4 \
|
||
--use-audio-in-video \
|
||
--output features/window.pt
|
||
```
|
||
|
||
If audio and video are separate:
|
||
|
||
```bash
|
||
python qwen_omni_features.py \
|
||
--video /path/to/window.mp4 \
|
||
--audio /path/to/window.wav \
|
||
--output features/window.pt
|
||
```
|
||
|
||
Do not provide the same audio both as a separate file and via
|
||
`--use-audio-in-video`.
|
||
|
||
Saved single-window fields are:
|
||
|
||
- `token_features`: `[B, token_length, 2048]`;
|
||
- `last_token`: `[B, 2048]`;
|
||
- `tail_mean`: `[B, 2048]`;
|
||
- `attention_mask`: valid-token mask.
|
||
|
||
## Installation
|
||
|
||
Install `ffmpeg`, then:
|
||
|
||
```bash
|
||
pip install -r requirements.txt
|
||
```
|
||
|
||
The implementation is pinned to `transformers==4.52.3` because the
|
||
open-stream replay path uses version-specific Thinker internals. On Linux,
|
||
`qwen-omni-utils[decord]` is recommended for faster video decoding. On a
|
||
compatible NVIDIA GPU, optionally install `flash-attn` and pass
|
||
`--flash-attention-2`.
|
||
|
||
The separate B1 local extractor, `extract_turn_local_features.py`, is tested
|
||
with both Transformers 4.52.3 and 4.57.6. An existing 4.57.6 GPU container does
|
||
not need to be downgraded for local extraction. This does not change the replay
|
||
API's dependency assumptions. Keep all local sidecars for one experiment on
|
||
the same version; their metadata records the actual runtime version. See
|
||
[the local feature guide](docs/local_feature_baseline.md).
|
||
|
||
## Ego4D Audio-Visual Diarization download
|
||
|
||
Do not download the complete Ego4D release for this experiment. Download only
|
||
the compressed Audio-Visual Diarization benchmark clips and annotations:
|
||
|
||
```bash
|
||
ego4d \
|
||
--output_directory=/path/to/ego4d \
|
||
--datasets clips annotations \
|
||
--benchmarks AV
|
||
```
|
||
|
||
The approximately 1 TB figure on the Ego4D download page refers to all
|
||
benchmark clips combined, not only the AV subset. The annotated AVD subset has
|
||
about 572 five-minute clips (roughly 50 hours). EgoSpeak uses a custom 346/87
|
||
train/test split, about 36.1 hours total. Reproducing its Table 2 requires the
|
||
exact IDs in EgoSpeak's `EGO4DSHUFFLE` split.
|
||
|
||
## Required validation before a full run
|
||
|
||
Do not launch full EasyCom/Ego4D extraction until all of these pass:
|
||
|
||
- no input audio sample or video frame has timestamp later than prediction
|
||
time `t`;
|
||
- `states` and `target_perframe` lengths/time origins are reconciled and the
|
||
chosen label offset is saved;
|
||
- open-stream cache replay is exactly equivalent to its from-scratch replay on
|
||
short clips;
|
||
- open-stream and causal-prefix predictions are compared on identical clips;
|
||
- a 20-second run fits the 3090 with the selected pixel budget and attention
|
||
implementation;
|
||
- saved outputs contain exactly ten future horizons in the order 0.2--2.0 s;
|
||
- the local metric reproduces EgoSpeak's evaluator on known predictions;
|
||
- test session IDs never enter training, validation, normalization statistics,
|
||
or hyperparameter selection.
|
||
|
||
Once these checks pass, the first publishable experiment is:
|
||
|
||
```text
|
||
frozen Qwen2.5-Omni-3B Thinker
|
||
+ trained 10x3 head
|
||
+ bounded Qwen-style 2-second open streaming
|
||
+ 20-second rolling causal memory
|
||
+ full-session 5 Hz output
|
||
+ exact EgoSpeak split and Table 2 metric
|
||
```
|
||
|
||
Repository workflow verified on 2026-08-19.
|