EasyCom frozen Qwen2.5-Omni-3B last-4 learned mixture
Recorded on 2026-08-26 from the user-reported remote training and
infer_turn_head.py output. This is a single-run diagnostic result, not yet a
multi-seed paper number.
Experiment contract
- Dataset: EasyCom.
- Test split: confirmed as EgoSpeak official Sessions 3, 2, and 1. Training is recorded as the script's default official Sessions 4--12; verify this against the remote checkpoint JSON because the supplied excerpt starts at epoch 24.
- Backbone: frozen Qwen2.5-Omni-3B Thinker.
- Streaming mode: open-stream replay with an 8-second causal-memory budget.
- Input sampling: 2 FPS video and raw audio.
- Prediction rate: 5 Hz, with ten future horizons from 0.2 to 2.0 seconds.
- Readout: the query states from the final four Thinker layers, ordered oldest
to newest, followed by a learned global softmax layer mixture and the same
10 x 3anticipation MLP. - Training: head and four scalar layer logits only; Thinker remains frozen.
- Evaluator: EgoSpeak full-timeline per-frame AP/mAP.
The remote artifact is named
easycom_frozen_3b_8s_last4_learned_mix_50ep.pth, but the supplied log and the
user report both show that the actual run ended at epoch 25/25. The
50ep suffix is therefore a naming error and must not be used as provenance.
Summary
All values below are percentages. The comparison baseline is the previously recorded frozen 3B, 8-second, open-stream, single-final-layer run evaluated on the same official test sessions.
| Readout | 0.2s full mAP | Full mean mAP | Delta at 0.2s | Delta mean |
|---|---|---|---|---|
| Single final-layer query | 71.99768 | 58.51937 | -- | -- |
| Last-4 learned mixture | 73.45085 | 60.53423 | +1.45317 | +2.01486 |
The last-4 readout improves every evaluated horizon. The largest gain is
+2.41084 at 1.0 seconds; the smallest is still a positive +1.45317 at
0.2 seconds.
Per-horizon full-timeline results
| Horizon | Last-4 mAP | Single-layer mAP | Delta | Background AP | Wearer AP | Others AP |
|---|---|---|---|---|---|---|
| 0.2s | 73.45085 | 71.99768 | +1.45317 | 56.69385 | 78.35111 | 85.30760 |
| 0.4s | 68.58645 | 66.74289 | +1.84356 | 49.77891 | 73.48058 | 82.49988 |
| 0.6s | 64.44213 | 62.38972 | +2.05241 | 44.51563 | 68.39317 | 80.41758 |
| 0.8s | 61.34852 | 59.12229 | +2.22623 | 40.98328 | 64.20310 | 78.85917 |
| 1.0s | 59.12464 | 56.71380 | +2.41084 | 38.99833 | 60.75541 | 77.62018 |
| 1.2s | 57.53667 | 55.36300 | +2.17367 | 37.76815 | 58.15110 | 76.69076 |
| 1.4s | 56.43710 | 54.21297 | +2.22413 | 37.22701 | 56.12642 | 75.95786 |
| 1.6s | 55.42035 | 53.43221 | +1.98814 | 36.72606 | 54.14045 | 75.39454 |
| 1.8s | 54.76334 | 52.84703 | +1.91631 | 36.45931 | 52.80226 | 75.02845 |
| 2.0s | 54.23227 | 52.37207 | +1.86020 | 36.45292 | 51.56488 | 74.67902 |
| Mean | 60.53423 | 58.51937 | +2.01486 | 41.56035 | 61.79685 | 78.24550 |
Averaged over horizons, the AP gains relative to the single-layer baseline are
+1.92941 for background, +2.88478 for camera-wearer speaking, and
+1.23041 for other-people speaking. The wearer class contributes the largest
improvement, which is encouraging for the short-horizon turn-taking objective.
Learned layer mixture
| Layer order | Weight |
|---|---|
| Oldest of selected four | 0.246384 |
| Second | 0.248327 |
| Third | 0.241774 |
| Final layer | 0.263515 |
The weights remain close to uniform: normalized entropy is 0.999619, the L1
distance from [0.25, 0.25, 0.25, 0.25] is only 0.02703, and the largest
weight exceeds the smallest by only 0.021741. Consequently, this experiment
supports multi-layer query readout, but does not yet show that learning the
mixture is better than a fixed last-4 mean.
Interpretation and next controls
- This is the strongest recorded frozen-Omni EasyCom result in the project so
far:
73.45085at 0.2 seconds and60.53423mean mAP. - The gain occurs at all horizons and all three classes, rather than coming from a single AP outlier.
- Because the learned weights are nearly uniform, the first required ablation is a fixed uniform last-4 mean with the same feature files, head recipe, epoch count, and seed.
- Then compare each of the four individual query layers. This separates layer ensembling from access to a specifically better intermediate layer.
- Repeat the single-layer, fixed-mean, and learned-mixture variants with at least three identical seeds before using the gain as a paper result.
- The supplied output contains no strict causal-valid table. Archive the
remote
metrics.jsonor rerun the evaluator before the publication table. - The single-layer comparison came from an earlier run; its random seed and head-training duration must be matched in the fixed-mean ablation.
Provenance
- Remote feature root:
/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack - Remote checkpoint:
checkpoints/easycom_frozen_3b_8s_last4_learned_mix_50ep.pth - Remote result directory:
results/easycom_frozen_3b_8s_last4_learned_mix_50ep - Final training loss:
0.822849. - Raw console excerpt:
raw_console.txt. - Machine-readable values:
summary.jsonandmetrics.csv. - Raw remote
metrics.jsonarchived locally: no. - Exact training-session list verified from the remote checkpoint JSON: no.
- Multi-seed: no.