EasyCom: sparse Thinker-LoRA weights-only continuation (epoch2 snapshot)
Recorded on 2026-08-24 from the remote EasyCom evaluation console. This is a single-run experiment snapshot, not a multi-seed paper result.
Experiment contract
- Dataset: EasyCom with EgoSpeak's official split.
- Train sessions: 7, 11, 10, 6, 9, 4, 8, 5, and 12.
- Test sessions: 3, 2, and 1 (
n=23048full-timeline rows). - Backbone: Qwen2.5-Omni-3B Thinker with LoRA plus the
10 x 3turn head. - Context: 8 seconds; 2 FPS video; predictions and labels at 5 Hz.
- Training sampling: transition-sparse anchors (ordinary 1 Hz plus dense 5 Hz anchors near target transitions).
- Evaluation: dense 5 Hz AV cache on every test timestamp.
- Continuation: adapter and head weights loaded from the epoch1 checkpoint, but optimizer moments and the learning-rate scheduler were reinitialized. Therefore, "epoch2" here means a weights-only second training run, not an exact optimizer-state resume.
- Reported checkpoint directory:
checkpoints/omni_turn_lora_easycom_sparse_8s_epoch2_ddp4/.
Summary
All values are percentages.
| Checkpoint | Full-timeline mean mAP | Strict causal mean mAP |
|---|---|---|
| Frozen Thinker dense-cache control | 58.16091 | 58.17146 |
| Sparse LoRA epoch1 | 56.52480 | 56.53450 |
| Sparse LoRA weights-only epoch2 | 54.19850 | 54.20711 |
| Epoch2 minus epoch1 | -2.32630 | -2.32739 |
| Epoch2 minus frozen control | -3.96241 | -3.96435 |
The second run is materially worse than both epoch1 and the frozen control. This is not consistent with normal evaluator noise.
Per-horizon results
| Horizon | Full mAP | Strict mAP | Background AP | Wearer AP | Others AP |
|---|---|---|---|---|---|
| 0.2 s | 65.55214 | 65.55385 | 49.03788 | 65.76312 | 81.85542 |
| 0.4 s | 60.96517 | 60.96812 | 43.06922 | 61.14251 | 78.68378 |
| 0.6 s | 56.79766 | 56.80301 | 39.37427 | 55.10528 | 75.91344 |
| 0.8 s | 54.11741 | 54.12457 | 36.94546 | 51.56720 | 73.83955 |
| 1.0 s | 52.09538 | 52.10400 | 35.40706 | 48.38835 | 72.49075 |
| 1.2 s | 51.35516 | 51.36504 | 35.30874 | 46.86261 | 71.89412 |
| 1.4 s | 50.99774 | 51.00914 | 35.29891 | 45.48818 | 72.20613 |
| 1.6 s | 50.47830 | 50.49067 | 34.98109 | 44.74277 | 71.71105 |
| 1.8 s | 50.00999 | 50.02296 | 34.64341 | 43.86161 | 71.52496 |
| 2.0 s | 49.61605 | 49.62970 | 34.26290 | 43.14791 | 71.43735 |
The AP columns above are from the full-timeline protocol. Complete AP values
for both protocols are stored in metrics.csv, and
summary.json stores the machine-readable aggregate result.
Interpretation and decision
The largest class-specific regression is camera-wearer speaking. Relative to epoch1, wearer AP falls from 68.77118 to 65.76312 at 0.2 seconds and from 47.53875 to 43.14791 at 2.0 seconds. Background and other-speaker AP decline less at the long horizon. The pattern indicates negative transfer in the wearer-speaking representation rather than a uniform metric offset.
Likely contributors are sparse-train/dense-evaluation distribution mismatch, too much LoRA capacity for EasyCom, and restarting the learning-rate schedule during weights-only continuation. Do not run a third continuation epoch with the same recipe. Keep the frozen 8-second result as the main representation baseline and next test the parameter-matched frozen overlap-working-memory ablation.
Provenance
- Source: user-reported
eval_omni_turn_lora.pyconsole output. - Raw remote
metrics.jsonarchived here: no; values were transcribed. - Multi-seed: no.
- Publication status: diagnostic negative result only.