EasyCom four-epoch loss-structure ablation

Recorded on 2026-08-26 from user-reported remote train_turn_head.py/infer_turn_head.py results. This experiment tests whether the three-column EgoSpeak target is better modeled as competing softmax classes, independent binary marginals, or four mutually exclusive speech-activity states.

Experiment contract

Dataset: EasyCom.
Train split: Sessions 4--12, all used for training.
Development split: none.
Test split: official Sessions 3, 2, and 1.
Backbone: frozen Qwen2.5-Omni-3B Thinker.
Streaming representation: 8-second open-stream replay, 2 FPS video, final four query-layer states stacked at each 5 Hz target step.
Readout: concat-mlp, followed by the anticipation MLP.
Training budget: exactly 4 epochs for every objective.
Reported command schedule: Adam, batch size 512, learning rate 7e-5, weight decay 5e-5, dropout 0.2, warmup factor 0.3, and 1.6 warmup epochs (the same 40% warmup fraction as the earlier 10/25 protocol).
Seeds: 0, 1, and 2.
Evaluation: causal_valid_only AP/mAP over ten future horizons from 0.2 to 2.0 seconds.

The objectives are:

lstr-mce: three softmax logits with the original multi-positive MCE.
independent-bce: three independent sigmoid/BCE marginal labels.
four-state-ce: softmax over silence, wearer-only, others-only, and overlap; evaluation marginalizes back to the original three labels.

Three-seed summary

All values are percentage AP points, reported as mean +/- sample standard deviation.

Objective	0.2s mAP	0.2s wearer AP	0.2s others AP	10-horizon mean mAP
LSTR-MCE	73.70581 +/- 0.23802	78.41206 +/- 0.48622	85.74830 +/- 0.03661	60.81597 +/- 0.06941
Independent BCE	73.70702 +/- 0.08593	78.58842 +/- 0.16884	86.01169 +/- 0.09991	60.91733 +/- 0.03540
Four-state CE	73.81030 +/- 0.05499	78.72040 +/- 0.14753	86.03151 +/- 0.06097	60.88488 +/- 0.05728

Relative to LSTR-MCE, four-state CE changes 0.2-second mAP by only +0.10449, wearer AP by +0.30834, others AP by +0.28321, and mean mAP by +0.06891. Independent BCE changes the same metrics by +0.00121, +0.17636, +0.26339, and +0.10136, respectively.

Individual runs

Run	0.2s mAP	Background AP	Wearer AP	Others AP	Mean mAP
four-state-ce seed 0	73.74955	56.71375	78.57366	85.96124	60.83261
four-state-ce seed 1	73.85667	56.63089	78.86871	86.07041	60.94612
four-state-ce seed 2	73.82467	56.69228	78.71884	86.06288	60.87591
independent-bce seed 0	73.62153	56.53017	78.40881	85.92561	60.87646
independent-bce seed 1	73.79339	56.64809	78.74388	85.98820	60.93730
independent-bce seed 2	73.70613	56.38458	78.61257	86.12125	60.93824
lstr-mce seed 0	73.44637	56.76438	77.86650	85.70824	60.84861
lstr-mce seed 1	73.75696	56.94418	78.57004	85.75664	60.73626
lstr-mce seed 2	73.91410	57.16263	78.79964	85.78001	60.86305

Seed-0 horizon comparison

Horizon	LSTR-MCE mAP	Independent BCE mAP	Four-state CE mAP
0.2s	73.44637	73.62153	73.74955
0.4s	68.63790	68.93990	68.82697
0.6s	64.64750	64.79984	64.80523
0.8s	61.50690	61.57880	61.52546
1.0s	59.35000	59.53538	59.44807
1.2s	58.08335	57.95764	57.96347
1.4s	57.22382	56.94317	56.95413
1.6s	56.15409	56.04943	55.84352
1.8s	55.11616	55.09841	54.89116
2.0s	54.32006	54.24044	54.31858
Mean	60.84861	60.87646	60.83261

The complete seed-0 class AP curves are stored in metrics.csv and summary.json.

Interpretation

This is not a clear aggregate improvement. The two non-competing objectives slightly improve wearer/others ranking, but generally trade away background ranking. Four-state CE is best for the pre-specified short-horizon wearer metric, while independent BCE is marginally best for ten-horizon mean mAP. The differences are small relative to seed variation, and four-state CE does not establish a decisive advantage over the simpler BCE formulation.

The defensible paper statement is therefore:

Non-competing label formulations provide small speech-class gains under a fixed four-epoch budget, but no loss formulation significantly dominates aggregate anticipation mAP.

Before interpreting this as an overlap-modeling result, evaluate overlap AP and per-session paired differences from the saved predictions. Because no dev set was used, these results describe a fixed-compute ablation rather than each objective's independently optimized performance. Do not select a primary loss post hoc from these test results.

Provenance

Feature root: /data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
Target root: /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
Remote checkpoint root: checkpoints/easycom_loss_structure_4ep
Remote result root: results/easycom_loss_structure_4ep
Full per-horizon output supplied for seed 0 only.
Three-seed aggregate and individual-run summaries supplied for all losses.
Raw remote checkpoint JSON and metrics.json files archived locally: no.

Machine-readable records: summary.json, metrics.csv, and raw_console.txt.

5.4 KiB Raw Blame History