118 lines
5.4 KiB
Markdown
118 lines
5.4 KiB
Markdown
# EasyCom four-epoch loss-structure ablation
|
|
|
|
Recorded on **2026-08-26** from user-reported remote
|
|
`train_turn_head.py`/`infer_turn_head.py` results. This experiment tests
|
|
whether the three-column EgoSpeak target is better modeled as competing
|
|
softmax classes, independent binary marginals, or four mutually exclusive
|
|
speech-activity states.
|
|
|
|
## Experiment contract
|
|
|
|
- Dataset: EasyCom.
|
|
- Train split: Sessions 4--12, all used for training.
|
|
- Development split: none.
|
|
- Test split: official Sessions 3, 2, and 1.
|
|
- Backbone: frozen Qwen2.5-Omni-3B Thinker.
|
|
- Streaming representation: 8-second open-stream replay, 2 FPS video, final
|
|
four query-layer states stacked at each 5 Hz target step.
|
|
- Readout: `concat-mlp`, followed by the anticipation MLP.
|
|
- Training budget: exactly 4 epochs for every objective.
|
|
- Reported command schedule: Adam, batch size 512, learning rate `7e-5`,
|
|
weight decay `5e-5`, dropout `0.2`, warmup factor `0.3`, and 1.6 warmup
|
|
epochs (the same 40% warmup fraction as the earlier 10/25 protocol).
|
|
- Seeds: 0, 1, and 2.
|
|
- Evaluation: `causal_valid_only` AP/mAP over ten future horizons from 0.2 to
|
|
2.0 seconds.
|
|
|
|
The objectives are:
|
|
|
|
1. `lstr-mce`: three softmax logits with the original multi-positive MCE.
|
|
2. `independent-bce`: three independent sigmoid/BCE marginal labels.
|
|
3. `four-state-ce`: softmax over silence, wearer-only, others-only, and
|
|
overlap; evaluation marginalizes back to the original three labels.
|
|
|
|
## Three-seed summary
|
|
|
|
All values are percentage AP points, reported as mean +/- sample standard
|
|
deviation.
|
|
|
|
| Objective | 0.2s mAP | 0.2s wearer AP | 0.2s others AP | 10-horizon mean mAP |
|
|
|---|---:|---:|---:|---:|
|
|
| LSTR-MCE | 73.70581 +/- 0.23802 | 78.41206 +/- 0.48622 | 85.74830 +/- 0.03661 | 60.81597 +/- 0.06941 |
|
|
| Independent BCE | 73.70702 +/- 0.08593 | 78.58842 +/- 0.16884 | 86.01169 +/- 0.09991 | **60.91733 +/- 0.03540** |
|
|
| Four-state CE | **73.81030 +/- 0.05499** | **78.72040 +/- 0.14753** | **86.03151 +/- 0.06097** | 60.88488 +/- 0.05728 |
|
|
|
|
Relative to LSTR-MCE, four-state CE changes 0.2-second mAP by only
|
|
`+0.10449`, wearer AP by `+0.30834`, others AP by `+0.28321`, and mean mAP by
|
|
`+0.06891`. Independent BCE changes the same metrics by `+0.00121`,
|
|
`+0.17636`, `+0.26339`, and `+0.10136`, respectively.
|
|
|
|
## Individual runs
|
|
|
|
| Run | 0.2s mAP | Background AP | Wearer AP | Others AP | Mean mAP |
|
|
|---|---:|---:|---:|---:|---:|
|
|
| four-state-ce seed 0 | 73.74955 | 56.71375 | 78.57366 | 85.96124 | 60.83261 |
|
|
| four-state-ce seed 1 | 73.85667 | 56.63089 | 78.86871 | 86.07041 | 60.94612 |
|
|
| four-state-ce seed 2 | 73.82467 | 56.69228 | 78.71884 | 86.06288 | 60.87591 |
|
|
| independent-bce seed 0 | 73.62153 | 56.53017 | 78.40881 | 85.92561 | 60.87646 |
|
|
| independent-bce seed 1 | 73.79339 | 56.64809 | 78.74388 | 85.98820 | 60.93730 |
|
|
| independent-bce seed 2 | 73.70613 | 56.38458 | 78.61257 | 86.12125 | 60.93824 |
|
|
| lstr-mce seed 0 | 73.44637 | 56.76438 | 77.86650 | 85.70824 | 60.84861 |
|
|
| lstr-mce seed 1 | 73.75696 | 56.94418 | 78.57004 | 85.75664 | 60.73626 |
|
|
| lstr-mce seed 2 | 73.91410 | 57.16263 | 78.79964 | 85.78001 | 60.86305 |
|
|
|
|
## Seed-0 horizon comparison
|
|
|
|
| Horizon | LSTR-MCE mAP | Independent BCE mAP | Four-state CE mAP |
|
|
|---:|---:|---:|---:|
|
|
| 0.2s | 73.44637 | 73.62153 | 73.74955 |
|
|
| 0.4s | 68.63790 | 68.93990 | 68.82697 |
|
|
| 0.6s | 64.64750 | 64.79984 | 64.80523 |
|
|
| 0.8s | 61.50690 | 61.57880 | 61.52546 |
|
|
| 1.0s | 59.35000 | 59.53538 | 59.44807 |
|
|
| 1.2s | 58.08335 | 57.95764 | 57.96347 |
|
|
| 1.4s | 57.22382 | 56.94317 | 56.95413 |
|
|
| 1.6s | 56.15409 | 56.04943 | 55.84352 |
|
|
| 1.8s | 55.11616 | 55.09841 | 54.89116 |
|
|
| 2.0s | 54.32006 | 54.24044 | 54.31858 |
|
|
| **Mean** | **60.84861** | **60.87646** | **60.83261** |
|
|
|
|
The complete seed-0 class AP curves are stored in [`metrics.csv`](metrics.csv)
|
|
and [`summary.json`](summary.json).
|
|
|
|
## Interpretation
|
|
|
|
This is not a clear aggregate improvement. The two non-competing objectives
|
|
slightly improve wearer/others ranking, but generally trade away background
|
|
ranking. Four-state CE is best for the pre-specified short-horizon wearer
|
|
metric, while independent BCE is marginally best for ten-horizon mean mAP.
|
|
The differences are small relative to seed variation, and four-state CE does
|
|
not establish a decisive advantage over the simpler BCE formulation.
|
|
|
|
The defensible paper statement is therefore:
|
|
|
|
> Non-competing label formulations provide small speech-class gains under a
|
|
> fixed four-epoch budget, but no loss formulation significantly dominates
|
|
> aggregate anticipation mAP.
|
|
|
|
Before interpreting this as an overlap-modeling result, evaluate overlap AP
|
|
and per-session paired differences from the saved predictions. Because no dev
|
|
set was used, these results describe a fixed-compute ablation rather than each
|
|
objective's independently optimized performance. Do not select a primary loss
|
|
post hoc from these test results.
|
|
|
|
## Provenance
|
|
|
|
- Feature root:
|
|
`/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack`
|
|
- Target root:
|
|
`/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame`
|
|
- Remote checkpoint root: `checkpoints/easycom_loss_structure_4ep`
|
|
- Remote result root: `results/easycom_loss_structure_4ep`
|
|
- Full per-horizon output supplied for seed 0 only.
|
|
- Three-seed aggregate and individual-run summaries supplied for all losses.
|
|
- Raw remote checkpoint JSON and `metrics.json` files archived locally: no.
|
|
|
|
Machine-readable records: [`summary.json`](summary.json),
|
|
[`metrics.csv`](metrics.csv), and [`raw_console.txt`](raw_console.txt).
|