wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Record EasyCom loss structure ablation

Browse Source
This commit is contained in:
Xin Wang
2026-08-26 16:30:48 +08:00
parent f35b9108ac
commit 0590df1e88
5 changed files with 325 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

117
results/easycom_loss_structure_4ep_2026-08-26/README.md Normal file
View File

@@ -0,0 +1,117 @@
# EasyCom four-epoch loss-structure ablation
Recorded on **2026-08-26** from user-reported remote
`train_turn_head.py`/`infer_turn_head.py` results. This experiment tests
whether the three-column EgoSpeak target is better modeled as competing
softmax classes, independent binary marginals, or four mutually exclusive
speech-activity states.
## Experiment contract
- Dataset: EasyCom.
- Train split: Sessions 4--12, all used for training.
- Development split: none.
- Test split: official Sessions 3, 2, and 1.
- Backbone: frozen Qwen2.5-Omni-3B Thinker.
- Streaming representation: 8-second open-stream replay, 2 FPS video, final
four query-layer states stacked at each 5 Hz target step.
- Readout: `concat-mlp`, followed by the anticipation MLP.
- Training budget: exactly 4 epochs for every objective.
- Reported command schedule: Adam, batch size 512, learning rate `7e-5`,
weight decay `5e-5`, dropout `0.2`, warmup factor `0.3`, and 1.6 warmup
epochs (the same 40% warmup fraction as the earlier 10/25 protocol).
- Seeds: 0, 1, and 2.
- Evaluation: `causal_valid_only` AP/mAP over ten future horizons from 0.2 to
2.0 seconds.
The objectives are:
1. `lstr-mce`: three softmax logits with the original multi-positive MCE.
2. `independent-bce`: three independent sigmoid/BCE marginal labels.
3. `four-state-ce`: softmax over silence, wearer-only, others-only, and
overlap; evaluation marginalizes back to the original three labels.
## Three-seed summary
All values are percentage AP points, reported as mean +/- sample standard
deviation.
| Objective | 0.2s mAP | 0.2s wearer AP | 0.2s others AP | 10-horizon mean mAP |
|---|---:|---:|---:|---:|
| LSTR-MCE | 73.70581 +/- 0.23802 | 78.41206 +/- 0.48622 | 85.74830 +/- 0.03661 | 60.81597 +/- 0.06941 |
| Independent BCE | 73.70702 +/- 0.08593 | 78.58842 +/- 0.16884 | 86.01169 +/- 0.09991 | **60.91733 +/- 0.03540** |
| Four-state CE | **73.81030 +/- 0.05499** | **78.72040 +/- 0.14753** | **86.03151 +/- 0.06097** | 60.88488 +/- 0.05728 |
Relative to LSTR-MCE, four-state CE changes 0.2-second mAP by only
`+0.10449`, wearer AP by `+0.30834`, others AP by `+0.28321`, and mean mAP by
`+0.06891`. Independent BCE changes the same metrics by `+0.00121`,
`+0.17636`, `+0.26339`, and `+0.10136`, respectively.
## Individual runs
| Run | 0.2s mAP | Background AP | Wearer AP | Others AP | Mean mAP |
|---|---:|---:|---:|---:|---:|
| four-state-ce seed 0 | 73.74955 | 56.71375 | 78.57366 | 85.96124 | 60.83261 |
| four-state-ce seed 1 | 73.85667 | 56.63089 | 78.86871 | 86.07041 | 60.94612 |
| four-state-ce seed 2 | 73.82467 | 56.69228 | 78.71884 | 86.06288 | 60.87591 |
| independent-bce seed 0 | 73.62153 | 56.53017 | 78.40881 | 85.92561 | 60.87646 |
| independent-bce seed 1 | 73.79339 | 56.64809 | 78.74388 | 85.98820 | 60.93730 |
| independent-bce seed 2 | 73.70613 | 56.38458 | 78.61257 | 86.12125 | 60.93824 |
| lstr-mce seed 0 | 73.44637 | 56.76438 | 77.86650 | 85.70824 | 60.84861 |
| lstr-mce seed 1 | 73.75696 | 56.94418 | 78.57004 | 85.75664 | 60.73626 |
| lstr-mce seed 2 | 73.91410 | 57.16263 | 78.79964 | 85.78001 | 60.86305 |
## Seed-0 horizon comparison
| Horizon | LSTR-MCE mAP | Independent BCE mAP | Four-state CE mAP |
|---:|---:|---:|---:|
| 0.2s | 73.44637 | 73.62153 | 73.74955 |
| 0.4s | 68.63790 | 68.93990 | 68.82697 |
| 0.6s | 64.64750 | 64.79984 | 64.80523 |
| 0.8s | 61.50690 | 61.57880 | 61.52546 |
| 1.0s | 59.35000 | 59.53538 | 59.44807 |
| 1.2s | 58.08335 | 57.95764 | 57.96347 |
| 1.4s | 57.22382 | 56.94317 | 56.95413 |
| 1.6s | 56.15409 | 56.04943 | 55.84352 |
| 1.8s | 55.11616 | 55.09841 | 54.89116 |
| 2.0s | 54.32006 | 54.24044 | 54.31858 |
| **Mean** | **60.84861** | **60.87646** | **60.83261** |
The complete seed-0 class AP curves are stored in [`metrics.csv`](metrics.csv)
and [`summary.json`](summary.json).
## Interpretation
This is not a clear aggregate improvement. The two non-competing objectives
slightly improve wearer/others ranking, but generally trade away background
ranking. Four-state CE is best for the pre-specified short-horizon wearer
metric, while independent BCE is marginally best for ten-horizon mean mAP.
The differences are small relative to seed variation, and four-state CE does
not establish a decisive advantage over the simpler BCE formulation.
The defensible paper statement is therefore:
> Non-competing label formulations provide small speech-class gains under a
> fixed four-epoch budget, but no loss formulation significantly dominates
> aggregate anticipation mAP.
Before interpreting this as an overlap-modeling result, evaluate overlap AP
and per-session paired differences from the saved predictions. Because no dev
set was used, these results describe a fixed-compute ablation rather than each
objective's independently optimized performance. Do not select a primary loss
post hoc from these test results.
## Provenance
- Feature root:
`/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack`
- Target root:
`/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame`
- Remote checkpoint root: `checkpoints/easycom_loss_structure_4ep`
- Remote result root: `results/easycom_loss_structure_4ep`
- Full per-horizon output supplied for seed 0 only.
- Three-seed aggregate and individual-run summaries supplied for all losses.
- Raw remote checkpoint JSON and `metrics.json` files archived locally: no.
Machine-readable records: [`summary.json`](summary.json),
[`metrics.csv`](metrics.csv), and [`raw_console.txt`](raw_console.txt).

31
results/easycom_loss_structure_4ep_2026-08-26/metrics.csv Normal file
View File

@@ -0,0 +1,31 @@
loss,seed,horizon_seconds,map,background_ap,camera_wearer_ap,other_people_ap
four-state-ce,0,0.2,73.74955,56.71375,78.57366,85.96124
four-state-ce,0,0.4,68.82697,49.50218,73.81613,83.16261
four-state-ce,0,0.6,64.80523,44.45318,69.01941,80.94310
four-state-ce,0,0.8,61.52546,40.84250,64.65755,79.07634
four-state-ce,0,1.0,59.44807,39.33581,61.01753,77.99088
four-state-ce,0,1.2,57.96347,38.34954,58.54043,77.00044
four-state-ce,0,1.4,56.95413,38.05607,56.56272,76.24359
four-state-ce,0,1.6,55.84352,37.22674,54.59497,75.70885
four-state-ce,0,1.8,54.89116,36.51531,52.91825,75.23991
four-state-ce,0,2.0,54.31858,36.65738,51.36987,74.92851
independent-bce,0,0.2,73.62153,56.53017,78.40881,85.92561
independent-bce,0,0.4,68.93990,49.83756,73.84668,83.13544
independent-bce,0,0.6,64.79984,44.35427,68.98534,81.05992
independent-bce,0,0.8,61.57880,40.95262,64.40600,79.37779
independent-bce,0,1.0,59.53538,39.46612,61.35583,77.78420
independent-bce,0,1.2,57.95764,38.45346,58.30468,77.11479
independent-bce,0,1.4,56.94317,38.20420,56.46439,76.16092
independent-bce,0,1.6,56.04943,37.65077,54.86724,75.63029
independent-bce,0,1.8,55.09841,37.00503,52.91908,75.37111
independent-bce,0,2.0,54.24044,36.46858,51.18916,75.06359
lstr-mce,0,0.2,73.44637,56.76438,77.86650,85.70824
lstr-mce,0,0.4,68.63790,49.90963,73.24394,82.76013
lstr-mce,0,0.6,64.64750,44.77128,68.47873,80.69249
lstr-mce,0,0.8,61.50690,41.37775,63.99127,79.15167
lstr-mce,0,1.0,59.35000,39.43757,61.10979,77.50264
lstr-mce,0,1.2,58.08335,39.28797,58.19467,76.76740
lstr-mce,0,1.4,57.22382,39.01062,56.66806,75.99277
lstr-mce,0,1.6,56.15409,38.03064,54.82056,75.61108
lstr-mce,0,1.8,55.11616,37.20218,53.15030,74.99598
lstr-mce,0,2.0,54.32006,37.21908,51.03765,74.70344
1 loss seed horizon_seconds map background_ap camera_wearer_ap other_people_ap
2 four-state-ce 0 0.2 73.74955 56.71375 78.57366 85.96124
3 four-state-ce 0 0.4 68.82697 49.50218 73.81613 83.16261
4 four-state-ce 0 0.6 64.80523 44.45318 69.01941 80.94310
5 four-state-ce 0 0.8 61.52546 40.84250 64.65755 79.07634
6 four-state-ce 0 1.0 59.44807 39.33581 61.01753 77.99088
7 four-state-ce 0 1.2 57.96347 38.34954 58.54043 77.00044
8 four-state-ce 0 1.4 56.95413 38.05607 56.56272 76.24359
9 four-state-ce 0 1.6 55.84352 37.22674 54.59497 75.70885
10 four-state-ce 0 1.8 54.89116 36.51531 52.91825 75.23991
11 four-state-ce 0 2.0 54.31858 36.65738 51.36987 74.92851
12 independent-bce 0 0.2 73.62153 56.53017 78.40881 85.92561
13 independent-bce 0 0.4 68.93990 49.83756 73.84668 83.13544
14 independent-bce 0 0.6 64.79984 44.35427 68.98534 81.05992
15 independent-bce 0 0.8 61.57880 40.95262 64.40600 79.37779
16 independent-bce 0 1.0 59.53538 39.46612 61.35583 77.78420
17 independent-bce 0 1.2 57.95764 38.45346 58.30468 77.11479
18 independent-bce 0 1.4 56.94317 38.20420 56.46439 76.16092
19 independent-bce 0 1.6 56.04943 37.65077 54.86724 75.63029
20 independent-bce 0 1.8 55.09841 37.00503 52.91908 75.37111
21 independent-bce 0 2.0 54.24044 36.46858 51.18916 75.06359
22 lstr-mce 0 0.2 73.44637 56.76438 77.86650 85.70824
23 lstr-mce 0 0.4 68.63790 49.90963 73.24394 82.76013
24 lstr-mce 0 0.6 64.64750 44.77128 68.47873 80.69249
25 lstr-mce 0 0.8 61.50690 41.37775 63.99127 79.15167
26 lstr-mce 0 1.0 59.35000 39.43757 61.10979 77.50264
27 lstr-mce 0 1.2 58.08335 39.28797 58.19467 76.76740
28 lstr-mce 0 1.4 57.22382 39.01062 56.66806 75.99277
29 lstr-mce 0 1.6 56.15409 38.03064 54.82056 75.61108
30 lstr-mce 0 1.8 55.11616 37.20218 53.15030 74.99598
31 lstr-mce 0 2.0 54.32006 37.21908 51.03765 74.70344

59
results/easycom_loss_structure_4ep_2026-08-26/raw_console.txt Normal file
View File

@@ -0,0 +1,59 @@
Individual runs
run 0.2s mAP BG AP wearer AP others AP mean mAP
four-state-ce_seed0 73.74955 56.71375 78.57366 85.96124 60.83261
four-state-ce_seed1 73.85667 56.63089 78.86871 86.07041 60.94612
four-state-ce_seed2 73.82467 56.69228 78.71884 86.06288 60.87591
independent-bce_seed0 73.62153 56.53017 78.40881 85.92561 60.87646
independent-bce_seed1 73.79339 56.64809 78.74388 85.98820 60.93730
independent-bce_seed2 73.70613 56.38458 78.61257 86.12125 60.93824
lstr-mce_seed0 73.44637 56.76438 77.86650 85.70824 60.84861
lstr-mce_seed1 73.75696 56.94418 78.57004 85.75664 60.73626
lstr-mce_seed2 73.91410 57.16263 78.79964 85.78001 60.86305
Mean +/- sample standard deviation
loss 0.2s mAP wearer AP others AP mean mAP
lstr-mce 73.70581 +/- 0.23802 78.41206 +/- 0.48622 85.74830 +/- 0.03661 60.81597 +/- 0.06941
independent-bce 73.70702 +/- 0.08593 78.58842 +/- 0.16884 86.01169 +/- 0.09991 60.91733 +/- 0.03540
four-state-ce 73.81030 +/- 0.05499 78.72040 +/- 0.14753 86.03151 +/- 0.06097 60.88488 +/- 0.05728
loss: four-state-ce
output space: four-state-softmax
0.2s: mAP=73.74955, background=56.71375, wearer=78.57366, others=85.96124
0.4s: mAP=68.82697, background=49.50218, wearer=73.81613, others=83.16261
0.6s: mAP=64.80523, background=44.45318, wearer=69.01941, others=80.94310
0.8s: mAP=61.52546, background=40.84250, wearer=64.65755, others=79.07634
1.0s: mAP=59.44807, background=39.33581, wearer=61.01753, others=77.99088
1.2s: mAP=57.96347, background=38.34954, wearer=58.54043, others=77.00044
1.4s: mAP=56.95413, background=38.05607, wearer=56.56272, others=76.24359
1.6s: mAP=55.84352, background=37.22674, wearer=54.59497, others=75.70885
1.8s: mAP=54.89116, background=36.51531, wearer=52.91825, others=75.23991
2.0s: mAP=54.31858, background=36.65738, wearer=51.36987, others=74.92851
10-horizon mean mAP: 60.83261
loss: independent-bce
output space: three-label-sigmoid
0.2s: mAP=73.62153, background=56.53017, wearer=78.40881, others=85.92561
0.4s: mAP=68.93990, background=49.83756, wearer=73.84668, others=83.13544
0.6s: mAP=64.79984, background=44.35427, wearer=68.98534, others=81.05992
0.8s: mAP=61.57880, background=40.95262, wearer=64.40600, others=79.37779
1.0s: mAP=59.53538, background=39.46612, wearer=61.35583, others=77.78420
1.2s: mAP=57.95764, background=38.45346, wearer=58.30468, others=77.11479
1.4s: mAP=56.94317, background=38.20420, wearer=56.46439, others=76.16092
1.6s: mAP=56.04943, background=37.65077, wearer=54.86724, others=75.63029
1.8s: mAP=55.09841, background=37.00503, wearer=52.91908, others=75.37111
2.0s: mAP=54.24044, background=36.46858, wearer=51.18916, others=75.06359
10-horizon mean mAP: 60.87646
loss: lstr-mce
output space: three-class-softmax
0.2s: mAP=73.44637, background=56.76438, wearer=77.86650, others=85.70824
0.4s: mAP=68.63790, background=49.90963, wearer=73.24394, others=82.76013
0.6s: mAP=64.64750, background=44.77128, wearer=68.47873, others=80.69249
0.8s: mAP=61.50690, background=41.37775, wearer=63.99127, others=79.15167
1.0s: mAP=59.35000, background=39.43757, wearer=61.10979, others=77.50264
1.2s: mAP=58.08335, background=39.28797, wearer=58.19467, others=76.76740
1.4s: mAP=57.22382, background=39.01062, wearer=56.66806, others=75.99277
1.6s: mAP=56.15409, background=38.03064, wearer=54.82056, others=75.61108
1.8s: mAP=55.11616, background=37.20218, wearer=53.15030, others=74.99598
2.0s: mAP=54.32006, background=37.21908, wearer=51.03765, others=74.70344
10-horizon mean mAP: 60.84861

105
results/easycom_loss_structure_4ep_2026-08-26/summary.json Normal file
View File

@@ -0,0 +1,105 @@
{
"schema_version": 1,
"recorded_date": "2026-08-26",
"status": "three_seed_fixed_budget_ablation",
"dataset": "EasyCom",
"split": {
"train_sessions": [4, 5, 6, 7, 8, 9, 10, 11, 12],
"dev_sessions": [],
"test_sessions": [3, 2, 1]
},
"model": {
"backbone": "Qwen2.5-Omni-3B",
"thinker_frozen": true,
"streaming_mode": "open-stream-replay",
"memory_seconds": 8,
"video_fps": 2,
"prediction_hz": 5,
"selected_layers": "last_4_query_states",
"layer_fusion": "concat-mlp"
},
"training": {
"epochs": 4,
"batch_size": 512,
"learning_rate": 0.00007,
"weight_decay": 0.00005,
"warmup_epochs": 1.6,
"warmup_factor": 0.3,
"dropout": 0.2,
"seeds": [0, 1, 2],
"checkpoint_selection": "final_epoch_no_dev"
},
"evaluation": {
"protocol": "causal_valid_only",
"test_sessions": [3, 2, 1],
"horizon_seconds": [0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0]
},
"three_seed_summary_percent": {
"lstr-mce": {
"horizon_0.2_map": {"mean": 73.70581, "sample_std": 0.23802},
"horizon_0.2_wearer_ap": {"mean": 78.41206, "sample_std": 0.48622},
"horizon_0.2_others_ap": {"mean": 85.74830, "sample_std": 0.03661},
"ten_horizon_mean_map": {"mean": 60.81597, "sample_std": 0.06941}
},
"independent-bce": {
"horizon_0.2_map": {"mean": 73.70702, "sample_std": 0.08593},
"horizon_0.2_wearer_ap": {"mean": 78.58842, "sample_std": 0.16884},
"horizon_0.2_others_ap": {"mean": 86.01169, "sample_std": 0.09991},
"ten_horizon_mean_map": {"mean": 60.91733, "sample_std": 0.03540}
},
"four-state-ce": {
"horizon_0.2_map": {"mean": 73.81030, "sample_std": 0.05499},
"horizon_0.2_wearer_ap": {"mean": 78.72040, "sample_std": 0.14753},
"horizon_0.2_others_ap": {"mean": 86.03151, "sample_std": 0.06097},
"ten_horizon_mean_map": {"mean": 60.88488, "sample_std": 0.05728}
}
},
"individual_run_summary_percent": [
{"loss": "four-state-ce", "seed": 0, "horizon_0.2_map": 73.74955, "horizon_0.2_background_ap": 56.71375, "horizon_0.2_wearer_ap": 78.57366, "horizon_0.2_others_ap": 85.96124, "ten_horizon_mean_map": 60.83261},
{"loss": "four-state-ce", "seed": 1, "horizon_0.2_map": 73.85667, "horizon_0.2_background_ap": 56.63089, "horizon_0.2_wearer_ap": 78.86871, "horizon_0.2_others_ap": 86.07041, "ten_horizon_mean_map": 60.94612},
{"loss": "four-state-ce", "seed": 2, "horizon_0.2_map": 73.82467, "horizon_0.2_background_ap": 56.69228, "horizon_0.2_wearer_ap": 78.71884, "horizon_0.2_others_ap": 86.06288, "ten_horizon_mean_map": 60.87591},
{"loss": "independent-bce", "seed": 0, "horizon_0.2_map": 73.62153, "horizon_0.2_background_ap": 56.53017, "horizon_0.2_wearer_ap": 78.40881, "horizon_0.2_others_ap": 85.92561, "ten_horizon_mean_map": 60.87646},
{"loss": "independent-bce", "seed": 1, "horizon_0.2_map": 73.79339, "horizon_0.2_background_ap": 56.64809, "horizon_0.2_wearer_ap": 78.74388, "horizon_0.2_others_ap": 85.98820, "ten_horizon_mean_map": 60.93730},
{"loss": "independent-bce", "seed": 2, "horizon_0.2_map": 73.70613, "horizon_0.2_background_ap": 56.38458, "horizon_0.2_wearer_ap": 78.61257, "horizon_0.2_others_ap": 86.12125, "ten_horizon_mean_map": 60.93824},
{"loss": "lstr-mce", "seed": 0, "horizon_0.2_map": 73.44637, "horizon_0.2_background_ap": 56.76438, "horizon_0.2_wearer_ap": 77.86650, "horizon_0.2_others_ap": 85.70824, "ten_horizon_mean_map": 60.84861},
{"loss": "lstr-mce", "seed": 1, "horizon_0.2_map": 73.75696, "horizon_0.2_background_ap": 56.94418, "horizon_0.2_wearer_ap": 78.57004, "horizon_0.2_others_ap": 85.75664, "ten_horizon_mean_map": 60.73626},
{"loss": "lstr-mce", "seed": 2, "horizon_0.2_map": 73.91410, "horizon_0.2_background_ap": 57.16263, "horizon_0.2_wearer_ap": 78.79964, "horizon_0.2_others_ap": 85.78001, "ten_horizon_mean_map": 60.86305}
],
"seed0_horizon_metrics_percent": {
"four-state-ce": {
"map": [73.74955, 68.82697, 64.80523, 61.52546, 59.44807, 57.96347, 56.95413, 55.84352, 54.89116, 54.31858],
"background_ap": [56.71375, 49.50218, 44.45318, 40.84250, 39.33581, 38.34954, 38.05607, 37.22674, 36.51531, 36.65738],
"camera_wearer_ap": [78.57366, 73.81613, 69.01941, 64.65755, 61.01753, 58.54043, 56.56272, 54.59497, 52.91825, 51.36987],
"other_people_ap": [85.96124, 83.16261, 80.94310, 79.07634, 77.99088, 77.00044, 76.24359, 75.70885, 75.23991, 74.92851],
"mean_map": 60.83261
},
"independent-bce": {
"map": [73.62153, 68.93990, 64.79984, 61.57880, 59.53538, 57.95764, 56.94317, 56.04943, 55.09841, 54.24044],
"background_ap": [56.53017, 49.83756, 44.35427, 40.95262, 39.46612, 38.45346, 38.20420, 37.65077, 37.00503, 36.46858],
"camera_wearer_ap": [78.40881, 73.84668, 68.98534, 64.40600, 61.35583, 58.30468, 56.46439, 54.86724, 52.91908, 51.18916],
"other_people_ap": [85.92561, 83.13544, 81.05992, 79.37779, 77.78420, 77.11479, 76.16092, 75.63029, 75.37111, 75.06359],
"mean_map": 60.87646
},
"lstr-mce": {
"map": [73.44637, 68.63790, 64.64750, 61.50690, 59.35000, 58.08335, 57.22382, 56.15409, 55.11616, 54.32006],
"background_ap": [56.76438, 49.90963, 44.77128, 41.37775, 39.43757, 39.28797, 39.01062, 38.03064, 37.20218, 37.21908],
"camera_wearer_ap": [77.86650, 73.24394, 68.47873, 63.99127, 61.10979, 58.19467, 56.66806, 54.82056, 53.15030, 51.03765],
"other_people_ap": [85.70824, 82.76013, 80.69249, 79.15167, 77.50264, 76.76740, 75.99277, 75.61108, 74.99598, 74.70344],
"mean_map": 60.84861
}
},
"interpretation": {
"clear_overall_winner": false,
"best_three_seed_horizon_0.2_wearer_ap": "four-state-ce",
"best_three_seed_ten_horizon_mean_map": "independent-bce",
"next_required_diagnostic": "overlap AP and per-session paired differences"
},
"provenance": {
"source": "user-reported remote console and summary output",
"remote_feature_root": "/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack",
"remote_target_root": "/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame",
"remote_checkpoint_root": "checkpoints/easycom_loss_structure_4ep",
"remote_result_root": "results/easycom_loss_structure_4ep",
"raw_remote_metrics_archived": false
}
}