Record preliminary transition auxiliary results
This commit is contained in:
Xin Wang
122
results/easycom_transition_auxiliary_4ep_2026-08-27/README.md
Normal file
122
results/easycom_transition_auxiliary_4ep_2026-08-27/README.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# EasyCom transition-aware auxiliary-head ablation
|
||||
|
||||
Recorded on 2026-08-27 from user-reported remote evaluation output. This is a
|
||||
preliminary two-seed snapshot: seed 2 has not been reported, so these numbers
|
||||
must not yet be used as a final multi-seed paper result.
|
||||
|
||||
## Experiment contract
|
||||
|
||||
- Dataset: EasyCom.
|
||||
- Train split: Sessions 4 through 12, with no development set.
|
||||
- Test split: EgoSpeak official Sessions 3, 2, and 1.
|
||||
- Backbone: frozen Qwen2.5-Omni-3B Thinker.
|
||||
- Representation: 8-second open-stream replay, 2 FPS video, 5 Hz states, final
|
||||
four query layers stacked.
|
||||
- Readout: concat-MLP followed by the activity MLP.
|
||||
- Activity objective: LSTR-MCE.
|
||||
- Training budget: 4 epochs, batch size 512, learning rate 7e-5, weight decay
|
||||
5e-5, dropout 0.2, 1.6 warmup epochs, warmup factor 0.3.
|
||||
- Completed seeds: 0 and 1. Missing seed: 2.
|
||||
- Primary evaluation protocol: causal-valid-only activity AP/mAP.
|
||||
|
||||
A positive transition at future horizon h means that the complete three-label
|
||||
activity state changes between target rows t+h-1 and t+h.
|
||||
|
||||
- R1: activity loss + 0.2 times weighted transition BCE, with positive weight
|
||||
3.
|
||||
- R2: R1 loss - 0.1 times exact expected weighted transition correctness.
|
||||
|
||||
## Individual results
|
||||
|
||||
All values below are percentage points.
|
||||
|
||||
| Variant | Seed | 0.2s mAP | BG AP | Wearer AP | Others AP | Mean mAP | Transition 0.2s AP | Transition 0.2s F1@0.5 | Transition mean AP |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| R1 | 0 | 73.51590 | 56.84019 | 77.92448 | 85.78303 | 60.86360 | 15.19153 | 0.46172 | 13.04354 |
|
||||
| R1 | 1 | 73.78692 | 56.92102 | 78.63090 | 85.80884 | 60.82023 | 15.32492 | 1.14068 | 13.06174 |
|
||||
| R2 | 0 | 73.53288 | 56.89701 | 77.92732 | 85.77432 | 60.86528 | 15.37608 | 0.00000 | 13.11854 |
|
||||
| R2 | 1 | 73.82947 | 56.97474 | 78.69340 | 85.82027 | 60.84178 | 15.47356 | 0.84227 | 13.13820 |
|
||||
|
||||
## Two-seed mean and sample standard deviation
|
||||
|
||||
| Variant | 0.2s mAP | Wearer AP | Others AP | Mean mAP | Transition 0.2s AP | Transition mean AP |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| R1 | 73.65141 +/- 0.19164 | 78.27769 +/- 0.49951 | 85.79594 +/- 0.01825 | 60.84192 +/- 0.03067 | 15.25823 +/- 0.09432 | 13.05264 +/- 0.01287 |
|
||||
| R2 | 73.68118 +/- 0.20972 | 78.31036 +/- 0.54169 | 85.79729 +/- 0.03249 | 60.85353 +/- 0.01662 | 15.42482 +/- 0.06893 | 13.12837 +/- 0.01390 |
|
||||
|
||||
## Matched-seed comparison with LSTR-MCE
|
||||
|
||||
The baseline values are taken from the existing four-epoch loss-structure
|
||||
ablation. Using only the same completed seeds 0 and 1 avoids comparing a
|
||||
two-seed transition result with a three-seed baseline mean.
|
||||
|
||||
| Comparison | Delta 0.2s mAP | Delta BG AP | Delta wearer AP | Delta others AP | Delta mean mAP |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| R1 minus LSTR-MCE | +0.04975 | +0.02633 | +0.05942 | +0.06350 | +0.04949 |
|
||||
| R2 minus LSTR-MCE | +0.07951 | +0.08160 | +0.09209 | +0.06486 | +0.06110 |
|
||||
| R2 minus R1 | +0.02977 | +0.05527 | +0.03267 | +0.00136 | +0.01162 |
|
||||
|
||||
All activity changes are much smaller than the observed seed variation. The
|
||||
current evidence therefore does not establish a meaningful activity
|
||||
anticipation improvement.
|
||||
|
||||
## Transition diagnostic
|
||||
|
||||
The 0.2-second transition prevalence is 11.21718%, which is also the expected
|
||||
AP of a random ranking. R1 reaches 15.19--15.32% AP, or 1.35--1.37 times the
|
||||
random baseline. R2 reaches 15.38--15.47%, or 1.37--1.38 times random. The
|
||||
auxiliary head therefore learns weak but non-random transition ranking.
|
||||
|
||||
F1 at threshold 0.5 is nearly zero because recall is only 0--0.58%. This fixed
|
||||
threshold is not a reliable model-quality measure under weighted BCE: positive
|
||||
weight 3 changes probability calibration and remains below the approximately
|
||||
7.9 negative-to-positive ratio implied by the test prevalence. Threshold-free
|
||||
AP is the appropriate diagnostic here. Any deployable threshold must be
|
||||
selected on training-held-out data, not the test set.
|
||||
|
||||
R2 improves transition AP over R1 by only 0.16660 points at 0.2 seconds and
|
||||
0.07573 points on the ten-horizon mean. Its activity mean-mAP improvement over
|
||||
R1 is only 0.01162 points. Exact expected correctness supplies almost the same
|
||||
gradient direction as weighted BCE, so this result is consistent with the two
|
||||
terms being largely redundant rather than R2 learning a new sequential policy.
|
||||
|
||||
## Current interpretation
|
||||
|
||||
This is a preliminary null ablation:
|
||||
|
||||
> Exact 200-ms transition supervision provides weak non-random boundary
|
||||
> ranking but no meaningful improvement in the primary activity-anticipation
|
||||
> metric. Adding exact expected correctness produces negligible further gain.
|
||||
|
||||
This experiment should not be described as interactive reinforcement learning.
|
||||
R2 computes the binary-action expectation analytically on fixed offline labels;
|
||||
it is a low-variance supervised reward control.
|
||||
|
||||
## Required follow-up
|
||||
|
||||
1. Complete and report seed 2 before freezing the table.
|
||||
2. Do not select a transition threshold or tune positive/reward weights on the
|
||||
test sessions.
|
||||
3. If the transition line is continued, replace the sparse exact-boundary
|
||||
target with a pre-specified cumulative transition-hazard target: whether at
|
||||
least one transition occurs within the next h steps.
|
||||
4. If that structured target also fails to improve activity mAP, treat
|
||||
transition auxiliary learning on frozen states as a rejected direction and
|
||||
move experiment budget to representation or temporal modeling.
|
||||
|
||||
## Provenance
|
||||
|
||||
- Feature root:
|
||||
/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
|
||||
- Target root:
|
||||
/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
|
||||
- Remote result roots:
|
||||
results/easycom-transition-r1-4ep-seed-N and
|
||||
results/easycom-transition-r2-4ep-seed-N
|
||||
- Source: user-reported summary and diagnostic console output.
|
||||
- Raw remote metrics JSON archived locally: no.
|
||||
- Full per-horizon activity and transition curves archived locally: no.
|
||||
|
||||
Machine-readable records are in summary.json and metrics.csv. The received
|
||||
console values are preserved in raw_console.txt.
|
||||
|
||||
Reference in New Issue
Block a user