wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add competing-risk evaluation and align ICASSP story

Browse Source
This commit is contained in:
Xin Wang
2026-09-10 11:12:02 +08:00
parent 0abefd6f77
commit 4b7d2c0c20
30 changed files with 3420 additions and 724 deletions
Download Patch File Download Diff File Expand all files Collapse all files

83
README.md
View File

@@ -46,8 +46,8 @@ It separates four questions that must not be conflated:
4. Does any suppression preserve addressed-to-wearer response recall/latency
and wearer yield behavior, rather than making the model globally passive?
The immediate order is: freeze the event protocol and competing-risk
attribution; run speaker-grounding probes and wrong-person/time-shift controls;
The immediate order is: run and audit the implemented competing-risk v2 event
protocol, then freeze it; run speaker-grounding probes and wrong-person/time-shift controls;
re-evaluate `G`, `G+A`, and `G+A+pooled-V`; then train an anonymous person-wise
causal side branch. A person-memory adapter or Thinker LoRA is a second-stage
experiment only after the simpler branch passes those diagnostic gates.
@@ -392,6 +392,40 @@ bits on CPU (pass `--dataset easycom` for EasyCom; default remains Ego4D).
four-epoch heads with no dev set. This native-concatenation baseline is not
capacity- or context-matched to the Thinker baseline.
The first epoch-budget sweep shows severe capacity overfitting. Four epochs
reach **73.42020** at 0.2 seconds and a new frozen-Omni best of **60.87144**
mean mAP, whereas 12 and 50 epochs fall to `57.98809` and `53.03515` despite
lower training losses. The 4-epoch concat-MLP is effectively tied with the
scalar mixture at 0.2 seconds (`-0.03065`) but improves mean mAP by `+0.33721`,
mainly at 1.2--1.6 seconds. Because the runs used different total-length cosine
schedules and the official test sessions were inspected, this is a diagnostic
capacity result rather than a valid final epoch selection. Full tables and
protocol caveats are archived in
[`results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/`](results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/README.md).
### Frozen ROMA last-four-layer transfer
Three frozen ROMA/Qwen2.5-Omni-7B readout runs were recorded on 2026-08-27
using the same 8-second open-stream, 2 FPS, 5 Hz EasyCom evaluation path. A
four-epoch fixed mean over the last four `[3584]` query states is the best ROMA
control at **69.21863** mAP for 0.2 seconds and **58.90402** mean mAP. A
51.4M-parameter concat-MLP reaches only `68.65486 / 56.52425` after four
epochs; relative to the design- and epoch-matched frozen Qwen-3B concat model
it is `-4.76534` at 0.2 seconds and `-4.34719` in mean mAP. This is not
parameter-matched because the 3584-dimensional ROMA fusion is much wider.
This is a negative-transfer diagnostic, not evidence that ROMA is universally
worse: the project used Qwen-style open-stream replay rather than ROMA's native
one-second streaming training recipe. The practical decision is nevertheless
clear: retain frozen Qwen2.5-Omni-3B as the main backbone and stop tuning the
wide ROMA concat head. Full horizon tables, class AP, validity caveats, JSON,
and a formatted workbook are archived in
[`results/easycom_roma_frozen_last4_2026-08-27/`](results/easycom_roma_frozen_last4_2026-08-27/README.md).
The related-work synthesis and proposed next method—recent-200ms payload
readout plus structured future-floor projection—are documented in
[`reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md`](reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md).
## Current Ego4D context-length results
The following single-seed snapshot was recorded on 2026-08-22. Both variants
@@ -1678,6 +1712,44 @@ At 0.2 seconds, sparse supervision alone changes mAP from `71.69695` to
`71.08589`. This isolates the short-horizon regression primarily to Thinker
updating rather than sparse target selection.
### Dense last-8 LoRA result
Restricting LoRA to `q_proj`, `v_proj`, and `o_proj` in Thinker layers 28--35,
reducing the learning rate to `5e-6`, using dense 5-Hz supervision, and freezing
the trained 8-second head substantially reduces but does not reverse negative
transfer. Five continuous epochs were selected on Session 12 by 0.2-second
mAP; epoch 3 / step 11910 was then evaluated once on Sessions 3, 2, and 1.
| Checkpoint | 0.2s full mAP | Full mean mAP | Delta from frozen dense mean |
|---|---:|---:|---:|
| Frozen Thinker dense-cache control | **71.69695** | **58.16091** | — |
| Dense last-8 q/v/o LoRA, frozen head, epoch 1, train 4--12 | 71.62100 | 58.03491 | -0.12600 |
| Dense last-8 q/v/o LoRA, frozen head, epoch 1, train 4--11 | 71.59585 | 58.01500 | -0.14591 |
| Dense last-8 q/v/o LoRA, frozen head, epoch 3, train 4--11 | 71.65000 | 57.88833 | -0.27258 |
| Prior frozen-head initialized sparse LoRA, 3 epochs | 71.08589 | 56.19608 | -1.96483 |
Epoch 1 is only `0.05415` below epoch 3 at 0.2 seconds but is `0.12667` higher
in mean mAP. It reduces the mean regression relative to frozen dense from
`-0.27258` to `-0.14591`, with better results than epoch 3 at every horizon
from 0.6 through 2.0 seconds. This supports fixing one epoch and retraining on
all official training Sessions 4--12 rather than selecting a later checkpoint.
That full-train run reaches `71.62100` at 0.2 seconds and `58.03491` mean,
improving the Sessions 4--11 epoch-1 result by only `0.02515` and `0.01991`.
It remains `0.12600` mean points below frozen dense, confirming that withholding
Session 12 was not the cause of the residual negative transfer. Its complete
record is in
[`results/easycom_dense_last8_qvo_full_train_epoch1_2026-08-25/`](results/easycom_dense_last8_qvo_full_train_epoch1_2026-08-25/README.md).
For epoch 3, the gap grows almost monotonically to `-0.45040` at 2.0 seconds
and camera-wearer AP is the main regression (`-0.62718` average).
The complete development table, per-horizon/class test table, configuration,
and provenance caveats are archived in
[`results/easycom_dense_last8_qvo_frozen_head_epoch3_2026-08-25/`](results/easycom_dense_last8_qvo_frozen_head_epoch3_2026-08-25/README.md).
This result should be reported as a negative-transfer ablation, not as an Omni
improvement. Any subsequent adapter interpolation or objective change must be
chosen without consulting Sessions 1--3; for a clean publication protocol,
the frozen head must also be retrained without the chosen development session.
For a speed-only alternative, `train_omni_turn_lora.py` supports a two-GPU
producer-consumer benchmark. The producer GPU runs the frozen AV towers and
places current-tail embeddings in a bounded CPU queue while the consumer GPU
@@ -1753,9 +1825,10 @@ The headline comparison cannot establish speaker grounding or social
understanding. Experiments therefore proceed through explicit gates:
1. **P0 — protocol correctness:** preserve physical 5 Hz timing and causal
masks, then prevent an other-offset response from being attributed after a
new other onset (and symmetrically censor yield after the next other
offset). Freeze scenario rules without tuning on EasyCom test Sessions 1--3.
masks. The evaluator now prevents an other-offset response from being
attributed at/after a new other onset and symmetrically censors yield at the
next other offset; run the real-data audit and freeze scenario rules without
tuning on EasyCom test Sessions 1--3.
2. **P1 — representation diagnosis:** compare EgoSpeak features, Qwen
encoder-only A/V/AV, the global Thinker state, and anonymous person slots on
aggregate activity and per-person active-speaker probes. Include wrong-person,