Add causal person track GRU residual experiment

This commit is contained in:
Xin Wang
2026-09-07 00:30:23 +08:00
parent 9fc4577aea
commit 4aed4697a5
15 changed files with 1185 additions and 43 deletions

View File

@@ -43,17 +43,27 @@ result.
- Ego4D batch and 8-GPU local-feature launchers, with strict frame audits.
- Timing-derived wearer floor-state and event diagnostics.
### Measured screening result
- F1 Frame-set Attention uses fused G+A as one query over the latest half-open
causal person set. On the existing three-seed, four-epoch test screen it
obtained `81.77745 +/- 0.15638` 0.2-second mAP and
`64.95099 +/- 0.09451` ten-horizon mean mAP. This is stable, but is not a
matched improvement over the historical no-dev pooled result; correct/wrong
visual controls remain missing.
### Implemented, awaiting remote runtime verification
- F1 Frame-set Attention loader/head/train/infer path. It uses fused G+A as
one query over the latest half-open causal person set, retains direct A in
the head, and dynamically pads people per batch. Do not report it as a
measured method until the L20 smoke and full controls pass.
- Two current-sidecar screening controls: `audio-person-frame-set` adds a
zero-gated audio-to-person read before the F1 query, and
`gated-person-state-adapter` adds a zero-gated person residual after the
frozen Thinker readout. They are explicitly recorded as F5/F6 screening,
not full history-conditioned F5 or an internal Thinker F6 adapter.
- `track-gru-residual` builds strictly causal histories from anonymous provided
track IDs, applies one shared GRU per track, lets G+A query current track
states, and adds a zero-gated correction to an exactly initialized pooled
`G+A+V` checkpoint. See
[`docs/easycom_track_gru_residual.md`](docs/easycom_track_gru_residual.md).
### Current blocker
@@ -69,12 +79,17 @@ semantic event ground truth.
### Next experiment gates
The user has temporarily chosen to run the dense-AP model screen while P0 is
unfinished. This does not relax the paper claim: event numbers remain proxy
diagnostics until competing-risk censoring is fixed.
1. **P0:** fix/freeze event attribution, splits and scenario definitions.
2. **P1:** probe aggregate activity and per-person active-speaker grounding;
compare correct, wrong-person and time-shifted visual inputs.
3. **P2:** re-evaluate `G`, `G+A`, `G+A+pooled-V` under the frozen event protocol.
4. **P3:** begin the progressive person-fusion route with Frame-set Attention,
then add spatial or track-temporal structure only after its controls pass.
4. **P3:** run the matched pooled/anonymous-track-GRU residual screen, then add
broken-ID, wrong-time and remove-person controls before claiming that track
continuity matters.
5. **P4:** attempt person-memory adapter/Thinker LoRA only if P1/P3 pass.
The detailed architecture, controls and stopping gates for P3 are frozen in
@@ -227,6 +242,7 @@ Do not reorganize them physically without compatibility wrappers.
| `infer_turn_head.py` | Save EgoSpeak-compatible predictions and AP/mAP |
| `evaluate_floor_state_changes.py` | CPU floor-state/event proxy evaluation |
| `summarize_local_baselines.py` | Aggregate local baseline `metrics.json` files |
| `run_easycom_track_gru_residual.sh` | Matched pooled and anonymous track-GRU training/inference launcher |
| `train_omni_turn_lora.py` | Thinker LoRA training from cached AV |
| `eval_omni_turn_lora.py` | Thinker LoRA evaluation |