Add causal person track GRU residual experiment
This commit is contained in:
@@ -43,17 +43,27 @@ result.
|
||||
- Ego4D batch and 8-GPU local-feature launchers, with strict frame audits.
|
||||
- Timing-derived wearer floor-state and event diagnostics.
|
||||
|
||||
### Measured screening result
|
||||
|
||||
- F1 Frame-set Attention uses fused G+A as one query over the latest half-open
|
||||
causal person set. On the existing three-seed, four-epoch test screen it
|
||||
obtained `81.77745 +/- 0.15638` 0.2-second mAP and
|
||||
`64.95099 +/- 0.09451` ten-horizon mean mAP. This is stable, but is not a
|
||||
matched improvement over the historical no-dev pooled result; correct/wrong
|
||||
visual controls remain missing.
|
||||
|
||||
### Implemented, awaiting remote runtime verification
|
||||
|
||||
- F1 Frame-set Attention loader/head/train/infer path. It uses fused G+A as
|
||||
one query over the latest half-open causal person set, retains direct A in
|
||||
the head, and dynamically pads people per batch. Do not report it as a
|
||||
measured method until the L20 smoke and full controls pass.
|
||||
- Two current-sidecar screening controls: `audio-person-frame-set` adds a
|
||||
zero-gated audio-to-person read before the F1 query, and
|
||||
`gated-person-state-adapter` adds a zero-gated person residual after the
|
||||
frozen Thinker readout. They are explicitly recorded as F5/F6 screening,
|
||||
not full history-conditioned F5 or an internal Thinker F6 adapter.
|
||||
- `track-gru-residual` builds strictly causal histories from anonymous provided
|
||||
track IDs, applies one shared GRU per track, lets G+A query current track
|
||||
states, and adds a zero-gated correction to an exactly initialized pooled
|
||||
`G+A+V` checkpoint. See
|
||||
[`docs/easycom_track_gru_residual.md`](docs/easycom_track_gru_residual.md).
|
||||
|
||||
### Current blocker
|
||||
|
||||
@@ -69,12 +79,17 @@ semantic event ground truth.
|
||||
|
||||
### Next experiment gates
|
||||
|
||||
The user has temporarily chosen to run the dense-AP model screen while P0 is
|
||||
unfinished. This does not relax the paper claim: event numbers remain proxy
|
||||
diagnostics until competing-risk censoring is fixed.
|
||||
|
||||
1. **P0:** fix/freeze event attribution, splits and scenario definitions.
|
||||
2. **P1:** probe aggregate activity and per-person active-speaker grounding;
|
||||
compare correct, wrong-person and time-shifted visual inputs.
|
||||
3. **P2:** re-evaluate `G`, `G+A`, `G+A+pooled-V` under the frozen event protocol.
|
||||
4. **P3:** begin the progressive person-fusion route with Frame-set Attention,
|
||||
then add spatial or track-temporal structure only after its controls pass.
|
||||
4. **P3:** run the matched pooled/anonymous-track-GRU residual screen, then add
|
||||
broken-ID, wrong-time and remove-person controls before claiming that track
|
||||
continuity matters.
|
||||
5. **P4:** attempt person-memory adapter/Thinker LoRA only if P1/P3 pass.
|
||||
|
||||
The detailed architecture, controls and stopping gates for P3 are frozen in
|
||||
@@ -227,6 +242,7 @@ Do not reorganize them physically without compatibility wrappers.
|
||||
| `infer_turn_head.py` | Save EgoSpeak-compatible predictions and AP/mAP |
|
||||
| `evaluate_floor_state_changes.py` | CPU floor-state/event proxy evaluation |
|
||||
| `summarize_local_baselines.py` | Aggregate local baseline `metrics.json` files |
|
||||
| `run_easycom_track_gru_residual.sh` | Matched pooled and anonymous track-GRU training/inference launcher |
|
||||
| `train_omni_turn_lora.py` | Thinker LoRA training from cached AV |
|
||||
| `eval_omni_turn_lora.py` | Thinker LoRA evaluation |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user