Add causal oracle person information screen

This commit is contained in:
Xin Wang
2026-09-07 08:41:34 +08:00
parent fb22108c57
commit ecf00185f1
17 changed files with 1364 additions and 20 deletions

View File

@@ -52,7 +52,7 @@ result.
matched improvement over the historical no-dev pooled result; correct/wrong
visual controls remain missing.
### Implemented, awaiting remote runtime verification
### Measured local-fusion screens
- Two current-sidecar screening controls: `audio-person-frame-set` adds a
zero-gated audio-to-person read before the F1 query, and
@@ -62,10 +62,15 @@ result.
- `track-gru-residual` builds strictly causal histories from anonymous provided
track IDs, applies one shared GRU per track, lets G+A query current track
states, and adds a zero-gated correction to an exactly initialized pooled
`G+A+V` checkpoint. Its three focused loader/head/CLI round-trip tests passed
on the remote PyTorch/L20 environment on 2026-09-07; the one-session GPU
smoke and measured experiment are still pending. See
[`docs/easycom_track_gru_residual.md`](docs/easycom_track_gru_residual.md).
`G+A+V` checkpoint. The matched Session-12 three-seed screen was null
(`+0.01306 +/- 0.02368` percentage points in ten-horizon mean mAP). A seed-0
Sessions-1--3 audit reproduced pooled predictions to within `-0.00030`
points. Saved residual gates were only `-0.0015` to `+0.0108`; because a zero
scalar gate also blocks the initial branch gradient, this is recorded as an
optimization failure rather than evidence that track continuity is useless.
See the immutable
[`result record`](results/easycom_track_gru_residual_4ep_2026-09-07/README.md)
and [`implementation note`](docs/easycom_track_gru_residual.md).
### Current blocker
@@ -89,9 +94,10 @@ diagnostics until competing-risk censoring is fixed.
2. **P1:** probe aggregate activity and per-person active-speaker grounding;
compare correct, wrong-person and time-shifted visual inputs.
3. **P2:** re-evaluate `G`, `G+A`, `G+A+pooled-V` under the frozen event protocol.
4. **P3:** run the matched pooled/anonymous-track-GRU residual screen, then add
broken-ID, wrong-time and remove-person controls before claiming that track
continuity matters.
4. **P3:** repair the matched anonymous-track-GRU residual optimization by
replacing the scalar zero gate with a zero-initialized residual output
layer, and verify nonzero branch gradients on Session 12. Add broken-ID,
wrong-time and remove-person controls only if that corrected screen improves.
5. **P4:** attempt person-memory adapter/Thinker LoRA only if P1/P3 pass.
The detailed architecture, controls and stopping gates for P3 are frozen in
@@ -228,6 +234,7 @@ Do not reorganize them physically without compatibility wrappers.
| `derive_easycom_interaction_events.py` | Deterministic transcript/timing proxies |
| `adjudicate_easycom_interaction_events.py` | Optional ambiguous-event adjudication |
| `audit_easycom_floor_events.py` | Count EOT/takeover/backchannel candidates |
| `prepare_easycom_oracle_features.py` | Build explicit past-only oracle person-activity sidecars for information-value screening |
### Local features
@@ -245,6 +252,7 @@ Do not reorganize them physically without compatibility wrappers.
| `evaluate_floor_state_changes.py` | CPU floor-state/event proxy evaluation |
| `summarize_local_baselines.py` | Aggregate local baseline `metrics.json` files |
| `run_easycom_track_gru_residual.sh` | Matched pooled and anonymous track-GRU training/inference launcher |
| `run_easycom_oracle_information.sh` | Prepare and screen aggregate/per-person/shuffled/delayed oracle conditions on Session 12 |
| `train_omni_turn_lora.py` | Thinker LoRA training from cached AV |
| `eval_omni_turn_lora.py` | Thinker LoRA evaluation |