EasyCom causal person adapter V2

Status and question

Implemented and CPU-testable as --local-fusion causal-person-adapter-v2; GPU results have not yet been measured. The mode is an optimization-corrected, incremental experiment after the matched pooled G+A+V baseline. It does not yet establish that track identity or visual grounding improves wearer floor behavior.

The experiment asks whether strictly causal, per-track visual histories contain information beyond equal person/frame pooling. It deliberately preserves the legacy track-gru-residual mode so its earlier null screen remains reproducible. The controlled causal-person-adapter-egospeak-gru ablation keeps this complete V2 fusion path and changes only the per-track temporal stem; see easycom_egospeak_style_person_gru.md.

Architecture

For 5 Hz row t, the media cutoff is (t+1)/5 seconds. The local loader selects only person frames strictly earlier than that cutoff. Each currently visible anonymous track contributes at most K observations:

p_i,t = shared_GRU(project(v_i,t-K+1:t))
q_t   = project(G_t) + project(A_t)
c_t   = MHA(q_t, [NULL, p_1,t, ..., p_N,t])
l_t   = MLP([c_t, projected_A_t, person_valid, audio_valid])
gate  = sigmoid(MLP([pooled_base_hidden_t, l_t]))
logits_t = pooled_logits_t + person_valid * W_delta(gate * l_t)

W_delta.weight and W_delta.bias are zero initialized. After copying a matched pooled checkpoint, V2 therefore starts with logits exactly equal to the pooled model. The first backward pass updates W_delta; subsequent passes send gradients into the shared GRU, attention and dynamic gate. This fixes the legacy mode's zero scalar gate, which blocked all upstream branch gradients at initialization.

Track IDs are routing metadata only and never become embeddings. Parameters are shared across people, the attention uses a real-person mask plus a learned NULL person, source-clip/max-gap boundaries reset links, and changing padded values or permuting the current person order cannot change predictions. At the 2 FPS local grid, this runbook uses K=4, covering at most two seconds. Repeated 5 Hz queries recompute/read the same causal history; they do not append duplicate visual observations.

This immediate V2 keeps pooled visual evidence in the exact base. If it passes, a later replacement ablation must compare G+A+person-track without pooled V; otherwise the final method cannot claim unconditional pooling is unnecessary.

L20 smoke test

The smoke test uses only training sessions and writes isolated outputs:

EPOCHS=1 \
WARMUP_EPOCHS=0 \
SESSIONS="4 5" \
DEV_SESSIONS=5 \
SCREEN_SESSIONS=5 \
CHECKPOINT_ROOT=checkpoints/easycom_person_adapter_v2_smoke \
RESULT_ROOT=results/easycom_person_adapter_v2_smoke \
CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_causal_person_adapter_v2.sh all 0

Confirm that:

the pooled checkpoint is accepted as an exact base;
person_adapter_diagnostics.residual_output_weight_norm becomes nonzero;
delta_logit_mean_abs becomes nonzero;
shapes and valid counts match the pooled run;
no Session 1--3 output is produced.

Three-seed Session-12 screen

Run one seed per GPU in separate foreground terminals. Filenames are disjoint, so the runs may share the same roots:

CUDA_VISIBLE_DEVICES=0 bash run_easycom_causal_person_adapter_v2.sh all 0
CUDA_VISIBLE_DEVICES=1 bash run_easycom_causal_person_adapter_v2.sh all 1
CUDA_VISIBLE_DEVICES=2 bash run_easycom_causal_person_adapter_v2.sh all 2

Summarize after all processes finish:

python summarize_local_baselines.py \
  --root results/easycom_causal_person_adapter_v2 \
  --dataset easycom \
  --protocol causal_valid_only \
  --paired-deltas

Do not run the official test split while choosing this architecture. Once the method, seed-independent settings and event protocol are frozen, test inference is explicit:

CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_causal_person_adapter_v2.sh test 0 1 2

Interpretation boundary

Dense AP/mAP is an architecture screen. The event evaluator still requires competing-risk censoring before response/yield metrics become the paper protocol. A publishable visual claim additionally requires correct-person, past-shifted, wrong-clip, NULL-person and parameter-matched controls. V2 alone does not predict active-speaker identity or addressee state.

4.3 KiB Raw Blame History