EasyCom anonymous track-GRU residual

Decision

The first track-temporal experiment keeps the strongest pooled local baseline as an exact, optionally frozen base and learns only a causal person-track correction:

pooled base:  G_t + A_t + pooled-V_t -> concat-MLP -> base logits

per visible person i:
  past crop observations on the same temporary track
      -> shared projection -> shared one-layer GRU -> H_t,i

G_t + A_t -> masked attention over {NULL, H_t,1, ..., H_t,N}
          -> zero-gated residual logits

final logits = base logits + tanh(alpha) * person_valid * residual logits
alpha starts at exactly zero

This is the smallest experiment that asks whether anonymous visual continuity adds information beyond G+A+pooled-V. It is not yet person-conditioned audio grounding and does not modify the Omni Thinker.

What was taken from EgoSpeak

The official EgoSpeak repository is cloned for inspection at third_party/EgoSpeak and excluded from Git. The inspected revision is b5bee35e796d8f2d6aa7da2bf1b07836270f8bfb.

Its MiniROADA model projects concatenated RGB/audio features, applies a single-layer GRU, expands the recurrent state into ten anticipation states, and uses a shared classifier. The EasyCom configuration uses a 128-step window, stride 4, 2048D embedding, 1024D GRU and ten future steps.

We reuse the defensible ideas—causal recurrence, shared weights and a ten-step anticipation output—but not the implementation. Our GRU is shared across anonymous person tracks, reads only crop observations at or before the current cutoff, and corrects the already measured pooled Omni baseline. Consequently, this method must be described as an EgoSpeak-inspired track-temporal adapter, not an EgoSpeak GRU reproduction.

Causal and identity contract

A 5 Hz row with cutoff (t+1)/5 reads the latest local video frame strictly before that cutoff.
Track history contains at most --person-history-frames real 2 FPS crop observations and is ordered oldest to newest.
Temporary IDs only link observations. They are never embedded or passed to the model, so global ID renaming cannot change a prediction.
A source-clip boundary always resets a track. A time gap larger than --person-track-max-gap-seconds also resets it.
Missing, -1, None, null and unknown IDs are isolated observations.
Duplicate known IDs in one frame fail instead of being silently merged.
Empty person sets produce exactly zero visual correction.

The existing EasyCom local sidecars already contain crop features, frame times, source-clip indices and person IDs; this experiment does not require feature re-extraction.

Fair initialization

track-gru-residual requires --initialize-pooled-checkpoint. The loader checks dataset, train/dev split, loss, layer fusion, dimensions and local feature contract before copying the pooled path. By default those copied parameters are frozen. The residual gate starts at zero, so before optimization the track model must reproduce the pooled logits exactly, element for element.

The matched screening protocol is:

EasyCom Session 4--11: training;
Session 12: development/model selection;
Session 1--3: final test, run only after history/gap and the method are frozen;
4 epochs, three seeds, identical optimizer/loss for pooled and track runs;
16 observations at 2 FPS (up to 8 seconds), 1-second maximum link gap.

This protocol intentionally re-runs the pooled base. Historical no-dev results cannot be used as the matched control.

Run

After pulling on the L20 machine, run the focused CPU regression tests first:

python -m unittest \
  test_local_turn_features.LocalTests.test_track_loader_builds_causal_histories_and_resets_tracks \
  test_local_turn_features.LocalTests.test_track_gru_residual_starts_from_exact_pooled_logits \
  test_local_turn_features.RoundTripTests.test_train_infer_easycom_and_ego4d

One-seed runtime smoke on an L20:

EPOCHS=1 \
SESSIONS="4 5" \
DEV_SESSIONS=5 \
SCREEN_SESSIONS=5 \
CHECKPOINT_ROOT=checkpoints/easycom_track_gru_smoke \
RESULT_ROOT=results/easycom_track_gru_smoke \
CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_track_gru_residual.sh all 0

Then use three GPUs in three foreground terminals. These runs report Session 12 development scores, not the official test set:

SUMMARIZE=0 CUDA_VISIBLE_DEVICES=0 bash run_easycom_track_gru_residual.sh all 0
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=1 bash run_easycom_track_gru_residual.sh all 1
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=2 bash run_easycom_track_gru_residual.sh all 2

After all development jobs finish:

python summarize_local_baselines.py \
  --root results/easycom_track_gru_residual \
  --dataset easycom \
  --protocol causal_valid_only \
  --paired-deltas

Only after choosing and freezing the method, evaluate the saved checkpoints once on Sessions 1--3:

CUDA_VISIBLE_DEVICES=0 bash run_easycom_track_gru_residual.sh test 0 1 2

This writes and summarizes results/easycom_track_gru_residual_test without retraining. A single GPU is sufficient because head inference is inexpensive.

Paths and hyperparameters can be overridden with the environment variables at the top of run_easycom_track_gru_residual.sh. The launcher runs in the foreground and refuses to overwrite an existing checkpoint or result.

Interpretation and stopping rule

The initial screen answers only whether track-temporal visual structure can improve the dense EgoSpeak-compatible metric. Advance this branch only if same-seed track_gru - pooled deltas are directionally consistent on Session 12 without sacrificing short-horizon wearer AP. The official test is a final confirmation, not another model-selection stage. Because prior F1 variation was roughly a few tenths of a percentage point, tiny single-seed changes are not evidence.

If the track branch wins, next run correct-track versus broken-track, past-shifted and no-ID temporal controls, followed by person-conditioned audio grounding. If it does not win, first compare 8/16/32 observations and optionally unfreeze the pooled base using Session 12 only; do not add deeper cross-attention merely to rescue a null result.

The paper-level claim still requires the frozen competing-risk event evaluator and matched-recall false-entry analysis. Dense AP/mAP improvement alone is a representation result, not proof that the model understands multi-person floor behavior.

6.3 KiB Raw Blame History