93 lines
3.8 KiB
Markdown
93 lines
3.8 KiB
Markdown
# EasyCom EgoSpeak-style person-track GRU ablation
|
|
|
|
## Status and purpose
|
|
|
|
Implemented as
|
|
`--local-fusion causal-person-adapter-egospeak-gru`; GPU results are not yet
|
|
measured. This is a controlled temporal-stem ablation of
|
|
`causal-person-adapter-v2`, not a reproduction of EgoSpeak MiniROADA.
|
|
|
|
The comparison changes only how each anonymous person's causal visual history
|
|
is embedded before the existing GRU/set-attention path:
|
|
|
|
| Component | Native V2 | EgoSpeak-style ablation |
|
|
|---|---|---|
|
|
| track input | `LayerNorm -> Linear -> GELU` | `Linear -> LayerNorm -> ReLU -> Dropout` |
|
|
| temporal layer | one-layer shared causal GRU | the same one-layer shared causal GRU |
|
|
| GRU output | final hidden state | `ReLU(final hidden state)` |
|
|
| all downstream fusion | unchanged | unchanged |
|
|
|
|
Both modes use the same raw local visual features, causal track histories,
|
|
history length, G+A query, masked multi-person attention, direct local-audio
|
|
state, dynamic vector gate, zero-initialized residual output, pooled base
|
|
checkpoint, loss, split, optimizer and seed. The actual trainable parameter
|
|
count is printed and stored because moving LayerNorm across the projection
|
|
causes a small parameter-count difference. With the default 2048D visual input
|
|
and 256D person state, the EgoSpeak-style stem has 3,584 fewer affine
|
|
LayerNorm parameters than native V2; all Linear and GRU matrix shapes match.
|
|
|
|
## Relation to EgoSpeak
|
|
|
|
EgoSpeak MiniROADA applies
|
|
`Linear -> LayerNorm -> ReLU -> Dropout -> one-layer GRU -> ReLU` to one
|
|
**global 5 Hz RGB+audio sequence**. Its EasyCom configuration uses a 128-step
|
|
window, 2048D embedding and 1024D hidden state, followed by its own anticipation
|
|
expansion and shared classifier.
|
|
|
|
This repository uses a clean-room PyTorch implementation of only that temporal
|
|
stem and applies it independently to each anonymous **visual person track**.
|
|
It intentionally retains V2's 256D screen, two-second person history and
|
|
residual forecasting head so the experiment isolates temporal parameterization.
|
|
The correct name in results and the paper is **EgoSpeak-style per-track GRU**,
|
|
not “EgoSpeak reproduction.” The audited reference is
|
|
[`third_party/EgoSpeak/MiniROAD/model/rnn/rnn.py`](../third_party/EgoSpeak/MiniROAD/model/rnn/rnn.py).
|
|
|
|
## L20 smoke test
|
|
|
|
Use only training sessions and isolated output roots:
|
|
|
|
```bash
|
|
EPOCHS=1 \
|
|
WARMUP_EPOCHS=0 \
|
|
SESSIONS="4 5" \
|
|
DEV_SESSIONS=5 \
|
|
SCREEN_SESSIONS=5 \
|
|
CHECKPOINT_ROOT=checkpoints/easycom_egospeak_person_gru_smoke \
|
|
RESULT_ROOT=results/easycom_egospeak_person_gru_smoke \
|
|
CUDA_VISIBLE_DEVICES=0 \
|
|
bash run_easycom_causal_person_adapter_v2.sh all 0
|
|
```
|
|
|
|
This runs the matched pooled base, native V2 and EgoSpeak-style ablation. Check
|
|
that all three use the same split and base-checkpoint hash, both adapter modes
|
|
start from the pooled logits, and their residual-output/delta diagnostics become
|
|
nonzero after training.
|
|
|
|
## Three-seed development screen
|
|
|
|
If matched pooled/V2 checkpoints already exist, run only the new ablation:
|
|
|
|
```bash
|
|
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=0 bash run_easycom_causal_person_adapter_v2.sh egospeak-gru 0
|
|
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=1 bash run_easycom_causal_person_adapter_v2.sh egospeak-gru 1
|
|
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=2 bash run_easycom_causal_person_adapter_v2.sh egospeak-gru 2
|
|
```
|
|
|
|
Otherwise, run `all` instead of `egospeak-gru` to create each matched pooled
|
|
checkpoint and both adapters. Do not run official Sessions 1--3 while selecting
|
|
the temporal stem.
|
|
|
|
After all jobs finish:
|
|
|
|
```bash
|
|
python summarize_local_baselines.py \
|
|
--root results/easycom_causal_person_adapter_v2 \
|
|
--dataset easycom \
|
|
--protocol causal_valid_only \
|
|
--paired-deltas
|
|
```
|
|
|
|
The relevant paired rows are `egospeak_gru - pooled` and
|
|
`egospeak_gru - person_adapter_v2`. Dense AP/mAP is only the architecture
|
|
screen; wearer event claims still wait for the frozen competing-risk evaluator.
|