Add causal person fusion and MuVAP stage-3 baselines
This commit is contained in:
92
docs/easycom_egospeak_style_person_gru.md
Normal file
92
docs/easycom_egospeak_style_person_gru.md
Normal file
@@ -0,0 +1,92 @@
|
||||
# EasyCom EgoSpeak-style person-track GRU ablation
|
||||
|
||||
## Status and purpose
|
||||
|
||||
Implemented as
|
||||
`--local-fusion causal-person-adapter-egospeak-gru`; GPU results are not yet
|
||||
measured. This is a controlled temporal-stem ablation of
|
||||
`causal-person-adapter-v2`, not a reproduction of EgoSpeak MiniROADA.
|
||||
|
||||
The comparison changes only how each anonymous person's causal visual history
|
||||
is embedded before the existing GRU/set-attention path:
|
||||
|
||||
| Component | Native V2 | EgoSpeak-style ablation |
|
||||
|---|---|---|
|
||||
| track input | `LayerNorm -> Linear -> GELU` | `Linear -> LayerNorm -> ReLU -> Dropout` |
|
||||
| temporal layer | one-layer shared causal GRU | the same one-layer shared causal GRU |
|
||||
| GRU output | final hidden state | `ReLU(final hidden state)` |
|
||||
| all downstream fusion | unchanged | unchanged |
|
||||
|
||||
Both modes use the same raw local visual features, causal track histories,
|
||||
history length, G+A query, masked multi-person attention, direct local-audio
|
||||
state, dynamic vector gate, zero-initialized residual output, pooled base
|
||||
checkpoint, loss, split, optimizer and seed. The actual trainable parameter
|
||||
count is printed and stored because moving LayerNorm across the projection
|
||||
causes a small parameter-count difference. With the default 2048D visual input
|
||||
and 256D person state, the EgoSpeak-style stem has 3,584 fewer affine
|
||||
LayerNorm parameters than native V2; all Linear and GRU matrix shapes match.
|
||||
|
||||
## Relation to EgoSpeak
|
||||
|
||||
EgoSpeak MiniROADA applies
|
||||
`Linear -> LayerNorm -> ReLU -> Dropout -> one-layer GRU -> ReLU` to one
|
||||
**global 5 Hz RGB+audio sequence**. Its EasyCom configuration uses a 128-step
|
||||
window, 2048D embedding and 1024D hidden state, followed by its own anticipation
|
||||
expansion and shared classifier.
|
||||
|
||||
This repository uses a clean-room PyTorch implementation of only that temporal
|
||||
stem and applies it independently to each anonymous **visual person track**.
|
||||
It intentionally retains V2's 256D screen, two-second person history and
|
||||
residual forecasting head so the experiment isolates temporal parameterization.
|
||||
The correct name in results and the paper is **EgoSpeak-style per-track GRU**,
|
||||
not “EgoSpeak reproduction.” The audited reference is
|
||||
[`third_party/EgoSpeak/MiniROAD/model/rnn/rnn.py`](../third_party/EgoSpeak/MiniROAD/model/rnn/rnn.py).
|
||||
|
||||
## L20 smoke test
|
||||
|
||||
Use only training sessions and isolated output roots:
|
||||
|
||||
```bash
|
||||
EPOCHS=1 \
|
||||
WARMUP_EPOCHS=0 \
|
||||
SESSIONS="4 5" \
|
||||
DEV_SESSIONS=5 \
|
||||
SCREEN_SESSIONS=5 \
|
||||
CHECKPOINT_ROOT=checkpoints/easycom_egospeak_person_gru_smoke \
|
||||
RESULT_ROOT=results/easycom_egospeak_person_gru_smoke \
|
||||
CUDA_VISIBLE_DEVICES=0 \
|
||||
bash run_easycom_causal_person_adapter_v2.sh all 0
|
||||
```
|
||||
|
||||
This runs the matched pooled base, native V2 and EgoSpeak-style ablation. Check
|
||||
that all three use the same split and base-checkpoint hash, both adapter modes
|
||||
start from the pooled logits, and their residual-output/delta diagnostics become
|
||||
nonzero after training.
|
||||
|
||||
## Three-seed development screen
|
||||
|
||||
If matched pooled/V2 checkpoints already exist, run only the new ablation:
|
||||
|
||||
```bash
|
||||
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=0 bash run_easycom_causal_person_adapter_v2.sh egospeak-gru 0
|
||||
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=1 bash run_easycom_causal_person_adapter_v2.sh egospeak-gru 1
|
||||
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=2 bash run_easycom_causal_person_adapter_v2.sh egospeak-gru 2
|
||||
```
|
||||
|
||||
Otherwise, run `all` instead of `egospeak-gru` to create each matched pooled
|
||||
checkpoint and both adapters. Do not run official Sessions 1--3 while selecting
|
||||
the temporal stem.
|
||||
|
||||
After all jobs finish:
|
||||
|
||||
```bash
|
||||
python summarize_local_baselines.py \
|
||||
--root results/easycom_causal_person_adapter_v2 \
|
||||
--dataset easycom \
|
||||
--protocol causal_valid_only \
|
||||
--paired-deltas
|
||||
```
|
||||
|
||||
The relevant paired rows are `egospeak_gru - pooled` and
|
||||
`egospeak_gru - person_adapter_v2`. Dense AP/mAP is only the architecture
|
||||
screen; wearer event claims still wait for the frozen competing-risk evaluator.
|
||||
Reference in New Issue
Block a user