wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'main' of http://182.92.86.220:8300/wx44wx/icassp2027-experiment

Browse Source
This commit is contained in:
Xin Wang
2026-09-10 07:50:25 +08:00
parent 838256a8ae d426b4740a
commit c6885d1aa5
225 changed files with 27716 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/README.md
View File

@@ -26,6 +26,10 @@ tests.
| [`ego4d_encoder_only.md`](ego4d_encoder_only.md) | Preparing and running the Ego4D encoder-only comparison |
| [`ego4d_local_baselines.md`](ego4d_local_baselines.md) | Previewing Ego4D boxes, extracting local features and running 8-GPU jobs |
| [`easycom_track_gru_residual.md`](easycom_track_gru_residual.md) | Matched pooled-to-track GRU experiment, causal identity contract and L20 commands |
| [`easycom_causal_person_adapter_v2.md`](easycom_causal_person_adapter_v2.md) | Dynamic-gated, zero-output causal person-track residual and matched L20 screen |
| [`easycom_egospeak_style_person_gru.md`](easycom_egospeak_style_person_gru.md) | Controlled V2 temporal-stem ablation using EgoSpeak MiniROADA's layer order |
| [`easycom_highrate_person_av.md`](easycom_highrate_person_av.md) | Frozen 25 Hz causal ASD person evidence fused with 5 Hz global Omni context |
| [`easycom_muvap_stage3_baseline.md`](easycom_muvap_stage3_baseline.md) | MuVAP-only frozen VAP+ASD Stage-3 paper reconstruction on EgoSpeak/EasyCom targets |
| [`easycom_oracle_information.md`](easycom_oracle_information.md) | Causal oracle per-person speaking-history upper bound and Session-12 screening commands |
## Design history and implementation options

113
docs/easycom_causal_person_adapter_v2.md Normal file
View File

@@ -0,0 +1,113 @@
# EasyCom causal person adapter V2
## Status and question
Implemented and CPU-testable as `--local-fusion causal-person-adapter-v2`;
GPU results have not yet been measured. The mode is an optimization-corrected,
incremental experiment after the matched pooled `G+A+V` baseline. It does not
yet establish that track identity or visual grounding improves wearer floor
behavior.
The experiment asks whether strictly causal, per-track visual histories contain
information beyond equal person/frame pooling. It deliberately preserves the
legacy `track-gru-residual` mode so its earlier null screen remains reproducible.
The controlled `causal-person-adapter-egospeak-gru` ablation keeps this complete
V2 fusion path and changes only the per-track temporal stem; see
[`easycom_egospeak_style_person_gru.md`](easycom_egospeak_style_person_gru.md).
## Architecture
For 5 Hz row `t`, the media cutoff is `(t+1)/5` seconds. The local loader selects
only person frames strictly earlier than that cutoff. Each currently visible
anonymous track contributes at most `K` observations:
```text
p_i,t = shared_GRU(project(v_i,t-K+1:t))
q_t = project(G_t) + project(A_t)
c_t = MHA(q_t, [NULL, p_1,t, ..., p_N,t])
l_t = MLP([c_t, projected_A_t, person_valid, audio_valid])
gate = sigmoid(MLP([pooled_base_hidden_t, l_t]))
logits_t = pooled_logits_t + person_valid * W_delta(gate * l_t)
```
`W_delta.weight` and `W_delta.bias` are zero initialized. After copying a
matched pooled checkpoint, V2 therefore starts with logits exactly equal to
the pooled model. The first backward pass updates `W_delta`; subsequent passes
send gradients into the shared GRU, attention and dynamic gate. This fixes the
legacy mode's zero scalar gate, which blocked all upstream branch gradients at
initialization.
Track IDs are routing metadata only and never become embeddings. Parameters are
shared across people, the attention uses a real-person mask plus a learned NULL
person, source-clip/max-gap boundaries reset links, and changing padded values
or permuting the current person order cannot change predictions. At the 2 FPS
local grid, this runbook uses `K=4`, covering at most two seconds. Repeated 5 Hz
queries
recompute/read the same causal history; they do not append duplicate visual
observations.
This immediate V2 keeps pooled visual evidence in the exact base. If it passes,
a later replacement ablation must compare `G+A+person-track` without pooled V;
otherwise the final method cannot claim unconditional pooling is unnecessary.
## L20 smoke test
The smoke test uses only training sessions and writes isolated outputs:
```bash
EPOCHS=1 \
WARMUP_EPOCHS=0 \
SESSIONS="4 5" \
DEV_SESSIONS=5 \
SCREEN_SESSIONS=5 \
CHECKPOINT_ROOT=checkpoints/easycom_person_adapter_v2_smoke \
RESULT_ROOT=results/easycom_person_adapter_v2_smoke \
CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_causal_person_adapter_v2.sh all 0
```
Confirm that:
- the pooled checkpoint is accepted as an exact base;
- `person_adapter_diagnostics.residual_output_weight_norm` becomes nonzero;
- `delta_logit_mean_abs` becomes nonzero;
- shapes and valid counts match the pooled run;
- no Session 1--3 output is produced.
## Three-seed Session-12 screen
Run one seed per GPU in separate foreground terminals. Filenames are disjoint,
so the runs may share the same roots:
```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_causal_person_adapter_v2.sh all 0
CUDA_VISIBLE_DEVICES=1 bash run_easycom_causal_person_adapter_v2.sh all 1
CUDA_VISIBLE_DEVICES=2 bash run_easycom_causal_person_adapter_v2.sh all 2
```
Summarize after all processes finish:
```bash
python summarize_local_baselines.py \
--root results/easycom_causal_person_adapter_v2 \
--dataset easycom \
--protocol causal_valid_only \
--paired-deltas
```
Do not run the official test split while choosing this architecture. Once the
method, seed-independent settings and event protocol are frozen, test inference
is explicit:
```bash
CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_causal_person_adapter_v2.sh test 0 1 2
```
## Interpretation boundary
Dense AP/mAP is an architecture screen. The event evaluator still requires
competing-risk censoring before response/yield metrics become the paper
protocol. A publishable visual claim additionally requires correct-person,
past-shifted, wrong-clip, NULL-person and parameter-matched controls. V2 alone
does not predict active-speaker identity or addressee state.

92
docs/easycom_egospeak_style_person_gru.md Normal file
View File

@@ -0,0 +1,92 @@
# EasyCom EgoSpeak-style person-track GRU ablation
## Status and purpose
Implemented as
`--local-fusion causal-person-adapter-egospeak-gru`; GPU results are not yet
measured. This is a controlled temporal-stem ablation of
`causal-person-adapter-v2`, not a reproduction of EgoSpeak MiniROADA.
The comparison changes only how each anonymous person's causal visual history
is embedded before the existing GRU/set-attention path:
| Component | Native V2 | EgoSpeak-style ablation |
|---|---|---|
| track input | `LayerNorm -> Linear -> GELU` | `Linear -> LayerNorm -> ReLU -> Dropout` |
| temporal layer | one-layer shared causal GRU | the same one-layer shared causal GRU |
| GRU output | final hidden state | `ReLU(final hidden state)` |
| all downstream fusion | unchanged | unchanged |
Both modes use the same raw local visual features, causal track histories,
history length, G+A query, masked multi-person attention, direct local-audio
state, dynamic vector gate, zero-initialized residual output, pooled base
checkpoint, loss, split, optimizer and seed. The actual trainable parameter
count is printed and stored because moving LayerNorm across the projection
causes a small parameter-count difference. With the default 2048D visual input
and 256D person state, the EgoSpeak-style stem has 3,584 fewer affine
LayerNorm parameters than native V2; all Linear and GRU matrix shapes match.
## Relation to EgoSpeak
EgoSpeak MiniROADA applies
`Linear -> LayerNorm -> ReLU -> Dropout -> one-layer GRU -> ReLU` to one
**global 5 Hz RGB+audio sequence**. Its EasyCom configuration uses a 128-step
window, 2048D embedding and 1024D hidden state, followed by its own anticipation
expansion and shared classifier.
This repository uses a clean-room PyTorch implementation of only that temporal
stem and applies it independently to each anonymous **visual person track**.
It intentionally retains V2's 256D screen, two-second person history and
residual forecasting head so the experiment isolates temporal parameterization.
The correct name in results and the paper is **EgoSpeak-style per-track GRU**,
not “EgoSpeak reproduction.” The audited reference is
[`third_party/EgoSpeak/MiniROAD/model/rnn/rnn.py`](../third_party/EgoSpeak/MiniROAD/model/rnn/rnn.py).
## L20 smoke test
Use only training sessions and isolated output roots:
```bash
EPOCHS=1 \
WARMUP_EPOCHS=0 \
SESSIONS="4 5" \
DEV_SESSIONS=5 \
SCREEN_SESSIONS=5 \
CHECKPOINT_ROOT=checkpoints/easycom_egospeak_person_gru_smoke \
RESULT_ROOT=results/easycom_egospeak_person_gru_smoke \
CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_causal_person_adapter_v2.sh all 0
```
This runs the matched pooled base, native V2 and EgoSpeak-style ablation. Check
that all three use the same split and base-checkpoint hash, both adapter modes
start from the pooled logits, and their residual-output/delta diagnostics become
nonzero after training.
## Three-seed development screen
If matched pooled/V2 checkpoints already exist, run only the new ablation:
```bash
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=0 bash run_easycom_causal_person_adapter_v2.sh egospeak-gru 0
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=1 bash run_easycom_causal_person_adapter_v2.sh egospeak-gru 1
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=2 bash run_easycom_causal_person_adapter_v2.sh egospeak-gru 2
```
Otherwise, run `all` instead of `egospeak-gru` to create each matched pooled
checkpoint and both adapters. Do not run official Sessions 1--3 while selecting
the temporal stem.
After all jobs finish:
```bash
python summarize_local_baselines.py \
--root results/easycom_causal_person_adapter_v2 \
--dataset easycom \
--protocol causal_valid_only \
--paired-deltas
```
The relevant paired rows are `egospeak_gru - pooled` and
`egospeak_gru - person_adapter_v2`. Dense AP/mAP is only the architecture
screen; wearer event claims still wait for the frozen competing-risk evaluator.

185
docs/easycom_highrate_person_av.md Normal file
View File

@@ -0,0 +1,185 @@
# Global Omni + shared 25 Hz audio + per-person evidence
## What this implementation does
The first version deliberately keeps the two jobs separate:
1. Existing Qwen2.5-Omni states provide the global scene/conversation context
`G_t` at the verified 5 Hz task grid.
2. MuVAP's frozen CPC frontend encodes the dense session's mixed 16 kHz audio
once into a shared causal 25 Hz stream `U_t`; it does not reset at the
artificial one-minute file boundaries.
3. MuVAP's causal visual track encoder produces one visual state per visible
person every 40 ms. Its audio-visual active-speaker readout contributes one
speaking logit, so a person token is `[visual_state, ASD_score]`; the shared
audio vector is not duplicated inside every person token.
4. At training time `G_t + U_t` queries the variable-sized person set using
masked attention.
5. A zero-initialized residual readout maps the selected person evidence into
a correction of the ordinary `G_t + U_t` logits. With matching shared
weights, the initial model is exactly the audio-only baseline; the person
branch must earn every change during training.
6. The ten-horizon floor head predicts the same 0.2--2.0 second EgoSpeak target.
In compact form:
```text
full scene + mixed audio -> frozen Omni -> G_t --------+--> G+U base logits -----+
mixed audio -> frozen CPC once -> U_t -----------------+ sum
^
25 Hz face track -> causal visual state --+ |
U_t + visual state -> ASD speaking score -+-> {P_t,i} -> zero-init residual ----+
G_t+U_t queries person set
```
The person branch uses MuVAP's released **ASD module**, not the unreleased/full
MuVAP multiparty forecasting module. This is a traditional high-rate
active-speaker frontend combined with an Omni global representation.
## Causality and rate contract
- MuVAP ASD runs at 25 Hz. Model state `k` consumes the complete interval
`[k/25, (k+1)/25)` and is timestamped by its right edge `(k+1)/25`.
- Every fifth ASD state ends on a 5 Hz task cutoff. Equality is causal here:
the saved state at `0.2s` has consumed samples strictly before `0.2s`.
- MuVAP-style chunks score 10 seconds at a time and prepend at most 2 seconds
of past context. Chunks overlap only into the past.
- A source-clip boundary or missing face observation resets a local face track.
The shared audio stream remains continuous on the same concatenated dense
timeline used by the targets and global Omni features.
- The released MuVAP data loader's whole-recording RMS normalization is not
reused: at an early cutoff that statistic would depend on future audio. The
extractor uses fixed float-PCM scaling with pointwise clipping only. Record
this causal deployment/domain-shift choice when reporting the experiment.
- Speech labels, wearer targets and future frames are never extractor inputs.
- Provided boxes isolate the representation/fusion question. They are not an
online detector result and must be described that way in the paper.
## External dependency
The audited MuVAP checkout and published ASD-CPC release are vendored under
`third_party/MuVAP`, including an immutable `.upstream-revision` record. For a
fresh checkout of this project, no separate MuVAP clone or weight download is
needed. The following commands document how that vendored snapshot was
obtained and are only needed when intentionally refreshing it:
```bash
git clone https://github.com/Haotian-Qi/MuVAP.git third_party/MuVAP
git -C third_party/MuVAP checkout a1ef9c8
python third_party/MuVAP/tools/fetch_frontends.py cpc
huggingface-cli download Haotian-Qi/MuVAP \
--include 'asd-cpc/*' \
--local-dir third_party/MuVAP/weights
```
The extractor imports the pinned vendored checkout directly; it does not need
an editable MuVAP install. Check that `torch`, `torchaudio`, `einops` and
`yaml` import in the existing Qwen container instead of upgrading its full
dependency stack.
The extractor checks the source revision and hashes the selected release
weights into every output contract. If a newer MuVAP revision is intentionally
tested, give its exact commit with `--muvap-revision` and use a new output root.
## EasyCom smoke extraction
Run a 10-second partial smoke first. A partial sidecar is intentionally
rejected by training.
```bash
export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
export HIGHRATE_LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_muvap_asd_25hz_v2
export ASD_WEIGHTS=third_party/MuVAP/weights/asd-cpc
CUDA_VISIBLE_DEVICES=0 python extract_highrate_person_av_features.py \
--dataset easycom \
--data-root "$DATASET_ROOT" \
--feature-root "$FEATURE_ROOT" \
--output-root features/highrate_person_av_smoke \
--samples 12 \
--muvap-root third_party/MuVAP \
--asd-weights "$ASD_WEIGHTS" \
--max-steps 50 \
--device cuda:0 \
--dtype bfloat16
```
Audit the result:
```bash
python - <<'PY'
import torch
p = torch.load("features/highrate_person_av_smoke/Session_12.pt", map_location="cpu", weights_only=False)
print("format:", p["format"])
print("partial:", p["partial"])
print("times:", p["times"][:3].tolist(), "...", p["times"][-1].item())
print("person features:", tuple(p["persons"]["features"].shape))
print("shared audio:", tuple(p["audio"].shape))
print("frame offsets:", tuple(p["persons"]["frame_offsets"].shape))
print("person-row coverage:", float(p["visual_valid"].float().mean()))
print("uses speech labels:", p["config"]["uses_speech_labels"])
print("causal policy:", p["config"]["causal_policy"])
PY
```
Expected invariants are `causal-highrate-person-av-v2`, `partial=True`, times
`[0.2, 0.4, ...]`, shared audio width 256, person width 257 (`256D causal
visual state + 1 AV activity logit`), and `uses_speech_labels=False`.
## Full EasyCom extraction
Run one command per GPU/terminal. The shard assignments are disjoint and the
destination contains real files, not symbolic links:
```bash
CUDA_VISIBLE_DEVICES=0 python extract_highrate_person_av_features.py \
--dataset easycom --data-root "$DATASET_ROOT" --feature-root "$FEATURE_ROOT" \
--output-root "$HIGHRATE_LOCAL_ROOT" --samples 1 2 3 4 5 6 7 8 9 10 11 12 \
--num-shards 8 --shard-index 0 --muvap-root third_party/MuVAP \
--asd-weights "$ASD_WEIGHTS" --device cuda:0 --dtype bfloat16
```
Repeat with physical GPU and shard index `1` through `7`. Do not add
`--overwrite` unless the exact file has been inspected and a replacement is
intentional.
## First matched training screen
The screen trains on Sessions 4--11 and uses Session 12 for checkpoint
selection. It does not inspect Sessions 1--3. Run different seeds on
different GPUs:
```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_highrate_person_av.sh all 0
CUDA_VISIBLE_DEVICES=1 bash run_easycom_highrate_person_av.sh all 1
CUDA_VISIBLE_DEVICES=2 bash run_easycom_highrate_person_av.sh all 2
```
This compares four matched conditions:
- `global`: frozen Omni global state `G` only;
- `qwen_audio`: existing `G+A_qwen` strong baseline;
- `traditional_audio`: global Omni `G` plus one shared CPC stream `U`;
- `highrate`: load that seed's trained `G+U` checkpoint, freeze the exact base,
and learn only the attention/residual over frozen 25 Hz causal ASD person
states. This first screen therefore asks whether the new person evidence
adds information, without allowing the base to drift.
The `all` command runs `traditional_audio` before `highrate`, so the initializer
exists. If running separately, run `traditional_audio` first for the same seed.
Only after selecting the architecture on Session 12 should the fixed
checkpoints be run on the official test set:
```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_highrate_person_av.sh test 0
CUDA_VISIBLE_DEVICES=1 bash run_easycom_highrate_person_av.sh test 1
CUDA_VISIBLE_DEVICES=2 bash run_easycom_highrate_person_av.sh test 2
```
Dense AP/mAP is a screening/comparability result. The event evaluator still
needs competing-risk censoring before claims about false entries, responses or
yields are final.

211
docs/easycom_muvap_stage3_baseline.md Normal file
View File

@@ -0,0 +1,211 @@
# MuVAP-only Stage-3 reconstruction on EasyCom/EgoSpeak targets
## Question and claim boundary
This baseline asks whether MuVAP's specialized acoustic turn-taking and
person-grounded active-speaker representations can solve the EgoSpeak future
wearer-state task without an Omni representation.
The official MuVAP repository currently releases VAP and ASD training code and
weights, but not its multiparty Stage-3 implementation. Therefore this code is
a clean-room reconstruction from MuVAP Sections 4.3 and 5.3, not an execution
of upstream Stage-3 code.
The frozen inputs are:
- `Z_VAP`: the published 25 Hz mono, role-relative CPC VAP embedding;
- `Z_ASD_i`: the published causal ASD fused embedding for visible track `i`.
The matched control uses the identical projections, frame-attention block,
temporal Transformer and head, but masks every observed `Z_ASD_i` token. It is
named `vap_only`; the full condition is named `stage3`. Comparing these two
conditions isolates aligned person evidence from a simple capacity increase.
The trainable path is:
```text
frozen mixed audio -> frozen role-VAP -> Z_VAP(t) ---- query ----+
frame attention
frozen mixed audio + provided face track i |
-> frozen ASD -> {Z_ASD_i(t)} -- key/value+
-> causal temporal Transformer
-> EgoSpeak 10-horizon activity head
```
Independent Linear-LayerNorm projections map both inputs to 256 dimensions.
At every 25 Hz frame, `Z_VAP` queries the variable person set. A one-layer,
four-head causal temporal Transformer then processes the last 8 seconds and
the state at each 5 Hz right edge predicts the ten EgoSpeak horizons from 0.2
to 2.0 seconds.
Differences from the unpublished original Stage 3 must remain explicit:
- supervision is EgoSpeak's `background/wearer/others` future activity, not
MuVAP's GVAP plus per-visible-person SVAP objectives;
- missing tracks are masked rather than padded as permanent known identities;
- provided EasyCom boxes isolate representation quality and are not an online
face detector result;
- the first strict-causal experiment does not apply whole-recording RMS
normalization because that would expose a future-dependent statistic;
- the temporal memory is fixed to 8 seconds for a reproducible streaming
contract.
## Required releases
The repository vendors the pinned MuVAP source, CPC frontend and the two small
published releases consumed here. After pulling the experiment commit, verify:
```bash
test -s third_party/MuVAP/weights/asd-cpc/weights.pt
test -s third_party/MuVAP/weights/vap-role-cpc/weights.pt
test -s third_party/MuVAP/models/assets/checkpoints/cpc/60k_epoch4-d0f474de.pt
```
Expected paths:
```text
third_party/MuVAP/weights/asd-cpc/{config.yaml,weights.pt}
third_party/MuVAP/weights/vap-role-cpc/{config.yaml,weights.pt}
third_party/MuVAP/models/assets/checkpoints/cpc/60k_epoch4-d0f474de.pt
```
On the L20 environment, run the CPU model/data-contract tests before touching
the full dataset:
```bash
python -m unittest test_muvap_egospeak_stage3
```
## Extraction smoke test
The reference root supplies only the already-audited 5 Hz timestamps and raw
source inventory. Its Omni `states` are never copied or loaded by the training
code.
```bash
export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
export REFERENCE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
export MUVAP_STAGE3_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/muvap_stage3_cpc_25hz_v1
CUDA_VISIBLE_DEVICES=0 python extract_muvap_egospeak_features.py \
--dataset easycom \
--data-root "$DATASET_ROOT" \
--reference-feature-root "$REFERENCE_ROOT" \
--output-root features/muvap_stage3_smoke \
--samples 12 \
--muvap-root third_party/MuVAP \
--vap-weights third_party/MuVAP/weights/vap-role-cpc \
--asd-weights third_party/MuVAP/weights/asd-cpc \
--max-steps 50 \
--device cuda:0 \
--dtype bfloat16
```
The smoke should report 250 MuVAP frames for 50 target states, finite
`vap=[250,D_vap]`, a ragged person tensor with width `D_asd`, exact task frame
indices `[4,9,...,249]`, and `partial=True`. Training intentionally rejects
this partial file.
## Full extraction
Use a fresh output directory. Eight terminals can run disjoint real-file
shards; no symbolic links are used. Replace both `GPU` and `SHARD` with the
same value from 0 through 7:
```bash
GPU=0
SHARD=0
CUDA_VISIBLE_DEVICES="$GPU" python extract_muvap_egospeak_features.py \
--dataset easycom \
--data-root "$DATASET_ROOT" \
--reference-feature-root "$REFERENCE_ROOT" \
--output-root "$MUVAP_STAGE3_ROOT" \
--samples 1 2 3 4 5 6 7 8 9 10 11 12 \
--num-shards 8 \
--shard-index "$SHARD" \
--muvap-root third_party/MuVAP \
--vap-weights third_party/MuVAP/weights/vap-role-cpc \
--asd-weights third_party/MuVAP/weights/asd-cpc \
--device cuda:0 \
--dtype bfloat16
```
Do not add `--overwrite` without inspecting the exact destination file.
## CPU contract audit
```bash
python - <<'PY'
from pathlib import Path
from muvap_egospeak_stage3 import load_muvap_session
feature_root = Path("/data1/yfzx/wangx/Datasets/easycom/omni_features/muvap_stage3_cpc_25hz_v1")
target_root = Path("/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame")
contract = None
for sid in range(1, 13):
data = load_muvap_session(feature_root, target_root, sid, expected_contract=contract)
contract = data.contract
print(
data.name,
"task=", len(data.targets),
"25Hz=", len(data.vap),
"VAP=", data.vap_dim,
"ASD=", data.asd_dim,
"max_people=", data.people.shape[1],
)
PY
```
## One-epoch L20 smoke
Use only training sessions. This verifies forward/backward/checkpoint and does
not touch Sessions 1--3:
```bash
CUDA_VISIBLE_DEVICES=0 python train_muvap_egospeak_stage3.py \
--feature-root "$MUVAP_STAGE3_ROOT" \
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
--output checkpoints/muvap_stage3_smoke.pth \
--sessions 4 5 \
--dev-sessions 5 \
--epochs 1 \
--batch-size 4 \
--history-seconds 8 \
--max-train-examples 64 \
--device cuda:0
```
Use a fresh checkpoint name if repeating the smoke.
## Three-seed experiment
First screen the full Stage-3 reconstruction and its capacity-matched VAP-only
control on Session 12. Run one seed per GPU/terminal; the examples below use
six GPUs concurrently:
```bash
CUDA_VISIBLE_DEVICES=0 PERSON_EVIDENCE=aligned bash run_easycom_muvap_stage3.sh all 0
CUDA_VISIBLE_DEVICES=1 PERSON_EVIDENCE=aligned bash run_easycom_muvap_stage3.sh all 1
CUDA_VISIBLE_DEVICES=2 PERSON_EVIDENCE=aligned bash run_easycom_muvap_stage3.sh all 2
CUDA_VISIBLE_DEVICES=3 PERSON_EVIDENCE=none bash run_easycom_muvap_stage3.sh all 0
CUDA_VISIBLE_DEVICES=4 PERSON_EVIDENCE=none bash run_easycom_muvap_stage3.sh all 1
CUDA_VISIBLE_DEVICES=5 PERSON_EVIDENCE=none bash run_easycom_muvap_stage3.sh all 2
```
`all` trains on Sessions 4--11, selects the best of five epochs on Session 12,
and writes the Session-12 screen. It does not touch the official test set.
After fixing the architecture and hyperparameters from training/Session 12,
run the official test once:
```bash
CUDA_VISIBLE_DEVICES=0 PERSON_EVIDENCE=aligned bash run_easycom_muvap_stage3.sh test 0
CUDA_VISIBLE_DEVICES=1 PERSON_EVIDENCE=aligned bash run_easycom_muvap_stage3.sh test 1
CUDA_VISIBLE_DEVICES=2 PERSON_EVIDENCE=aligned bash run_easycom_muvap_stage3.sh test 2
CUDA_VISIBLE_DEVICES=3 PERSON_EVIDENCE=none bash run_easycom_muvap_stage3.sh test 0
CUDA_VISIBLE_DEVICES=4 PERSON_EVIDENCE=none bash run_easycom_muvap_stage3.sh test 1
CUDA_VISIBLE_DEVICES=5 PERSON_EVIDENCE=none bash run_easycom_muvap_stage3.sh test 2
```
Dense AP/mAP is currently the primary screening output. Floor-event results
remain timing-derived proxies until competing-risk censoring is frozen in
`evaluate_floor_state_changes.py`.

23
docs/progressive_feature_fusion_route_2026-09-02.md
View File

@@ -231,6 +231,29 @@ ten-horizon mean mAP 配对增量仅为 `+0.01306 +/- 0.02368` 个百分点，
residual 最后输出层零初始化，并在 dev 上记录 delta-logit、概率变化和非零梯度；详见
[`结果记录`](../results/easycom_track_gru_residual_4ep_2026-09-07/README.md)。
修正版已实现为 `causal-person-adapter-v2`，但尚无 GPU 结果。它仍从 matched pooled
`G+A+V` checkpoint 精确初始化并默认冻结 base；每条当前 track 经共享因果 GRU 后由 G+A
query 读取，再与直接 local audio 构成局部 state。模型使用逐样本向量 gate，并仅把最终
residual-logit projection 的 weight/bias 置零。这样初始 logits 与 pooled baseline
逐元素相同，第一次更新 residual output，后续更新才可进入 GRU、attention 和动态 gate。
运行契约与命令见
[`easycom_causal_person_adapter_v2.md`](easycom_causal_person_adapter_v2.md)。它保留 pooled V，
因此若结果通过，还必须补一个移除 pooled V 的 replacement ablation，才能决定最终方法。
为减少“GRU 具体写法”这一实验变量，另实现
`causal-person-adapter-egospeak-gru`：保留 V2 的所有下游融合与优化设计，只把逐人输入改成
EgoSpeak MiniROADA 风格的 `Linear-LN-ReLU-Dropout-GRU-ReLU`。这不是 EgoSpeak 的全局
5 Hz RGB+audio GRU 复现，而是人物 track 上的受控 temporal-stem 消融；运行说明见
[`easycom_egospeak_style_person_gru.md`](easycom_egospeak_style_person_gru.md)。
为直接检验“低频 Qwen crop 是否遗漏唇音动态”，另实现独立的
`highrate-person-av-attention` 分支。它不把同一张 2 FPS 人脸重复送进 GRU：MuVAP CPC 在
25 Hz 上只编码一次 mixed audio 得到共享 `U`，每条人脸轨迹另产生 causal visual state 和
一个 AV speaking score，只在每个 5 Hz 右边界保存。随后由 `G+U` query 做集合注意力，并
用零初始化 residual 修正已经训练且冻结的 matched `G+U` logits。它首先回答传统高频
audio/person 证据能否弥补全局 Omni，而不是检验 identity continuity。实现和命令见
[`easycom_highrate_person_av.md`](easycom_highrate_person_av.md)。
## 8. F4：Person-conditioned audio/Omni history
F4 让每个匿名人物分别读取已有的 5 Hz 因果历史：