Add causal person fusion and MuVAP stage-3 baselines
This commit is contained in:
Xin Wang
185
docs/easycom_highrate_person_av.md
Normal file
185
docs/easycom_highrate_person_av.md
Normal file
@@ -0,0 +1,185 @@
|
||||
# Global Omni + shared 25 Hz audio + per-person evidence
|
||||
|
||||
## What this implementation does
|
||||
|
||||
The first version deliberately keeps the two jobs separate:
|
||||
|
||||
1. Existing Qwen2.5-Omni states provide the global scene/conversation context
|
||||
`G_t` at the verified 5 Hz task grid.
|
||||
2. MuVAP's frozen CPC frontend encodes the dense session's mixed 16 kHz audio
|
||||
once into a shared causal 25 Hz stream `U_t`; it does not reset at the
|
||||
artificial one-minute file boundaries.
|
||||
3. MuVAP's causal visual track encoder produces one visual state per visible
|
||||
person every 40 ms. Its audio-visual active-speaker readout contributes one
|
||||
speaking logit, so a person token is `[visual_state, ASD_score]`; the shared
|
||||
audio vector is not duplicated inside every person token.
|
||||
4. At training time `G_t + U_t` queries the variable-sized person set using
|
||||
masked attention.
|
||||
5. A zero-initialized residual readout maps the selected person evidence into
|
||||
a correction of the ordinary `G_t + U_t` logits. With matching shared
|
||||
weights, the initial model is exactly the audio-only baseline; the person
|
||||
branch must earn every change during training.
|
||||
6. The ten-horizon floor head predicts the same 0.2--2.0 second EgoSpeak target.
|
||||
|
||||
In compact form:
|
||||
|
||||
```text
|
||||
full scene + mixed audio -> frozen Omni -> G_t --------+--> G+U base logits -----+
|
||||
mixed audio -> frozen CPC once -> U_t -----------------+ sum
|
||||
^
|
||||
25 Hz face track -> causal visual state --+ |
|
||||
U_t + visual state -> ASD speaking score -+-> {P_t,i} -> zero-init residual ----+
|
||||
G_t+U_t queries person set
|
||||
```
|
||||
|
||||
The person branch uses MuVAP's released **ASD module**, not the unreleased/full
|
||||
MuVAP multiparty forecasting module. This is a traditional high-rate
|
||||
active-speaker frontend combined with an Omni global representation.
|
||||
|
||||
## Causality and rate contract
|
||||
|
||||
- MuVAP ASD runs at 25 Hz. Model state `k` consumes the complete interval
|
||||
`[k/25, (k+1)/25)` and is timestamped by its right edge `(k+1)/25`.
|
||||
- Every fifth ASD state ends on a 5 Hz task cutoff. Equality is causal here:
|
||||
the saved state at `0.2s` has consumed samples strictly before `0.2s`.
|
||||
- MuVAP-style chunks score 10 seconds at a time and prepend at most 2 seconds
|
||||
of past context. Chunks overlap only into the past.
|
||||
- A source-clip boundary or missing face observation resets a local face track.
|
||||
The shared audio stream remains continuous on the same concatenated dense
|
||||
timeline used by the targets and global Omni features.
|
||||
- The released MuVAP data loader's whole-recording RMS normalization is not
|
||||
reused: at an early cutoff that statistic would depend on future audio. The
|
||||
extractor uses fixed float-PCM scaling with pointwise clipping only. Record
|
||||
this causal deployment/domain-shift choice when reporting the experiment.
|
||||
- Speech labels, wearer targets and future frames are never extractor inputs.
|
||||
- Provided boxes isolate the representation/fusion question. They are not an
|
||||
online detector result and must be described that way in the paper.
|
||||
|
||||
## External dependency
|
||||
|
||||
The audited MuVAP checkout and published ASD-CPC release are vendored under
|
||||
`third_party/MuVAP`, including an immutable `.upstream-revision` record. For a
|
||||
fresh checkout of this project, no separate MuVAP clone or weight download is
|
||||
needed. The following commands document how that vendored snapshot was
|
||||
obtained and are only needed when intentionally refreshing it:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Haotian-Qi/MuVAP.git third_party/MuVAP
|
||||
git -C third_party/MuVAP checkout a1ef9c8
|
||||
python third_party/MuVAP/tools/fetch_frontends.py cpc
|
||||
|
||||
huggingface-cli download Haotian-Qi/MuVAP \
|
||||
--include 'asd-cpc/*' \
|
||||
--local-dir third_party/MuVAP/weights
|
||||
```
|
||||
|
||||
The extractor imports the pinned vendored checkout directly; it does not need
|
||||
an editable MuVAP install. Check that `torch`, `torchaudio`, `einops` and
|
||||
`yaml` import in the existing Qwen container instead of upgrading its full
|
||||
dependency stack.
|
||||
|
||||
The extractor checks the source revision and hashes the selected release
|
||||
weights into every output contract. If a newer MuVAP revision is intentionally
|
||||
tested, give its exact commit with `--muvap-revision` and use a new output root.
|
||||
|
||||
## EasyCom smoke extraction
|
||||
|
||||
Run a 10-second partial smoke first. A partial sidecar is intentionally
|
||||
rejected by training.
|
||||
|
||||
```bash
|
||||
export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
|
||||
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
|
||||
export HIGHRATE_LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_muvap_asd_25hz_v2
|
||||
export ASD_WEIGHTS=third_party/MuVAP/weights/asd-cpc
|
||||
|
||||
CUDA_VISIBLE_DEVICES=0 python extract_highrate_person_av_features.py \
|
||||
--dataset easycom \
|
||||
--data-root "$DATASET_ROOT" \
|
||||
--feature-root "$FEATURE_ROOT" \
|
||||
--output-root features/highrate_person_av_smoke \
|
||||
--samples 12 \
|
||||
--muvap-root third_party/MuVAP \
|
||||
--asd-weights "$ASD_WEIGHTS" \
|
||||
--max-steps 50 \
|
||||
--device cuda:0 \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
Audit the result:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
import torch
|
||||
|
||||
p = torch.load("features/highrate_person_av_smoke/Session_12.pt", map_location="cpu", weights_only=False)
|
||||
print("format:", p["format"])
|
||||
print("partial:", p["partial"])
|
||||
print("times:", p["times"][:3].tolist(), "...", p["times"][-1].item())
|
||||
print("person features:", tuple(p["persons"]["features"].shape))
|
||||
print("shared audio:", tuple(p["audio"].shape))
|
||||
print("frame offsets:", tuple(p["persons"]["frame_offsets"].shape))
|
||||
print("person-row coverage:", float(p["visual_valid"].float().mean()))
|
||||
print("uses speech labels:", p["config"]["uses_speech_labels"])
|
||||
print("causal policy:", p["config"]["causal_policy"])
|
||||
PY
|
||||
```
|
||||
|
||||
Expected invariants are `causal-highrate-person-av-v2`, `partial=True`, times
|
||||
`[0.2, 0.4, ...]`, shared audio width 256, person width 257 (`256D causal
|
||||
visual state + 1 AV activity logit`), and `uses_speech_labels=False`.
|
||||
|
||||
## Full EasyCom extraction
|
||||
|
||||
Run one command per GPU/terminal. The shard assignments are disjoint and the
|
||||
destination contains real files, not symbolic links:
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 python extract_highrate_person_av_features.py \
|
||||
--dataset easycom --data-root "$DATASET_ROOT" --feature-root "$FEATURE_ROOT" \
|
||||
--output-root "$HIGHRATE_LOCAL_ROOT" --samples 1 2 3 4 5 6 7 8 9 10 11 12 \
|
||||
--num-shards 8 --shard-index 0 --muvap-root third_party/MuVAP \
|
||||
--asd-weights "$ASD_WEIGHTS" --device cuda:0 --dtype bfloat16
|
||||
```
|
||||
|
||||
Repeat with physical GPU and shard index `1` through `7`. Do not add
|
||||
`--overwrite` unless the exact file has been inspected and a replacement is
|
||||
intentional.
|
||||
|
||||
## First matched training screen
|
||||
|
||||
The screen trains on Sessions 4--11 and uses Session 12 for checkpoint
|
||||
selection. It does not inspect Sessions 1--3. Run different seeds on
|
||||
different GPUs:
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 bash run_easycom_highrate_person_av.sh all 0
|
||||
CUDA_VISIBLE_DEVICES=1 bash run_easycom_highrate_person_av.sh all 1
|
||||
CUDA_VISIBLE_DEVICES=2 bash run_easycom_highrate_person_av.sh all 2
|
||||
```
|
||||
|
||||
This compares four matched conditions:
|
||||
|
||||
- `global`: frozen Omni global state `G` only;
|
||||
- `qwen_audio`: existing `G+A_qwen` strong baseline;
|
||||
- `traditional_audio`: global Omni `G` plus one shared CPC stream `U`;
|
||||
- `highrate`: load that seed's trained `G+U` checkpoint, freeze the exact base,
|
||||
and learn only the attention/residual over frozen 25 Hz causal ASD person
|
||||
states. This first screen therefore asks whether the new person evidence
|
||||
adds information, without allowing the base to drift.
|
||||
|
||||
The `all` command runs `traditional_audio` before `highrate`, so the initializer
|
||||
exists. If running separately, run `traditional_audio` first for the same seed.
|
||||
|
||||
Only after selecting the architecture on Session 12 should the fixed
|
||||
checkpoints be run on the official test set:
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 bash run_easycom_highrate_person_av.sh test 0
|
||||
CUDA_VISIBLE_DEVICES=1 bash run_easycom_highrate_person_av.sh test 1
|
||||
CUDA_VISIBLE_DEVICES=2 bash run_easycom_highrate_person_av.sh test 2
|
||||
```
|
||||
|
||||
Dense AP/mAP is a screening/comparability result. The event evaluator still
|
||||
needs competing-risk censoring before claims about false entries, responses or
|
||||
yields are final.
|
||||
Reference in New Issue
Block a user