icassp2027-experiment/third_party/MuVAP/README.md

MuVAP

MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild — Interspeech 2026.

Demo: https://interspeech2026anonymous-eng.github.io/demo/

MuVAP is built from modules that share one codebase and are released one at a time. Each is self-contained: it trains, evaluates, and publishes weights on its own, without the modules that have not landed yet.

Module	Status	Entry point
VAP — voice activity projection from audio	released	train_vap.py
ASD — audio-visual active speaker detection	released	train_asd.py
MuVAP — multiparty fusion of the two	to follow	—

Install

python -m pip install -e .              # add ".[mimi]" for the Mimi frontend

Install the PyTorch build matching your CUDA version first.

The frozen audio frontends download their pretrained weights the first time a model is built. Fetch them ahead of time — worth doing before a batch job, on a machine whose compute nodes have no outbound network:

python tools/fetch_frontends.py          # or: fetch_frontends.py cpc

VAP

Voice Activity Projection: a causal model that predicts who will hold the floor next, from audio alone. Every configuration shares one training program and differs only in the config file, so a result difference between them comes from the setup rather than from a different pipeline.

Configuration	Config	Input	Codebook
Role-based VAP	config/yaml/vap_role.yaml	1 mixed channel	136 role-relative classes
Speaker-based VAP	config/yaml/vap_speaker.yaml	2 channels, one per speaker	256 classes, one channel per row
Speaker-based, mono	config/yaml/vap_speaker_mono_mimi.yaml	1 mixed channel + VAD	256 classes, one channel per row

The mono setup is the original VAP arrangement: one channel carries the audio and the causal voice activity behind each frame supplies the speaker identity a second channel would have carried. config/yaml/ also holds the frontend and frame-rate variants of each - *_mimi_12hz, *_cpc_50hz.

python train_vap.py --config config/yaml/vap_role.yaml --name role

Positional encoding

vap.temporal.pos_encoding selects alibi, causal rope, sinusoidal, or none. The stack stays causal regardless — only how position reaches attention changes.

See docs/vap.md for what each setup means, what it logs, how to prepare Fisher, and the exact causality and alignment contracts.

ASD

Audio-visual active speaker detection: given a face track and the audio around it, decide frame by frame whether that face is speaking. The reported number is the official AVA-ActiveSpeaker mAP.

Configuration	Config	Frontend
ASD	config/yaml/asd.yaml	CPC
ASD + Mimi	config/yaml/asd_mimi.yaml	Mimi

The frontends are the same two the VAP module uses, and both emit features at the 25 Hz rate the visual frontend runs at, so they substitute for each other here too.

python train_asd.py --config config/yaml/asd.yaml --name asd

Faces reach the model as 112x112 grayscale, the geometry the packs are written in - see preprocessing for where that is fixed and why. On top of it, asd.augment applies one geometric transform per chunk, drawn uniformly from none, horizontal flip, random crop rescaled back to 112, and rotation; crop_scale is the side of that crop as a fraction of the frame. Audio is augmented separately: with probability audio_mix_prob a donor clip is mixed in at an SNR drawn from audio_snr_db while the label stays unchanged, so the model has to reject speech it cannot see on the face.

augment:
  audio_mix_prob: 0.5          # chance of mixing in off-screen speech
  audio_snr_db: [-5.0, 5.0]    # how loud that donor is against the real audio
  visual: true                 # geometric transforms on the face track
  crop_scale: [0.7, 1.0]       # random-crop side, as a fraction of 112
  rotate_degrees: 15.0         # maximum rotation
  substitute_audio_prob: 0.0   # ablation: replace the audio, relabel silent

Every source corpus is packed into one memory-mapped format first. A pack is a decoded mirror of the corpus, not a model-ready tensor set: grayscale face crops and one label per frame at the native video rate, plus 16 kHz audio at its original gain. Sampling onto the 25 Hz model timeline and loudness normalization happen in the dataloader, so one pack can serve models with different conventions.

python -m preprocess.asd.prepare ava --split train \
  --loader /data/AVA/csv/train_loader.csv \
  --visual-root /data/AVA/clips_videos --audio-root /data/AVA/clips_audios
python -m preprocess.asd.validate /data/AVA/packed/train

python -m preprocess.asd.prepare --help lists the accepted source schemas. See docs/preprocessing.md and docs/data_alignment.md for the storage and alignment contracts.

Audio convention. The model expects audio normalized to -25 dBFS RMS, capped at 20 dB of gain and a 0.99 peak, with the level measured over the whole recording rather than the evaluated slice. This is not the raw-waveform convention other ASD codebases use, and a checkpoint evaluated on unnormalized audio will not reproduce the mAP it was trained to. Packs store audio unnormalized and the loader applies this; the constants live in data/media.py.

Running either module

--set KEY=VALUE overrides any config value (dotted path, YAML-parsed) and can be repeated; it fails on a key the config does not already define, so a typo cannot silently do nothing. --wandb enables tracking, --name chooses the run directory.

checkpoint_monitor picks what best.ckpt selects on — val/loss for VAP, val/mAP_official for ASD. Clear it where there is no held-out split to select on, and the final epoch is the only artifact the run keeps:

python train_vap.py --config config/yaml/vap_role.yaml --set vap.checkpoint_monitor=

--test then falls back to last instead of best. See Weight releases for what each checkpoint is.

Audio frontends

vap.audio_encoder selects the frozen frontend. Both emit features at the rate the projection window labels, so they are drop-in alternatives at a shared rate.

	cpc (default)	mimi
Weights	CPC, LibriLight 60k	kyutai/mimi, from Moshi
Frozen params	1.8M	37.8M
Feature dim	256	512
Frame rates	100 / 50 / 25 / 12.5	25 / 12.5

CPC halves a 100 Hz convolutional stack down to the requested rate; Mimi taps either its 25 Hz encoder output or its 12.5 Hz latent, so 50 Hz is a CPC-only option.

Both are causal, which is what makes the model usable as a streaming predictor.

Weight releases

Trained weights live on the Hub at Haotian-Qi/MuVAP.

Two ASD releases, asd-cpc and asd-mimi, scoring 90.50 and 92.04 mAP_official on AVA-ActiveSpeaker. The VAP releases span three codebooks, two frontends and three frame rates - docs/vap.md lists them with their scores.

Fetch one, then evaluate it:

huggingface-cli download Haotian-Qi/MuVAP --include 'vap-role-mimi/*' --local-dir weights

python train_vap.py --config config/yaml/vap_role.yaml \
    --test --weights weights/vap-role-mimi \
    --set fisher_path=/path/to/fisher

That reproduces the score in the table exactly. All of them at once:

huggingface-cli download Haotian-Qi/MuVAP --local-dir weights

A run produces two checkpoints. last.ckpt is the final epoch — the artifact the run committed to. best.ckpt is the epoch a validation metric picked, and exists only when checkpoint_monitor is set; where there is no held-out split to select on, clear it and the final epoch is the only artifact.

Publish either one with:

python tools/export_checkpoint.py <run>/last.ckpt <release dir> --module vap

A release carries the trained weights alone: no optimizer state, and no copy of the frozen audio frontend, which is fetched from its own pretrained source when the model is built. It ships config.yaml (the architecture) and provenance.json, which records the checkpoint it came from and what that checkpoint scored. Point --weights at a release directory to evaluate it:

python train_vap.py --config config/yaml/vap_role.yaml --test --weights <release>

The architecture comes from the release's own config.yaml; dataset roots and output directories come from the --config you pass, so a release runs anywhere without reproducing the paths it was trained with. A release that does not fit the config it is given fails immediately, naming the offending weights.

Or load one directly:

import yaml
from models.release import load_weights, resolve
from models.vap import build_vap

weights, config = resolve("path/to/release")
cfg = yaml.safe_load(open(config))["vap"]
model = load_weights(build_vap(cfg), weights).eval()
logits = model(waveform)          # [batch, frames, classes] at the release's rate