4.0 KiB
Audio-visual data contract
Canonical timeline
MuVAP+ uses a 25 Hz model timeline, because the CPC stack emits one audio embedding per 640 samples at 16 kHz. Original videos are not transcoded or rewritten to 25 fps.
For model frame k, the timestamp is the centre of its interval:
t(k) = (k + 0.5) / 25 seconds
The visual face and label are selected from the native source interval that
contains t(k):
source_index(k) = floor(t(k) * source_fps)
The exact same index is used for the face and its label. This prevents label drift. At 30 fps some source frames are skipped; below 25 fps some are repeated.
This happens in the loader, not in the pack. Packed shards store the native
frame rate, so the sampling above is applied when a batch is built and can be
changed - or replaced with a different model_fps - without re-packing.
Evaluation on the native timeline
Metrics defined on the source annotations — the official AVA mAP above all —
need one score per native frame, not per model frame. model_indices reverses
the mapping by asking which model interval contains each source-frame centre:
model_index(j) = floor(((j + 0.5) / source_fps) * 25)
The result always has exactly source_frames entries, so every groundtruth row
receives a score at any frame rate. Below 25 fps this is an exact inverse of
source_indices. Above it, several source frames share a model frame, which is
information the 25 Hz timeline genuinely does not carry. Only the final source
frame can fall past the last complete model interval; it is clamped to that
interval, and no interior frame is ever mis-scored.
Audio
Audio is loaded directly from WAV/FLAC (or NumPy waveforms), mixed to mono, and resampled to 16 kHz for CPC. Packed shards keep the full span the face frames cover, at the recording's own gain.
The loader takes it from there. A batch of T model frames always carries
exactly T * 640 samples, right-padded with silence if the recording is short.
Loudness normalization to -25 dBFS RMS is applied at that point, using level
statistics measured over the whole recording and stored in the index -
computing gain from a chunk alone would make a sample's level depend on which
chunk it landed in.
Required manifest fields
Each row must resolve to:
| Field | Meaning |
|---|---|
sample_id |
Stable, unique face-track/clip identifier |
audio_path |
Recording or clip audio path |
visual_path |
NumPy face array shaped [source_frames, H, W] |
source_fps |
Native FPS used by faces and labels |
labels |
One active-speaker label per native face frame |
start_sec |
Clip start on the audio recording timeline; default 0 only when audio is already clip-local |
The canonical file is a tab-separated text file with a header. labels is a
JSON array; paths may be absolute or relative to the manifest directory:
sample_id\taudio_path\tvisual_path\tsource_fps\tlabels\tstart_sec
clip_001\taudio/recording.wav\tfaces/clip_001.npy\t29.97\t[0,0,1,1]\t12.48
len(labels) and the face-frame count must match. Preprocessing rejects a
sample where they disagree rather than trimming to the shorter one: a length
mismatch means the face track and the annotation disagree about the clip, which
would silently shift every later label.
Dataset policy
- AVA, WASD, and MSDWild use the same alignment implementation.
- FPS is never ignored and must never be replaced with a global constant.
- Packs are decode-only. Any transform whose parameters belong to the model - frame rate, loudness, crop duration - happens in the loader, so a pack stays usable by code that does not share those conventions.
- Labels are aligned before projection-window targets are computed.
- Train-time batching may crop examples to a shared duration, but audio, faces, VAD labels, and projected labels must all use that same duration.
- A WASD face track using full-recording audio must provide
start_sec. It is mathematically impossible to recover that offset from FPS and labels alone.