Add causal person fusion and MuVAP stage-3 baselines

third_party/MuVAP/docs/data_alignment.md

@@ -0,0 +1,96 @@
+# Audio-visual data contract
+
+## Canonical timeline
+
+MuVAP+ uses a **25 Hz model timeline**, because the CPC stack emits one audio
+embedding per 640 samples at 16 kHz. Original videos are not transcoded or
+rewritten to 25 fps.
+
+For model frame `k`, the timestamp is the centre of its interval:
+
+```text
+t(k) = (k + 0.5) / 25 seconds
+```
+
+The visual face and label are selected from the native source interval that
+contains `t(k)`:
+
+```text
+source_index(k) = floor(t(k) * source_fps)
+```
+
+The exact same index is used for the face and its label. This prevents label
+drift. At 30 fps some source frames are skipped; below 25 fps some are repeated.
+
+**This happens in the loader, not in the pack.** Packed shards store the native
+frame rate, so the sampling above is applied when a batch is built and can be
+changed - or replaced with a different `model_fps` - without re-packing.
+
+## Evaluation on the native timeline
+
+Metrics defined on the source annotations — the official AVA mAP above all —
+need one score per native frame, not per model frame. `model_indices` reverses
+the mapping by asking which model interval contains each source-frame centre:
+
+```text
+model_index(j) = floor(((j + 0.5) / source_fps) * 25)
+```
+
+The result always has exactly `source_frames` entries, so every groundtruth row
+receives a score at any frame rate. Below 25 fps this is an exact inverse of
+`source_indices`. Above it, several source frames share a model frame, which is
+information the 25 Hz timeline genuinely does not carry. Only the final source
+frame can fall past the last complete model interval; it is clamped to that
+interval, and no interior frame is ever mis-scored.
+
+## Audio
+
+Audio is loaded directly from WAV/FLAC (or NumPy waveforms), mixed to
+mono, and resampled to 16 kHz for CPC. Packed shards keep the full span the face
+frames cover, at the recording's own gain.
+
+The loader takes it from there. A batch of `T` model frames always carries
+exactly `T * 640` samples, right-padded with silence if the recording is short.
+Loudness normalization to -25 dBFS RMS is applied at that point, using level
+statistics measured over the **whole recording** and stored in the index -
+computing gain from a chunk alone would make a sample's level depend on which
+chunk it landed in.
+
+## Required manifest fields
+
+Each row must resolve to:
+
+| Field | Meaning |
+|---|---|
+| `sample_id` | Stable, unique face-track/clip identifier |
+| `audio_path` | Recording or clip audio path |
+| `visual_path` | NumPy face array shaped `[source_frames, H, W]` |
+| `source_fps` | Native FPS used by faces and labels |
+| `labels` | One active-speaker label per native face frame |
+| `start_sec` | Clip start on the audio recording timeline; default `0` only when audio is already clip-local |
+
+The canonical file is a tab-separated text file with a header. `labels` is a
+JSON array; paths may be absolute or relative to the manifest directory:
+
+```text
+sample_id\taudio_path\tvisual_path\tsource_fps\tlabels\tstart_sec
+clip_001\taudio/recording.wav\tfaces/clip_001.npy\t29.97\t[0,0,1,1]\t12.48
+```
+
+`len(labels)` and the face-frame count must match. Preprocessing rejects a
+sample where they disagree rather than trimming to the shorter one: a length
+mismatch means the face track and the annotation disagree about the clip, which
+would silently shift every later label.
+
+## Dataset policy
+
+- AVA, WASD, and MSDWild use the same alignment implementation.
+- FPS is never ignored and must never be replaced with a global constant.
+- Packs are decode-only. Any transform whose parameters belong to the model -
+  frame rate, loudness, crop duration - happens in the loader, so a pack stays
+  usable by code that does not share those conventions.
+- Labels are aligned before projection-window targets are computed.
+- Train-time batching may crop examples to a shared duration, but audio, faces,
+  VAD labels, and projected labels must all use that same duration.
+- A WASD face track using full-recording audio must provide `start_sec`. It is
+  mathematically impossible to recover that offset from FPS and labels alone.