Add causal person fusion and MuVAP stage-3 baselines

This commit is contained in:
Xin Wang
2026-09-10 06:54:22 +08:00
parent 0534c9d3d0
commit d426b4740a
225 changed files with 27716 additions and 119 deletions

38
third_party/README.md vendored
View File

@@ -1,7 +1,9 @@
# Third-party reference code
Reference repositories are inspected here but are not vendored into this
project. They are not runtime dependencies.
Audited reference repositories are vendored here so a normal clone of this
project contains the exact source used by the experiments. The nested
upstream `.git` directories are deliberately removed; each checkout records
its immutable source commit in `.upstream-revision`.
The track-GRU design was checked against the official EgoSpeak repository at
revision `b5bee35e796d8f2d6aa7da2bf1b07836270f8bfb`:
@@ -11,5 +13,33 @@ git clone https://github.com/jun297/EgoSpeak.git third_party/EgoSpeak
git -C third_party/EgoSpeak checkout b5bee35e796d8f2d6aa7da2bf1b07836270f8bfb
```
`third_party/EgoSpeak/` is intentionally ignored. No EgoSpeak source file is
copied or imported by the project implementation.
The project implementation does not import EgoSpeak at runtime. Its source is
retained for implementation comparison and license/provenance auditing.
The high-rate local AV extractor imports the vendored official MuVAP ASD
implementation at audited revision `a1ef9c8` and its published `asd-cpc`
weights. The MuVAP-only Stage-3 reconstruction additionally uses the published
25 Hz `vap-role-cpc` release. To refresh this vendored copy intentionally,
clone and audit the new revision outside the project first, then update the
source, weights and `.upstream-revision` together:
```bash
git clone https://github.com/Haotian-Qi/MuVAP.git third_party/MuVAP
git -C third_party/MuVAP checkout a1ef9c8
python third_party/MuVAP/tools/fetch_frontends.py cpc
huggingface-cli download Haotian-Qi/MuVAP \
--include 'asd-cpc/*' 'vap-role-cpc/*' \
--local-dir third_party/MuVAP/weights
```
The published `asd-cpc`/`vap-role-cpc` releases and CPC frontend are retained
in this directory. The extractors hash each consumed release into every
feature contract.
It imports this exact checkout directly, so an editable install is not
required. In an established Qwen container, avoid letting MuVAP's package
metadata replace the working PyTorch/Transformers stack. The additional import
requirements are `torchaudio`, `einops`, and `PyYAML`; verify/install only the
missing packages.
See [`docs/easycom_highrate_person_av.md`](../docs/easycom_highrate_person_av.md)
for the causality contract and experiment commands.