docs: refine person-aware manuscript method

literature_notes/33_muvap.md

@@ -10,6 +10,12 @@
 
 将单声道混合音频与逐人脸轨迹结合，先学习声学轮次和因果 ASD 表示，再融合为全局及逐人未来活动预测。用相对发言角色控制多人联合状态空间；任务包含 shift/hold 和 next speaker。是我们逐人时序旁路最直接的相关工作。
 
+## 可复用的结构细节
+
+MuVAP 不是对多人脸做普通平均。它先用 CPC 与 causal Transformer 得到全局音频 VAP 表示；每条脸轨再经修改后的 causal TalkNet 得到 ASD 表示。两路分别投影到 256 维后，全局音频作为 query、同一时刻的可见人物作为 key/value 做 frame cross-attention，随后用 temporal Transformer 建模群体动态。全局表示再通过 gated addition 调制每个人物表示，并分别输出 GlobalVAP 与 SpeakerVAP。ASD 预训练还包含当前逐人活动辅助损失和音视频对比损失。
+
+对本项目最有价值的迁移是“全局 floor query + 匿名人物集合 + 人内因果历史 + 门控残差”，而不是它的 Role-Relative Projection。我们的输出主体始终是不可见 wearer 的 entry/yield，不需要把动态人数展开为 next-visible-speaker 分类。实际批处理可用 padding mask，零人场景加入 NULL/OFFSCREEN token，临时 track ID 只路由共享时序状态而不作为 embedding。
+
 不能把“单声道＋逐人视觉＋因果接话”作为独有贡献。我们的区别需要落在不可见 wearer、讲话对象及受召回约束的误开口上，而不是只换编码器。
 
-作者承认角色识别存在延迟；验证主要为第三人称双人/三人英语场景。数据预处理的全片身份锚点也需与严格在线 tracking 分开讨论。仓库当前只有 README/LICENSE，写明 code coming soon；不要称已有可直接运行的复现。
+作者承认角色识别存在延迟；验证主要为第三人称双人/三人英语场景。其数据准备使用 RetinaFace/SCRFD 检测、ArcFace 特征、匈牙利匹配，并从每段完整视频随机抽取 50 帧建立全局身份锚点。这适合构建稳定标注，但会使用未来/全片信息，不能作为本项目部署时的严格在线 tracking。论文模型本身宣称 causal，不代表这套数据身份初始化也是可在线运行的。仓库当前只有 README/LICENSE，写明 code coming soon；不要称已有可直接运行的复现。

manuscript/main/.gitignore

@@ -1 +1,15 @@
 /build/
+
+# LaTeX intermediates accidentally emitted beside main.tex when latexmk/pdflatex
+# is invoked without the Makefile's build directory.
+/*.aux
+/*.bbl
+/*.bcf
+/*.blg
+/*.fdb_latexmk
+/*.fls
+/*.log
+/*.out
+/*.run.xml
+/*.synctex.gz
+/*.toc

manuscript/main/main.tex

@@ -109,14 +109,16 @@ Related work supplies complementary parts of the problem. Egocentric
 talking-to-me detection predicts whether a completed utterance addresses the
 camera wearer and benefits from long-term social context
 \cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face
-tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}.
+tracks to predict floor shifts and next visible speakers \cite{qi2026muvap},
-The former is generally utterance-level rather than continuous anticipation;
+while AV-CONV reconstructs current person-level speaking/listening relations
-the latter predicts visible participants rather than the action of an
+from egocentric audio--video \cite{jia2024avconv}. The former talking-to-me
-egocentric conversational agent represented by a wearer-derived target. Our
+task is generally utterance-level rather than continuous anticipation;
-gap is therefore narrower than ``audio--visual multiparty turn taking'': we
+MuVAP predicts visible participants from a static third-person view; and
-study whether omni representations support direct, continuous agent
+AV-CONV estimates a current interaction graph rather than strictly causal
-floor-action prediction and whether causal local evidence helps when the
+future wearer action. Our gap is therefore narrower than ``audio--visual
-visible participant set changes.
+multiparty turn taking'': we study whether omni representations support
+direct, continuous agent floor-action prediction and whether causal local
+evidence helps when the visible participant set changes.
 
 We call this task \emph{egocentric floor-action projection}. The study asks
 four questions: (RQ1) how accurately a frozen global omni state predicts the
@@ -142,10 +144,11 @@ final method only if it beats pooled, no-ID, and capacity-matched controls.
 \centering
 \fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
 \todo{After model selection, draw the final system figure. For now it must show
-the shared experiment interface: raw causal AV, backbone-specific global
+the shared experiment interface: raw causal AV; online face/head detection and
-readout, frozen/PEFT/local-adapter branches, optional recent audio and anonymous
+causal tracking; backbone-specific global readout; recent audio and a masked
-person histories, and the common $10\times3$ head. Mark causal cutoffs,
+set of anonymous track states; the gated local adapter; and the common
-trainable modules, parameter budgets, and source-boundary resets.}}}
+$10\times3$ head. Mark causal cutoffs, trainable modules, parameter budgets,
+NULL/OFFSCREEN handling, and source-boundary resets.}}}
 \caption{Planned omni-model study for egocentric conversational floor action.
 The local adapter remains a candidate family until the controlled train-side
 comparison is complete.}
@@ -209,27 +212,56 @@ protocol is locked.}
 The current hypothesis is that global omni states preserve broad semantic
 context but may make recent prosody, activity boundaries, and individual
 visual motion difficult for a shallow floor head to recover. From the most
-recent $L_a$ seconds, a local encoder produces audio evidence $\vect A_t$ and,
+recent $L_a$ seconds, a causal local audio encoder produces $\vect a_t$. At
-optionally, an anonymous visible-person set $\mathcal P_t$. A generic candidate
+time $t$, an online detector and tracker produce a variable-size anonymous set
-adapter is
+$\mathcal P_t=\{(\vect v_{t,i},\vect b_{t,i},\vect r_{t,i})\}_{i=1}^{N_t}$,
+where $\vect v_{t,i}$ is a crop feature, $\vect b_{t,i}$ is box geometry, and
+$\vect r_{t,i}$ contains only causal reliability variables such as visibility,
+track age, and time since observation. A shared causal track encoder updates
 \begin{equation}
- \vect z^{(m)}_t=\vect g^{(m)}_t+
+ \vect h_{t,i}=F_{\mathrm{trk}}\!\left(\vect h^-_{t,i},
- \gamma_t H_\phi(\vect g^{(m)}_t,\vect A_t,\mathcal P_t),
+ W_v\vect v_{t,i}+W_b\phi(\vect b_{t,i})+W_r\vect r_{t,i}\right).
- \qquad \gamma_t\vert_{\mathrm{init}}=0,
+ \label{eq:track}
- \label{eq:adapter}
 \end{equation}
-followed by the common prediction head. Local evidence is derived from the
+All tracks share $F_{\mathrm{trk}}$; the temporary ID only retrieves
-same causal raw media, not close microphones, transcripts, speaker labels, or
+$\vect h^-_{t,i}$ and is never embedded. The global-audio query then reads the
-future frames.
+current person set by masked cross-attention,
+\begin{align}
+ \vect u_t &= \mathrm{MHA}\!\left(
+   W_q[\vect g^{(m)}_t;\vect a_t],
+   [\vect h_{\varnothing};\vect h_{t,1};\ldots;\vect h_{t,N_t}]
+ \right), \\
+ \vect z^{(m)}_t &= \vect g^{(m)}_t+\tanh(\alpha)W_o\vect u_t,
+ \qquad \alpha\vert_{\mathrm{init}}=0,
+ \label{eq:adapter}
+\end{align}
+where $\vect h_{\varnothing}$ is a learned NULL/OFFSCREEN token. Padding is
+masked, so the same network handles $N_t=0$ and changing group size. For a
+bounded implementation, $N_t>N_{\max}$ is resolved by a deterministic ranking
+using detector confidence and track age available at $t$, and truncation is
+reported. The operator is permutation equivariant
+\cite{lee2019settransformer}; temporary tracker IDs provide continuity, not
+identity semantics.
+
+This decomposition is informed by MuVAP: its global audio representation
+queries same-frame causal ASD track representations, followed by temporal
+modeling and gated global-to-speaker fusion \cite{qi2026muvap}. We retain the
+useful separation between global floor dynamics and person-grounded evidence,
+but not its role-relative output: our fixed target is the invisible wearer's
+entry/yield action, not the next visible speaker. The zero-initialized gate
+also follows the conservative adapter principle used to inject visual evidence
+into strong audio models \cite{rouditchenko2024whisperflamingo}. Local evidence
+is derived from the same causal raw media, not close microphones, transcripts,
+speaker labels, global identity anchors, or future frames.
 
 We evaluate candidates in increasing structural complexity: pooled recent
 audio/vision; a capacity-matched temporal branch; a per-frame person set with
 no cross-time identity; a shared track-wise temporal encoder; and person--audio
-cross-attention. Only a candidate that improves the pre-registered event metric
+cross-attention as in Eqs.~\eqref{eq:track}--\eqref{eq:adapter}. Only a
-over simpler and capacity-matched controls becomes the final method. If
+candidate that improves the pre-registered event metric over simpler and
-person continuity is selected, temporary IDs route hidden state only---there
+capacity-matched controls becomes the final method. A limited latent
-is no ID embedding, name recognition, or cross-recording identity---and the set
+bottleneck is tested only if large visible sets make full person attention a
-operator is permutation equivariant \cite{lee2019settransformer}.
+runtime bottleneck; it is not assumed to improve accuracy.
 
 \todo{Run the candidate ladder before fixing $H_\phi$. Match trainable
 parameters and effective local history. Compare a shared local encoder across
@@ -263,19 +295,34 @@ fold; do not inspect Sessions 1--3 during selection.}
 
 \subsection{Track source and deployment contract}
 
-Person-aware candidates first use dataset-provided boxes and anonymous IDs to
+The intended inference pipeline runs a face/head detector on each incoming
-isolate the value of local visual structure. A deployable version instead
+frame and associates detections with an online multi-object tracker using only
-obtains causal tracklets from a face/head detector followed by an online
+observations at or before $t$. Its temporary IDs route the shared recurrent
-multi-object tracker; no future-frame smoothing or cross-session recognition
+state in Eq.~\eqref{eq:track}; they do not denote names, fixed seats, or
-is permitted. EasyCom IDs are continued only across genuinely adjacent source
+cross-session identities. New tracks initialize from a learned state, short
-segments and reset across redacted gaps. Ego4D IDs are local to a clip.
+misses retain a causally aged state, and expired tracks are removed. With no
+visible face, the global/audio path and NULL/OFFSCREEN token remain valid. No
+future-frame smoothing, full-video identity anchor, or cross-session
+recognition is permitted. EasyCom state is continued only across genuinely
+adjacent source segments and reset across redacted gaps; Ego4D state is local
+to a clip.
 
-\todo{If a person-aware candidate is selected, state precisely whether face or
+During method isolation, dataset-provided boxes and anonymous IDs form a
-head boxes are used. Report provided-track
+separate \emph{provided-track} condition. This answers whether person
-coverage, mean/maximum people, occlusion gaps, track fragments, and source-gap
+structure can help under accurate localization, but is not reported as online
-resets for both datasets. Add either one online detector/tracker experiment or
+detector performance. The deployable \emph{online-track} condition runs the
-controlled box jitter, dropout, fragmentation, and ID-switch curves; otherwise
+detector/tracker above and includes its errors and latency. This separation is
-limit every deployment claim to ``given causal participant tracklets.''}
+important because MuVAP used random full-video identity anchors to build its
+dataset tracks; that preprocessing is useful for annotation but is not a
+strictly online association algorithm \cite{qi2026muvap}.
+
+\todo{Select and freeze the face/head detector, causal tracker, thresholds,
+maximum age, and $N_{\max}$ on training-side data. Report detector recall,
+provided/online-track coverage, mean/maximum people, truncations, ID switches,
+fragments, occlusion gaps, source resets, and P50/P95 detector--tracker latency
+for both datasets. Compare provided tracks, online tracks, and controlled box
+jitter/dropout/fragmentation. Until the online row is run, limit deployment
+claims to the intended pipeline and label all numbers ``provided-track.''}
 
 \section{Experiments}
 \label{sec:setup}
@@ -339,7 +386,8 @@ This compact matrix tests the representation hypothesis before investing in a
 complex fusion module. If local vision survives three seeds and the event
 gate, we then compare a parameter-matched temporal branch without person
 continuity, a separately trained per-frame/no-ID set branch, and a
-track-consistent branch. On a frozen track-aware checkpoint, a clip-consistent
+track-consistent masked cross-attention branch. Provided-track and online-track
+inputs are reported separately. On a frozen track-aware checkpoint, a clip-consistent
 global renaming must leave outputs unchanged, whereas \emph{broken
 continuity} periodically remaps IDs while preserving every crop, timestamp,
 audio sample, and target. Time-shifted and removed vision provide additional
@@ -365,7 +413,8 @@ decision latency, throughput, peak memory, and trainable parameter count.
 
 \todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions,
 $L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter
-count, hardware, and real-time latency after the architecture is frozen.}
+count, tracker timeout/ranking, hardware, and end-to-end real-time latency
+after the architecture is frozen.}
 
 \section{Results and Analysis}
 \label{sec:results}

manuscript/main/references.bib

@@ -63,6 +63,22 @@
   year      = {2019}
 }
 
+@inproceedings{rouditchenko2024whisperflamingo,
+  author    = {Andrew Rouditchenko and Yuan Gong and Samuel Thomas and Leonid Karlinsky and Hilde Kuehne and Rogerio Feris and James Glass},
+  title     = {{Whisper-Flamingo}: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation},
+  booktitle = {Proc. Interspeech},
+  pages     = {2420--2424},
+  year      = {2024}
+}
+
+@inproceedings{jia2024avconv,
+  author    = {Wenqi Jia and Miao Liu and Hao Jiang and Ishwarya Ananthabhotla and James M. Rehg and Vamsi Krishna Ithapu and Ruohan Gao},
+  title     = {The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective},
+  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
+  pages     = {26396--26405},
+  year      = {2024}
+}
+
 @inproceedings{grauman2022ego4d,
   author    = {Kristen Grauman and Andrew Westbury and Eugene Byrne and others},
   title     = {{Ego4D}: Around the World in 3,000 Hours of Egocentric Video},