diff --git a/AGENTS.md b/AGENTS.md index 76604b5..988a66d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -62,6 +62,30 @@ EasyCom 遵循官方 Sessions 4--12/1--3 划分;十个 0.2--2.0 秒 horizon raw onset/offset、局部音视频和 oracle/时序消融应区分实测、历史非匹配 screen、 仅用于分析的 privileged oracle 与待测方案。raw endpoint AP 不等同语义 EOT/INT。 +## Targeted manuscript 的 Method 固定结构 + +Method 按以下三个小节展开,除非用户明确要求,不再改变这个组织顺序;具体特征和 +训练状态以代码、结果记录与 [`manuscript/targeted/main.tex`](manuscript/targeted/main.tex) +为准。 + +1. **Task and Global--Local Fusion。** 定义 5 Hz 因果截止点、十个未来 horizon 和 + background/wearer/others 输出;说明 Thinker 全局状态本身已含音视频,再定义 SWE + 的近期音频及可选视觉旁路。区分 pooled 特征的拼接聚合与 F2/F3 保留视觉 token 的 + query attention。训练范围分开写:主方案只训练 readout/局部分支/预测头;Thinker + LoRA、限制在上层的 LoRA、预测头冻结或联合训练是参数适配控制。最后四层 + **readout** 与 LoRA 更新哪些层是两个独立维度。已归档 LoRA 结果为非匹配探索, + 不能写成层数消融或优劣定论。 +2. **Causal Partial-Tail Refresh。** 完成块进入持久历史;每个 200 ms 截止点只临时查询 + 当前 2 秒块已观测的前缀,尾块不提交、不中途引入未来媒体。与 completed-only + 在同一截止点和上下文上限下配对比较;未完成实验前只称因果访问规则。 +3. **Short-Window Feature Extraction。** 默认局部音频为截止点前一秒混合波形经冻结 + Qwen 音频编码器;EgoSpeak 式 wav2vec2 卷积编码是待做的同窗口、同预测头对照, + 不能冒充已测 G+A。视觉区分最近单帧、最近一秒多帧人物框 pooled 特征、无框的 + F2 最新全画面空间 token,以及 F3 同网格位置两帧 token 经共享 GRU 后注意力聚合。 + 另列一种特殊的短窗信息审计:由转写标注生成、仅含截止点之前的可见非 wearer + 活动历史,比较逐人、打乱、延迟和 OR 聚合等形式。它不是部署时可获得的视觉特征, + 也不是性能上界。F3 不是人物跟踪,F3--F2 差异也不能单独归因于运动理解。 + ## 不可破坏的研究契约 - 严格因果:任何输入音频、视频帧、人物框或归一化统计不得来自预测截止时间之后。 diff --git a/manuscript/targeted/main.tex b/manuscript/targeted/main.tex index 6c15c90..4371b42 100644 --- a/manuscript/targeted/main.tex +++ b/manuscript/targeted/main.tex @@ -143,7 +143,7 @@ committed to persistent history.} \section{Method} \label{sec:method} -\subsection{Task and Short-Window Encoder Evidence} +\subsection{Task and Global--Local Fusion} At the 5-Hz cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and predicts activity at $t_n+h$, where @@ -152,32 +152,48 @@ horizon has three channels in fixed order: background, camera wearer, and aggregate others. Wearer and others may overlap; no output identifies an individual speaker, addressee, or semantic end of turn. -The global feature $\vect g_n$ concatenates the final four frozen -Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It -already contains audio and video context; $\G$ is not audio-free. SWE -provides a separate short-window path so the head need not recover all -cutoff-local detail from the integrated state. Local audio $\vect a_n$ crops -the latest one-second mixed -waveform \emph{before} preprocessing, runs the frozen Qwen audio encoder, and -averages valid tokens. Optional $\vect v_n$ averages frozen vision tokens -inside provided person boxes, then across people and valid causal 2-FPS -frames in the latest second. This pooling has no identity or mouth-motion trajectory; -provided boxes do not imply an online detector. +The global representation $\vect g_n$ concatenates the final four frozen +Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It +already integrates audio and video over the available context; $\G$ is not +an audio-free baseline. Short-window encoder evidence (SWE) adds a direct, +cutoff-aligned path for recent audio $\vect a_n$ and, when enabled, visual +evidence $\vect v_n$. This separates the question of what the global state +encodes from whether a small prediction head can access transient cues. -Each enabled local feature is layer-normalized, projected to 256 dimensions, -and paired with a validity bit. The trainable head is +In the pooled fusion, each available local vector is layer-normalized, +projected to 256 dimensions, and accompanied by a validity bit. The +concatenated global and local features are passed to a layer-normalized MLP: \begin{equation} - \widehat{\vect Y}_{n}= - \operatorname{softmax}_{c}F_{\theta} - \bigl[\vect g_n;\widetilde{\vect a}_n;m_n^a; - \widetilde{\vect v}_n;m_n^v\bigr], + \widehat{\vect Y}_{n,h,c}=\operatorname{softmax}_{c} + F_{\theta}\!\left([\vect g_n;\widetilde{\vect a}_n;m_n^a; + \widetilde{\vect v}_n;m_n^v]\right)_{h,c}, \label{eq:fusion} \end{equation} -where omitted modalities remove their terms and $F_\theta$ is a -layer-normalized MLP with a 30-logit output. We train only the projections and -head with the EgoSpeak multi-positive cross-entropy over causal-valid -source--horizon pairs. The Omni and modality encoders remain frozen, so this -is frozen-backbone adaptation rather than end-to-end training. +where a disabled branch is omitted and $F_\theta$ produces ten sets of +three logits. For token-preserving visual variants, $\vect v_n$ denotes +an attention-pooled context: a query formed from $\vect g_n$ and valid +$\vect a_n$ attends to the causal visual token set (and a learned null +token), rather than averaging it beforehand. The resulting context and +validity bit enter the same prediction head. We train the projections, +visual aggregator when present, and head with the EgoSpeak multi-positive +cross-entropy over causal-valid source--horizon pairs. The principal SWE +configuration freezes all feature encoders and the Thinker: this is +frozen-backbone adaptation, not task-level end-to-end training. + +We separate \emph{where} parameters are updated from \emph{which} Thinker +layers are read. The primary comparison trains only the layer-combination +projection, local-fusion modules, and 30-logit prediction head on cached +frozen features. Parameter-adaptation controls instead attach low-rank +adapters to Thinker attention projections, either across the configured +Thinker or restricted to its upper layers; the prediction head may be +updated jointly or held fixed to isolate representation changes. These +controls require the same causal media, source cutoffs, supervision, +initialization, and model-selection protocol as the frozen-head condition. +Archived LoRA runs, including an upper-eight-layer setting, are +exploratory and not a matched layer-scope ablation. Last-four-layer +\emph{readout} fusion changes the head input, not the LoRA update scope; +neither its gain nor an unmatched LoRA score establishes an adaptation +advantage. \subsection{Causal Partial-Tail Refresh} @@ -193,18 +209,42 @@ completed-only versus PTR readouts with $\A$ on/off at an identical 8-s cap; the separate 2--12-s cap sweep tests context length. Those comparisons are pending, so PTR is a causal access rule rather than a demonstrated gain. -\subsection{Visible-Activity Information Audit} +\subsection{Short-Window Feature Extraction} -To distinguish missing information from ineffective learned vision, we build -analysis-only features from transcript-derived \emph{past} activity. At each -cutoff, visible non-wearer participants receive two seconds of binary history -sampled on the same 5-Hz grid. The preparation fails if its aggregate -activity disagrees with the released target. We compare: (i) visibility with -activity zeroed, (ii) per-person histories, (iii) identities shuffled while -preserving active-person counts at every lag, (iv) histories delayed by one -second, and (v) one OR-pooled aggregate-visible history. These conditions -never expose future activity, but their speech labels make them privileged -diagnostics, not deployable features or upper bounds. +We define every local observation by its prediction cutoff, before applying +any encoder-specific preprocessing. The default audio branch encodes the +mixed waveform in $[t_n-1\,\mathrm{s},t_n)$ with the frozen Qwen2.5-Omni +audio encoder and averages valid output tokens. A wav2vec~2.0 convolutional +audio representation, as used by EgoSpeak \cite{kim2025egospeak}, is an +encoder-choice control; it must use the same one-second input interval and +forecasting head before differences can be attributed to the encoder. It is +not the source of the reported $\G+\A$ result. + +For vision, a single-frame variant uses the latest frame strictly before +$t_n$; the pooled person-crop variant averages frozen Qwen vision tokens +within provided boxes, across visible people, and over valid 2-FPS frames +in the preceding second. These pooled features summarize appearance but +discard person identity and explicit motion. Boxes are dataset-provided, +not predictions from an evaluated online detector. Two no-box full-frame +alternatives retain the post-merger spatial-token grid: F2 lets the +global--audio query attend to the latest causal frame; F3 first encodes up +to two observations at each matching grid location in chronological order +with a shared GRU, then applies the same query to the resulting grid. F3 +links image coordinates, not people or tracks. Its difference from F2 +confounds additional history and temporal capacity, so it cannot alone +establish a motion or speaker-grounding mechanism. Empty or unavailable +local evidence is indicated by the validity masks, never filled with a +future observation. + +As a distinct, privileged short-window feature, we also derive visible +non-wearer speech activity from transcript annotations \emph{before} each +cutoff. Each visible participant contributes a two-second binary history +on the 5-Hz grid. We test visibility-only, per-person, identity-shuffled, +one-second-delayed, and OR-pooled aggregate histories to ask which timely +activity information a learned visual branch would need to recover. +Although strictly past-only, these annotations are not available from raw +media at inference; this is an information audit, not a deployable feature +or a performance upper bound. \input{tables_compact} % Historical full-precision table layouts retained as source audit only.