From e0cb06198a3fca3bf456dc4e9d19f76b7940d323 Mon Sep 17 00:00:00 2001 From: Xin Wang Date: Thu, 24 Sep 2026 18:52:10 +0800 Subject: [PATCH] Streamline targeted methods and experiment presentation --- AGENTS.md | 23 +++-- manuscript/targeted/main.tex | 114 +++++++++---------------- manuscript/targeted/tables_compact.tex | 4 +- 3 files changed, 55 insertions(+), 86 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 988a66d..0eba5bc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -50,7 +50,7 @@ Introduction 的故事或改换核心研究问题；可继续核实文献、修 泛称所有 Omni 模型都缺乏局部感知。 4. **本文观点与三项贡献。** 先提出保留 Omni 全局状态、同时在每个因果截止点显式接入 近期模态证据的观点，再以冒号引出三项贡献：(1) SWE（short-window encoder - evidence，短窗编码器证据）将近期音频与可选的因果人物框视觉特征接入冻结 Omni + evidence，短窗编码器证据）将近期音频与可选的因果视觉编码特征接入冻结 Omni 预测头；“短窗”指时间局部，不是浅层网络或任务级端到端训练。(2) PTR（partial-tail refresh，部分尾块刷新）在复用已完成块时只查询未完成块已观测前缀，不把它提交到 持久历史；其收益须由相同 cutoff、相同上下文上限的配对实验验证。(3) EasyCom @@ -70,7 +70,7 @@ Method 按以下三个小节展开，除非用户明确要求，不再改变这 1. **Task and Global--Local Fusion。** 定义 5 Hz 因果截止点、十个未来 horizon 和 background/wearer/others 输出；说明 Thinker 全局状态本身已含音视频，再定义 SWE - 的近期音频及可选视觉旁路。区分 pooled 特征的拼接聚合与 F2/F3 保留视觉 token 的 + 的近期音频及可选视觉旁路。区分 pooled 特征的拼接聚合与 SVA/TSVA 保留视觉 token 的 query attention。训练范围分开写：主方案只训练 readout/局部分支/预测头；Thinker LoRA、限制在上层的 LoRA、预测头冻结或联合训练是参数适配控制。最后四层 **readout** 与 LoRA 更新哪些层是两个独立维度。已归档 LoRA 结果为非匹配探索， @@ -78,13 +78,18 @@ Method 按以下三个小节展开，除非用户明确要求，不再改变这 2. **Causal Partial-Tail Refresh。** 完成块进入持久历史；每个 200 ms 截止点只临时查询 当前 2 秒块已观测的前缀，尾块不提交、不中途引入未来媒体。与 completed-only 在同一截止点和上下文上限下配对比较；未完成实验前只称因果访问规则。 -3. **Short-Window Feature Extraction。** 默认局部音频为截止点前一秒混合波形经冻结 - Qwen 音频编码器；EgoSpeak 式 wav2vec2 卷积编码是待做的同窗口、同预测头对照， - 不能冒充已测 G+A。视觉区分最近单帧、最近一秒多帧人物框 pooled 特征、无框的 - F2 最新全画面空间 token，以及 F3 同网格位置两帧 token 经共享 GRU 后注意力聚合。 - 另列一种特殊的短窗信息审计：由转写标注生成、仅含截止点之前的可见非 wearer - 活动历史，比较逐人、打乱、延迟和 OR 聚合等形式。它不是部署时可获得的视觉特征， - 也不是性能上界。F3 不是人物跟踪，F3--F2 差异也不能单独归因于运动理解。 +3. **Short-Window Feature Extraction。** 默认局部音频 `A` 为截止点前一秒混合波形经冻结 + Qwen 音频编码器；EgoSpeak 式 wav2vec2 卷积编码记为 `A_E`，是待做的同窗口、同预测头对照， + 不能冒充已测 G+A。视觉区分最近一秒多帧人物框池化的 Pool、无框的 + SVA（spatial visual attention）最新单帧全画面空间 token，以及 TSVA（temporal + spatial visual attention）同网格位置两帧 token 经共享 GRU 后注意力聚合。 + TSVA 不是人物跟踪，TSVA--SVA 差异也不能单独归因于运动理解。由转写标注生成的 + 过去可见人活动历史只在 Results 的信息审计中定义和分析，不放进 Method 2.3； + 它不是部署时可获得的视觉特征，也不是性能上界。 + +主结果表固定为 EasyCom 与 Ego4D 各四行：EgoSpeak A+V、Omni G、Omni G+A、 +Omni G+A+SVA，并保留十个 0.2--2.0 秒 horizon 与均值。尚未取得的 SVA/Ego4D +分数用 `--` 占位，不插值、不把计划写成结果；pooled-V 与 TSVA 只进入视觉分析表。 ## 不可破坏的研究契约 diff --git a/manuscript/targeted/main.tex b/manuscript/targeted/main.tex index 248f39b..af53744 100644 --- a/manuscript/targeted/main.tex +++ b/manuscript/targeted/main.tex @@ -211,49 +211,28 @@ pending, so PTR is a causal access rule rather than a demonstrated gain. \subsection{Short-Window Feature Extraction} -We define every local observation by its prediction cutoff, before applying -any encoder-specific preprocessing. We denote the default Qwen-audio branch -by $\A$: it encodes the -mixed waveform in $[t_n-1\,\mathrm{s},t_n)$ with the frozen Qwen2.5-Omni -audio encoder and averages valid output tokens. We denote by $\A_{\rm E}$ -the wav2vec~2.0 convolutional representation used by EgoSpeak -\cite{kim2025egospeak}. This encoder-choice control must use the same -one-second input interval and -forecasting head before differences can be attributed to the encoder. It is -not the source of the reported $\G+\A$ result. +Each local branch observes only media preceding the prediction cutoff +$t_n$; unavailable observations retain explicit validity masks rather +than borrowing later samples. All encoders are frozen. The notation below +distinguishes encoder choice from fusion choice: $\G$ is the global Thinker +state, and local branches are appended to the same forecasting head. -We denote three visual branches by \emph{Pool}, \emph{SVA}, and \emph{TSVA}. -Pool averages frozen Qwen vision tokens -within provided boxes, across visible people, and over valid 2-FPS frames -in the preceding second. These pooled features summarize appearance but -discard person identity and explicit motion. Boxes are dataset-provided, -not predictions from an evaluated online detector. Two no-box full-frame -alternatives retain the post-merger spatial-token grid. Spatial visual -attention (SVA) lets the global--audio query attend to the latest frame -strictly before $t_n$; temporal spatial visual attention (TSVA) first encodes up -to two observations at each matching grid location in chronological order -with a shared GRU, then applies the same query to the resulting grid. TSVA -links image coordinates, not people or tracks. Its difference from SVA -confounds additional history and temporal capacity, so it cannot alone -establish a motion or speaker-grounding mechanism. Empty or unavailable -local evidence is indicated by the validity masks, never filled with a -future observation. +The default audio feature $\A$ encodes the mixed waveform in +$[t_n-1\,\mathrm{s},t_n)$ with the Qwen2.5-Omni audio encoder and averages +valid tokens. $\A_{\rm E}$ denotes EgoSpeak's wav2vec~2.0 convolutional +representation \cite{kim2025egospeak}; comparing $\G+\A$ with +$\G+\A_{\rm E}$ requires the same window and head. Released EgoSpeak A+V +remains an external baseline, not this encoder substitution. -The shorthand $\G$, $\G+\A$, and $\G+\A+$SVA identifies the three Omni -rows in the cross-dataset table; $\G+\A+$Pool, $\G+\A+$TSVA, and -$\G+\A_{\rm E}$ isolate visual or audio choices. Released EgoSpeak A+V -is an external baseline, not an encoder choice within our frozen-Omni head. -An unmeasured label specifies a comparison, not a result. - -As a distinct, privileged short-window feature, we also derive visible -non-wearer speech activity from transcript annotations \emph{before} each -cutoff. Each visible participant contributes a two-second binary history -on the 5-Hz grid. We test visibility-only, per-person, identity-shuffled, -one-second-delayed, and OR-pooled aggregate histories to ask which timely -activity information a learned visual branch would need to recover. -Although strictly past-only, these annotations are not available from raw -media at inference; this is an information audit, not a deployable feature -or a performance upper bound. +For vision, Pool averages Qwen tokens within provided person boxes and +across valid 2-FPS frames in the preceding second; the boxes are not +outputs of an evaluated online detector. Spatial visual attention (SVA) +instead lets $\G+\A$ query the unpooled spatial tokens of the latest +strictly pre-cutoff full frame. Temporal spatial visual attention (TSVA) +first applies a shared GRU to up to two causal tokens at each matching +grid position, then uses the same query. TSVA links image coordinates, +not person tracks. We denote these variants by $\G+\A+$Pool, +$\G+\A+$SVA, and $\G+\A+$TSVA. \input{tables_compact} % Historical full-precision table layouts retained as source audit only. @@ -405,23 +384,19 @@ Full path & -- & -- & -- \\ \section{Experiments} \label{sec:experiments} -\textbf{Data and protocol.} -We use EasyCom with the EgoSpeak official split: Sessions~4--12 for training -and Sessions~1--3 for testing. Inputs and targets share a 5-Hz grid. Heads -are trained for four fixed epochs on all training sessions without test-time -checkpoint selection. The deployable feature matrix is an archived -single-seed screen; the information audit uses seeds 0, 1, and 2 with an -identical fixed recipe. We therefore report standard deviations only for the -latter and treat small differences in the former descriptively. - -\textbf{Ego4D transfer protocol (pending results).} -Ego4D audiovisual clips \cite{grauman2022ego4d} use the existing EgoSpeak -manifest and targets, with 346 training and 87 test clips. The same 5-Hz -cutoffs and local 1-s encoder windows define $\G$, $\G+\A$, $\G+\V$, and -$\G+\A+\V$ comparisons; visual validity is retained rather than silently -dropping no-box rows. The repository contains the extraction and training -workflow but no archived Ego4D score or matched PTR control. Hence this -protocol is specified, not reported as an observed cross-dataset gain. +\textbf{Data Preparation.} +We use the EgoSpeak EasyCom split (Sessions~4--12 for training and +Sessions~1--3 for testing) and its Ego4D audiovisual manifest +\cite{donley2021easycom,grauman2022ego4d}, which contains 346 training and +87 test clips. Both datasets use 5-Hz causal cutoffs and targets, the same +one-second local-audio window, and visual samples strictly preceding each +cutoff; unavailable vision is masked rather than silently dropping rows. +EasyCom heads use four fixed training epochs without test-set checkpoint +selection. Its deployable feature matrix is an archived single-seed screen, +whereas the privileged information audit uses three seeds under one recipe; +small single-seed differences are descriptive. Ego4D extraction and training +workflows exist, but test scores and a matched PTR comparison have not yet +been archived, so cross-dataset gains are not claimed. \textbf{Metrics.} At each horizon we compute AP for the three activity channels over common @@ -433,22 +408,6 @@ speech and scores future silence with $1-p(\mathrm{wearer})$. These threshold-free quantities average ten horizons. They are not one-event-one- vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding. -\textbf{Comparisons.} -The cross-dataset main matrix contains released EgoSpeak A+V and the three -Omni variants defined in Sec.~\ref{sec:method}. Missing SVA and Ego4D -cells remain blank until measured. EasyCom-only encoder choices serve as -diagnostic ablations rather than extra cross-dataset main rows. A three-seed -current-frame person-set attention -screen used Session~12 model selection whereas the historical pooled row did -not; their difference is not a matched causal attribution. - -\textbf{Pre-specified analyses.} -We freeze six control families: (L) capacity and recency, (T) tail -refresh crossed with local audio, (C) six context caps, (B) paired backbone -transfer, (H/Q) error slices and timelines, and (S) full-path efficiency. -They share the official split and physical cutoffs; the 8-s T/C row reuses -one run. The source retains unmeasured hypotheses and falsifiers. - \section{Results and Analysis} \label{sec:results} @@ -549,7 +508,12 @@ shared GRU, while head capacity and training batch differ; raw boundary metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point difference does not isolate visual motion or floor behavior. -Table~\ref{tab:oracle} locates the missing information. Aggregate visible +The analysis-only audit in Table~\ref{tab:oracle} substitutes two seconds +of transcript-derived, strictly past visible-person activity on the 5-Hz +grid. It tests visibility alone, per-person histories, shuffled identities, +a one-second delay, and OR-pooled activity. These privileged labels are +neither deployable visual features nor a performance upper bound. +Aggregate visible activity improves over the audio baseline by 4.826 points at 0.2~s and 1.775 mean-mAP points. Visibility alone changes mean mAP by only +0.040, and one-second-delayed activity by +0.164. Thus availability or extra parameters diff --git a/manuscript/targeted/tables_compact.tex b/manuscript/targeted/tables_compact.tex index 416a9fb..ac3a3d9 100644 --- a/manuscript/targeted/tables_compact.tex +++ b/manuscript/targeted/tables_compact.tex @@ -15,13 +15,13 @@ rows are exploratory and not all training-matched.} \cmidrule(lr){2-11} Dataset / model & .2 & .4 & .6 & .8 & 1.0 & 1.2 & 1.4 & 1.6 & 1.8 & 2.0 & Mean \\ \midrule -\multicolumn{12}{@{}l}{\emph{EasyCom (measured)}} \\ +\multicolumn{12}{@{}l}{\emph{EasyCom}} \\ EgoSpeak A+V & 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82 & 61.19 \\ Omni $\G$ & 73.45 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 60.85 \\ Omni $\G+\A$ & 81.72 & 75.92 & 70.38 & 66.46 & 63.81 & 61.37 & 60.00 & 58.32 & 56.95 & 56.16 & 65.11 \\ Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ \midrule -\multicolumn{12}{@{}l}{\emph{Ego4D (transfer pending)}} \\ +\multicolumn{12}{@{}l}{\emph{Ego4D}} \\ EgoSpeak A+V & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ Omni $\G$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ Omni $\G+\A$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\