diff --git a/manuscript/main/README.md b/manuscript/main/README.md index 3f2e3d1..b5ec23f 100644 --- a/manuscript/main/README.md +++ b/manuscript/main/README.md @@ -47,7 +47,10 @@ Its single candidate contribution is a causal local floor-evidence adapter: - the primary local method uses an anonymous, no-ID visible-person set; - track continuity is conditional and must defeat no-ID/shuffled/capacity controls before entering the paper; -- MiniCPM-o and a cross-family sweep are removed from the core plan; +- MiniCPM-o remains outside the core method and main comparison; it appears + only as a scoped streaming/cache diagnostic with a matched Qwen 1-Hz + causal-hold control, and its 5-Hz cache row requires equivalence and leakage + audits before reporting; - the oracle is analysis-only motivation, not a deployable method or headline; - the publication gate is lower competing-risk-censored multi-other false entry at matched wearer-onset recall without worse valid onset or yield. diff --git a/manuscript/main/REVIEWER_CHECKLIST.md b/manuscript/main/REVIEWER_CHECKLIST.md index fdbd93d..a72c695 100644 --- a/manuscript/main/REVIEWER_CHECKLIST.md +++ b/manuscript/main/REVIEWER_CHECKLIST.md @@ -17,7 +17,9 @@ behavior-aligned multi-other event protocol. Current Qwen2.5-Omni-3B results are exploratory motivation only. They neither prove a limitation shared by Omni models nor establish track-consistent fusion as the final method. Qwen2.5-Omni-3B is primary; Qwen2.5-Omni-7B is an optional -capacity control. MiniCPM-o and a cross-family sweep are outside the core paper. +capacity control. MiniCPM-o remains outside the core method and main comparison; +it is permitted only as a diagnostic streaming/cache ablation with a matched +Qwen refresh-rate control. The four-page paper has only three contributions: temporal--role adaptation mismatch, one causal local floor adapter, and causal behavioral evidence. @@ -63,6 +65,26 @@ a model cannot expose a comparable causal state, report that protocol as unsupported. Raw scores are not a pure architecture ranking; emphasize within-backbone deltas. +For the diagnostic streaming/cache ablation: + +- [ ] compare Qwen full causal-prefix recomputation and cache reuse at the + same 5-Hz cutoffs, recording numerical agreement, throughput, and memory; +- [ ] map the native MiniCPM-o one-second output to 5-Hz rows by causal + forward-hold only, never by backfilling the state into earlier rows; +- [ ] run a Qwen 1-Hz update-and-hold control with the same physical chunk + boundaries and report the within-Qwen 5-Hz versus 1-Hz delta; +- [ ] freeze initialization before the first completed chunk and reset every + modality/language-model cache at session and source-gap boundaries; +- [ ] before admitting a MiniCPM-o 5-Hz cached-branch row, compare it against + full causal-prefix recomputation at sampled cutoffs under a frozen cosine and + maximum-absolute-error tolerance; +- [ ] run future-media intervention tests at every tested cutoff and record + throughput, peak memory, feature dimensionality, and exact code revision. + +If the 5-Hz cached branch fails equivalence or leakage tests, omit that row and +report only the native 1-Hz causal-hold diagnostic. Do not present unmatched +Qwen--MiniCPM raw scores as a pure backbone comparison. + ## P2 — Establish the compact adaptation matrix Run three seeds for the primary backbone with the same data, physical cutoffs, @@ -144,17 +166,23 @@ Examples explain rates; they never replace them. 2. Audit the Qwen2.5-Omni-3B readout and matched PEFT baseline. 3. Complete the frozen/PEFT/audio/matched-temporal/local-AV matrix. 4. Run correct/time-shift/remove-vision controls and the optional 7B scaling row. -5. Consider track continuity only if the no-ID method already passes. -6. Produce one confirmatory event table and one mechanism-ablation table. -7. Draw one compact final system figure; remove rejected methods from the main +5. Run the Qwen refresh-rate control and the scoped MiniCPM-o streaming/cache + diagnostic; admit the 5-Hz MiniCPM-o row only after equivalence and leakage + audits. +6. Consider track continuity only if the no-ID method already passes. +7. Produce one confirmatory event table and one mechanism-ablation table. +8. Draw one compact final system figure; remove rejected methods from the main narrative. -8. Replace every visible TODO and verify the ICASSP 2027 kit/page limit. +9. Replace every visible TODO and verify the ICASSP 2027 kit/page limit. ## Stop/go decisions - **Claim boundary:** with Qwen-only evidence, describe a Qwen2.5-Omni case study and the tested adaptation mechanism; do not generalize to all Omni models. +- **Streaming boundary:** call native MiniCPM-o 1-Hz plus causal hold a held-state + baseline, not native 5-Hz inference; a 5-Hz cache result exists only after + cached/full-prefix equivalence and future-input invariance pass. - **Audio-local reframe:** local audio improves but all visual variants fail; claim a fine-grained acoustic readout bottleneck, not social visual reasoning. - **Track-aware method:** correct continuity beats pooled vision, temporal diff --git a/manuscript/main/main.tex b/manuscript/main/main.tex index 8bbf4e2..cc6cff5 100644 --- a/manuscript/main/main.tex +++ b/manuscript/main/main.tex @@ -157,8 +157,9 @@ is not interpreted as a general limitation of LoRA or all Omni models. \todo{Freeze the exact Qwen2.5-Omni-3B revision, processor, input rates, streaming/cache semantics, readout layer, context, dtype, LoRA targets/rank, trainable parameter budget, and cutoff audit. Add Qwen2.5-Omni-7B only as a -capacity-scaling control. Do not add another backbone family unless it is -needed to support and experimentally earns a cross-family claim.} +capacity-scaling control. MiniCPM-o 4.5 is restricted to the diagnostic +streaming/cache ablation below and does not enter the core method or support a +cross-family superiority claim.} \subsection{Causal local floor adapter} @@ -267,6 +268,17 @@ control. All learned variants use identical causal examples, prediction horizons, prediction heads, and the class-balanced binary cross-entropy objective in Eq.~\eqref{eq:loss}. +The diagnostic streaming ablation additionally uses MiniCPM-o 4.5 +\cite{cui2026minicpmo45}. Its native cached stream is evaluated at its native +one-second update interval: a state becomes available to the 200-ms row whose +cutoff equals the chunk end and is then causally held until the next update. A +Qwen row subjected to the same 1-Hz update-and-hold schedule separates +refresh-rate effects from backbone effects. A MiniCPM-o +5-Hz cache row is admissible only if a cached branch at every cutoff agrees +with full causal-prefix recomputation within a frozen numerical tolerance and +passes future-input intervention tests. Thus the native 1-Hz condition is a +strictly causal held-state baseline, not a claim of native 5-Hz refresh. + Model selection is performed using training-side data only. Event operating points are selected from session-level out-of-fold predictions over Sessions~4--12; the final model is then retrained on all nine sessions before @@ -392,27 +404,40 @@ numbers from the four-page version.} \subsection{Mechanism Analysis} To determine whether the gain arises from correctly aligned visual floor -evidence rather than additional parameters or generic visual context, we -evaluate the controls in Table~\ref{tab:ablation}. +evidence rather than additional parameters or generic visual context, and to +separate streaming refresh from representation family, we evaluate the +controls in Table~\ref{tab:ablation}. \begin{table}[t] -\caption{Mechanism controls for the selected Omni-3B local adapter. All -variants use the same prediction head and comparable temporal capacity; FPRs -are measured at matched recall $R=0.8$.} +\caption{Mechanism and streaming/cache controls. The streaming panel uses +frozen global states and capacity-matched downstream heads; all rows share +targets and evaluation. FPRs are measured at matched recall $R=0.8$. +Cross-family raw scores are diagnostic.} \label{tab:ablation} \centering \small \resizebox{\columnwidth}{!}{% \begin{tabular}{lcccc} \hline -Local condition & Mean mAP $\uparrow$ & EOT FPR $\downarrow$ +Condition & Mean mAP $\uparrow$ & EOT FPR $\downarrow$ & O$\rightarrow$O FPR $\downarrow$ & INT FPR $\downarrow$\\ \hline +\multicolumn{5}{l}{\emph{Local-evidence controls (Qwen2.5-Omni-3B)}}\\ Remove vision & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ Time-shift vision & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ Full-frame vision & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ Correct local AV & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ \hline +\multicolumn{5}{l}{\emph{Streaming/cache controls}}\\ +Qwen, full causal prefix at 5 Hz (no cache) + & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +Qwen, cached 5-Hz update & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +Qwen, cached 1-Hz + causal hold & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +MiniCPM-o, native cached 1-Hz + causal hold + & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +MiniCPM-o, audited cached 5-Hz branch + & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +\hline \end{tabular}} \end{table} @@ -430,7 +455,7 @@ evidence-routing claim. If dense mAP improves but event behavior does not, do not claim improved floor control. If legitimate onset or yield degrades, do not describe lower false entry as success.} -\subsection{Scaling Control} +\subsection{Scaling and Streaming Controls} Increasing the frozen Omni backbone from 3B to 7B yields \todo{$X$ points} in 0.2-s mAP and \todo{EOT/INT FPR of $Y/Z$} at matched @@ -443,6 +468,25 @@ provides a complementary benefit to backbone scaling}. Because both backbones belong to the Qwen2.5-Omni family, we do not claim that this conclusion generalizes to all omni-modal models. +The streaming rows answer two different questions. First, full causal-prefix +recomputation and streaming cache reuse at the same 5-Hz Qwen cutoffs test +whether cache/chunk semantics change the representation; equivalent outputs +make this an efficiency control rather than a distinct predictive method. +Second, the matched Qwen 5-Hz and 1-Hz cached rows test whether an advantage is +explained by more frequent causal state refresh rather than the proposed local +evidence route. For native 1-Hz streams, the state emitted at the end of each +one-second chunk is never backfilled into earlier 200-ms rows; source gaps and +session boundaries reset all modality and language-model caches. MiniCPM-o is +a secondary cross-family diagnostic under the same 1-Hz schedule, not a +same-frequency architecture ranking. Its 5-Hz cached-branch row is omitted if +cached/full-prefix equivalence, cutoff invariance, or reset audits fail. + +\todo{Freeze native chunk boundaries, initialization of rows before the first +complete chunk, cache-reset rules, the cached/full-prefix tolerance, and the +future-media intervention test. Report within-backbone 5-Hz versus 1-Hz +deltas plus cache/no-cache throughput and memory; do not interpret unmatched +Qwen--MiniCPM raw scores as a pure backbone effect.} + \subsection{Limitations and Qualitative Analysis} The results support \todo{precise supported claim}, but several limitations