Restructure local-evidence manuscript and record analysis controls
This commit is contained in:
@@ -1,5 +1,51 @@
|
||||
# Main ICASSP manuscript
|
||||
|
||||
## Current structure (2026-09-17)
|
||||
|
||||
The current draft uses **causal local evidence adaptation** as the single
|
||||
umbrella for recent modality evidence and cutoff-aligned Omni tail access.
|
||||
It supersedes the older scope description below; historical strategy docs
|
||||
remain records of their dates. The new scoped decision is
|
||||
[`../../docs/manuscript_local_evidence_structure_2026-09-17.md`](../../docs/manuscript_local_evidence_structure_2026-09-17.md).
|
||||
|
||||
- Method has two subsections: **Model Architecture** (task, global/local
|
||||
features, fusion and learning) and **Streaming Inference Architecture**
|
||||
(8-s maximum context, completed-history reuse, temporary current tail,
|
||||
cutoff alignment). Only one displayed fusion equation remains.
|
||||
- The core is frozen Qwen Omni `G` + recent 1-s Qwen audio `A`; pooled
|
||||
person vision is optional, not an established performance contribution.
|
||||
- The three contributions cover global--local adaptation, its partial-tail
|
||||
streaming realization, and controlled dense versus onset/offset analysis.
|
||||
Caching alone is not claimed as a new algorithm; mechanism gains await
|
||||
matched controls.
|
||||
- The method figure now depicts that actual core rather than an unmeasured
|
||||
25-Hz MuVAP person-attention branch. Preserved-EgoSpeak residual fusion and
|
||||
its block-phase embedding are auxiliary controls, not the default head.
|
||||
- Semantic opportunity evaluation is deferred during annotation auditing.
|
||||
Its historical protocol and workload are preserved in
|
||||
[`annotation_protocol_notes.tex`](annotation_protocol_notes.tex), which is
|
||||
deliberately not included in the submission PDF. Historical results remain
|
||||
in the original `results/` and `reports/` records.
|
||||
- Analysis uses Main Comparison, Local Evidence Analysis, Streaming and
|
||||
Context Analysis, and Error and Qualitative Analysis. Omni LoRA is deferred.
|
||||
Planned experiments are in the
|
||||
[dated experiment log](../../docs/local_evidence_analysis_experiment_log_2026-09-17.md).
|
||||
- Locked main rows and the tail-refresh x local-audio / context-cap table are
|
||||
unmeasured placeholders, not zero or mixed-protocol confirmatory numbers.
|
||||
Historical main-table screens are identified explicitly. Already inspected
|
||||
official-test runs remain exploratory despite additional seed repetitions.
|
||||
- Backbone transfer adds paired G/GA controls for Qwen2.5-Omni 3B/7B and
|
||||
MiniCPM-o 4.5, fixing the same local audio sidecar. Compact table deltas are
|
||||
pending and require readout/refresh/history audits; MiniCPM is not yet
|
||||
GPU-verified or an established matched 8-s stream. This scoped transfer
|
||||
overrides the older cache-diagnostic-only restriction below, not the 3B core.
|
||||
- The body targets four technical pages with one method figure, one main
|
||||
table, one compact streaming/transfer controls table, and one error/qualitative figure slot.
|
||||
The optional fifth page is references only; replace TODOs rather than keep
|
||||
appending, and finalize author kit and applicable AI-drafting disclosure.
|
||||
|
||||
## Build and historical scope
|
||||
|
||||
This is the editable paper directory. It uses the unmodified official ICASSP
|
||||
2026 LaTeX assets stored in `../ICASSP2026_Paper_Templates/`. When the ICASSP
|
||||
2027 author kit is released, replace or add the official template and update
|
||||
|
||||
@@ -2,6 +2,47 @@
|
||||
|
||||
This is an internal execution checklist. It is not part of the submitted PDF.
|
||||
|
||||
## Current priority override (2026-09-17)
|
||||
|
||||
Use the [local-evidence structure decision](../../docs/manuscript_local_evidence_structure_2026-09-17.md)
|
||||
for the current draft. The older semantic/person-grounded checklist below is
|
||||
a conditional extension, not authorization to describe it as implemented or
|
||||
a prerequisite for this narrower activity-anticipation study.
|
||||
|
||||
- [x] Separate model architecture from streaming inference architecture.
|
||||
- [x] Describe actual G+A fusion, optional pooled vision, and observed-tail
|
||||
recomputation; retain only one displayed method equation.
|
||||
- [x] Move the provisional LLM annotation protocol/workload outside the paper
|
||||
body without relabeling it as human ground truth.
|
||||
- [x] Reserve local-evidence, streaming/context, and combined error/qualitative
|
||||
analysis slots; record pending experiments without LoRA expansion in the
|
||||
[dated log](../../docs/local_evidence_analysis_experiment_log_2026-09-17.md).
|
||||
- [ ] Complete a training-side-frozen A/G/G+A matrix with three seeds and common
|
||||
causal-valid pairs; audit capacity and distinguish their history ranges.
|
||||
- [ ] Evaluate optional vision on/off before retaining a visual-gain claim.
|
||||
- [ ] Complete tail refresh x local-audio four-group comparison with enabled
|
||||
local audio fixed at 5 Hz; report phase strata and same-decomposition replay.
|
||||
- [ ] Re-extract/train the predeclared 4/8/16-s context-cap controls; add G at
|
||||
each cap before claiming local fusion reduces context sensitivity.
|
||||
- [ ] Add a predicted-current-activity persistence reference and capacity/recency
|
||||
controls; do not supply GT activity as model input.
|
||||
- [ ] Report scene-level activity-error counts and fixed-horizon timelines,
|
||||
including a shared failure; semantic negatives cannot override raw labels.
|
||||
- [ ] Report dense AP/mAP and conditional endpoint onset/offset AP separately,
|
||||
including denominators, per-session results, and limits of the three-session
|
||||
official test. Do not claim one-event-one-vote detection.
|
||||
- [ ] Measure full extraction latency/memory before claiming real-time FPS.
|
||||
- [ ] Run paired G/GA backbone-transfer controls for Qwen Omni 3B/7B and
|
||||
MiniCPM-o 4.5 with the same local A; report within-backbone deltas and audit
|
||||
readout, dimensions, refresh, and actual history. MiniCPM 5-Hz rows require
|
||||
GPU tail/cache/causality verification; native 1-Hz pilots need Qwen 1-Hz
|
||||
controls and cannot substitute for matched 5-Hz rows. The dated log overrides
|
||||
the older cache-diagnostic-only scope below, not its safety checks.
|
||||
- [ ] Acknowledge that already viewed Sessions 1--3 results are exploratory;
|
||||
additional seeds do not create a fresh independent test.
|
||||
- [ ] Resolve all TODOs, retain only supported mechanisms, and check the
|
||||
four-technical-page limit, author kit, and IEEE AI-drafting disclosure.
|
||||
|
||||
## Reviewer verdict and frozen paper shape
|
||||
|
||||
The potentially publishable question is: **can a general-purpose Omni readout
|
||||
|
||||
196
manuscript/main/annotation_protocol_notes.tex
Normal file
196
manuscript/main/annotation_protocol_notes.tex
Normal file
@@ -0,0 +1,196 @@
|
||||
% ARCHIVED INTERNAL NOTES: not included in the four-page submission draft.
|
||||
% Source: main.tex before the 2026-09-17 local-evidence restructure.
|
||||
% Provisional annotation workload and protocol notes, not model results.
|
||||
% Table references refer to the previous manuscript layout.
|
||||
|
||||
% Upstream 2026-09-17 merge: semantic evaluation remains deferred in main.tex.
|
||||
% Protocol-v1 opportunity AP is superseded: ENTRY cannot cross another other
|
||||
% onset/intervening wearer activity; multi-competitor INT must not use an
|
||||
% arbitrary onset. Re-run protocol v2 and audit labels before semantic claims.
|
||||
% Retained upstream semantic qualitative plan (not compiled in main.tex):
|
||||
\subsection{Qualitative Floor-Event Analysis}
|
||||
|
||||
We will compare EgoSpeak with the final matched model on human-confirmed
|
||||
examples selected by pre-declared event subtype rather than post-hoc score
|
||||
extremes. Each case will align participant-resolved activity, minimal transcript
|
||||
context, the opportunity anchor and competing-risk censor, and the two models'
|
||||
causally emitted wearer forecasts. The comparison will include a genuine wearer
|
||||
entry, an other-to-other false-entry opportunity, and an INT-yield paired with a
|
||||
non-takeover overlap; a shared failure case will be retained when space permits.
|
||||
|
||||
\begin{figure*}[t]
|
||||
\centering
|
||||
\fbox{\parbox[c][0.115\textheight][c]{0.965\textwidth}{
|
||||
\centering
|
||||
\textbf{Qualitative comparison placeholder}\\[0.7em]
|
||||
\begin{tabular}{ccc}
|
||||
Case A: wearer entry & Case B: other$\rightarrow$other HN
|
||||
& Case C: INT-yield / failed takeover\\
|
||||
participant timeline + transcript & opportunity anchor + censor
|
||||
& EgoSpeak vs. proposed forecast\\
|
||||
\end{tabular}
|
||||
}}
|
||||
\caption{Planned qualitative floor-event comparison. Final panels will use
|
||||
human-confirmed protocol-v2 cases and show the same causal source time for both
|
||||
models; the figure is a layout placeholder and contains no result claim.}
|
||||
\label{fig:qualitative_floor_events}
|
||||
\end{figure*}
|
||||
|
||||
\begin{table}[h]
|
||||
\caption{LLM pre-label workload before human adjudication. Agreement is
|
||||
between two independent passes.}
|
||||
\label{tab:behavior_annotation_stats}
|
||||
\centering
|
||||
\scriptsize
|
||||
\begin{tabular}{@{}lrr@{}}
|
||||
\hline
|
||||
Measure & EasyCom S1--4 & Ego4D AV val \\
|
||||
\hline
|
||||
Candidates & 2,128 & 3,608 \\
|
||||
AI positive & 367 & 541 \\
|
||||
AI hard negative & 1,017 & 1,174 \\
|
||||
AI excluded & 744 & 1,893 \\
|
||||
Two-pass agreement & 93.4\% & 82.1\% \\
|
||||
Human-confirmed & 0 & 0 \\
|
||||
\hline
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\appendix
|
||||
\section{LLM-Assisted Behavioral Event Annotation}
|
||||
\label{app:event_annotation}
|
||||
|
||||
EasyCom's raw speech boundaries are acoustic annotations rather than floor
|
||||
labels. We preserve them unchanged for dense forecasting and create a separate
|
||||
behavioral-event layer. Its design principle is to anchor an observed wearer
|
||||
behavior first and then retrieve the conversational opportunity that could make
|
||||
that behavior meaningful. False-positive opportunities are mined independently
|
||||
because, by definition, many contain no wearer transition.
|
||||
|
||||
\subsection{Positive-event mining}
|
||||
|
||||
\textbf{Wearer activity transitions.}
|
||||
We preserve every raw wearer $0\!\rightarrow\!1$ and $1\!\rightarrow\!0$
|
||||
boundary as a semantic review candidate. Candidate construction does not merge
|
||||
short silence gaps, infer within-turn resumption, or decide whether overlapping
|
||||
speech constitutes takeover. It only records participant-resolved temporal and
|
||||
transcript facts. The source intervals and dense targets remain unchanged.
|
||||
|
||||
\textbf{Wearer entry.}
|
||||
Every raw wearer onset at $t_b$ is proposed as one wearer-entry review item.
|
||||
We search backward within the retrieval window for non-wearer offsets and store
|
||||
each one with its response delay and factual indicators of intervening wearer
|
||||
speech, intervening other onsets, and concurrent activity. These are possible
|
||||
context anchors rather than deterministically admissible causes. The LLM and
|
||||
human reviewer decide whether the boundary is a genuine new entry, a
|
||||
within-turn resumption, a wearer backchannel, or an excluded non-event. A
|
||||
preceding release is not required for a positive entry: the wearer may enter
|
||||
after a release, from silence, or while another speaker remains active. When a
|
||||
release clearly prompts the entry, the LLM or human may select one supplied
|
||||
option; neither may invent or move a timestamp.
|
||||
|
||||
\textbf{INT-yield.}
|
||||
Every raw wearer offset $t_y$ is proposed as an INT-yield review item. The
|
||||
generator records the current wearer activity-run start, all nearby non-wearer
|
||||
onsets and offsets, activity immediately before and after $t_y$, whether an
|
||||
other speaker is pre-existing or begins during wearer speech, strict and
|
||||
50-ms-tolerant boundary crossing, overlap duration, the next speech onset,
|
||||
transcripts, and target-of-speech metadata. Relations such as
|
||||
$t_w<t_o<t_y<t_o^{\mathrm{off}}$ are descriptive evidence only: they neither
|
||||
admit nor reject a candidate and do not establish takeover. The LLM and human
|
||||
annotator answer whether the wearer stopped because a non-wearer genuinely took
|
||||
over the floor. Natural completion, post-completion activity, and within-turn
|
||||
pauses are therefore resolved semantically rather than pre-filtered.
|
||||
When a wearer activity run is itself only a backchannel within a non-wearer's
|
||||
retained turn, its onset supplies the false-entry opportunity; its trailing
|
||||
offset is excluded rather than counted again as a false-yield opportunity.
|
||||
|
||||
\subsection{False-positive opportunity mining}
|
||||
|
||||
Hard-negative regions are retrieved independently of positive wearer
|
||||
transitions. For entry, a same-speaker pause followed by resumption and an
|
||||
other-to-other handover while the wearer stays silent are false-entry
|
||||
opportunities. For INT, a short non-wearer backchannel or a non-takeover overlap
|
||||
during which the wearer retains and continues the floor is a false-yield
|
||||
opportunity. The timing rules provide candidate types, not final semantic
|
||||
labels.
|
||||
|
||||
\subsection{Semantic pre-labeling and human adjudication}
|
||||
|
||||
The annotation assistant jointly receives immutable timestamps,
|
||||
participant-resolved activity, neighboring boundaries, gap and overlap
|
||||
measurements, transcripts, and target-of-speech metadata when available. It is
|
||||
instructed not to equate raw onset with floor entry, offset with yield, or silence
|
||||
with an entry opportunity,
|
||||
or overlap with interruption, and not to invent unavailable prosody, gaze, or
|
||||
intent. EasyCom's \texttt{[L]} marker supplies laughter evidence only and is not
|
||||
by itself a sustained floor take or takeover. Preceding release options are
|
||||
optional metadata for a positive wearer-entry pre-label.
|
||||
|
||||
We use temperature-zero DeepSeek pre-labeling with a primary and a skeptical
|
||||
pass (\texttt{deepseek-flash} for the EasyCom snapshot and
|
||||
\texttt{deepseek-v4-flash} for the Ego4D AV snapshot). Disagreement on verdict,
|
||||
event type, or hard-negative subtype produces an \textsc{Exclude} suggestion.
|
||||
Every candidate then receives mandatory
|
||||
human adjudication: accept the complete pre-label and optional release metadata,
|
||||
relabel it, or exclude an
|
||||
unscorable region. Only human-confirmed \textsc{Wearer-Entry},
|
||||
\textsc{INT-Yield}, and hard-negative opportunities enter evaluation;
|
||||
\textsc{Exclude} candidates do not. Thus \textsc{Exclude} is the sole
|
||||
non-scoring verdict, but every exclusion retains a closed-vocabulary reason and
|
||||
derived aggregation group. We report valid-but-out-of-scope behavior (e.g.,
|
||||
natural wearer completion coincident with feedback), insufficient semantic
|
||||
evidence, and data-quality failures separately; model-pass disagreement belongs
|
||||
to insufficient evidence and is never counted as corrupt data.
|
||||
Annotation-only transcripts, identities, target-of-speech metadata, LLM output,
|
||||
and future observations are never supplied to the evaluated forecasting model.
|
||||
|
||||
% Snapshot source: reports/easycom_event_ai_prelabel_stats_v16/summary.json
|
||||
% and reports/ego4d_av_event_candidates_2026-09-15/ego4d_av_val_candidates_ai.jsonl.
|
||||
\begin{table}[t]
|
||||
\caption{Descriptive LLM pre-label audit before human adjudication. Counts in
|
||||
parentheses are percentages of all candidates in that column; agreement uses
|
||||
all candidates as its denominator. These values describe annotation workload,
|
||||
not ground truth or model performance.}
|
||||
\label{tab:llm_prelabel_audit}
|
||||
\centering
|
||||
\scriptsize
|
||||
\begin{tabular}{@{}lrr@{}}
|
||||
\hline
|
||||
Pre-label outcome & EasyCom S1--4 & Ego4D AV val \\
|
||||
\hline
|
||||
All candidates & 2,128 & 3,608 \\
|
||||
Positive: wearer entry & 324 (15.2\%) & 503 (13.9\%) \\
|
||||
Positive: INT-yield & 43 (2.0\%) & 38 (1.1\%) \\
|
||||
HN: entry family & 892 (41.9\%) & 940 (26.1\%) \\
|
||||
HN: INT family & 125 (5.9\%) & 234 (6.5\%) \\
|
||||
Excluded: valid/out of scope & 602 (28.3\%) & 775 (21.5\%) \\
|
||||
Excluded: insufficient evidence & 142 (6.7\%) & 1,098 (30.4\%) \\
|
||||
Excluded: data quality & 0 (0.0\%) & 20 (0.6\%) \\
|
||||
Two-pass semantic agreement & 1,987 (93.4\%)& 2,961 (82.1\%) \\
|
||||
Human-confirmed & 0 & 0 \\
|
||||
\hline
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Table~\ref{tab:llm_prelabel_audit} expands the unreviewed AI summary in
|
||||
Table~\ref{tab:behavior_annotation_stats}; neither table reports final human
|
||||
annotation results.
|
||||
The EasyCom column covers only Sessions 1--4, while the Ego4D AV column covers
|
||||
the 3,608 validation candidates; the 32,768 Ego4D AV training candidates have
|
||||
not yet received LLM pre-labels. All 5,736 reported rows remain unreviewed.
|
||||
Because the candidate sets deliberately include mined hard-negative regions,
|
||||
their class proportions are not estimates of natural event prevalence. The
|
||||
higher-level counts are descriptive only and were not used for forecasting
|
||||
threshold or model selection. The
|
||||
higher Ego4D insufficient-evidence rate (30.4\% versus 6.7\%) and lower
|
||||
two-pass agreement (82.1\% versus 93.4\%) identify a larger human-review burden;
|
||||
they do not establish a cross-dataset accuracy difference without adjudicated
|
||||
reference labels.
|
||||
|
||||
\todo{Before freezing this annotation protocol, audit randomly sampled
|
||||
continuous timeline segments without candidate hints to estimate candidate
|
||||
recall for entry, INT, and each hard-negative family. Independently double-label
|
||||
a pre-registered subset, report its size and agreement statistic, freeze all
|
||||
candidate parameters and prompt/model identifiers on Sessions 4--12, and
|
||||
report final candidate, control, exclusion, and adjudicated-label counts.}
|
||||
Binary file not shown.
@@ -1,7 +1,6 @@
|
||||
\documentclass[tikz,border=2pt]{standalone}
|
||||
\usepackage{amsmath}
|
||||
\usetikzlibrary{arrows.meta,calc,positioning}
|
||||
|
||||
\definecolor{navy}{HTML}{17324D}
|
||||
\definecolor{bluefill}{HTML}{EAF2F8}
|
||||
\definecolor{teal}{HTML}{137C8B}
|
||||
@@ -9,89 +8,81 @@
|
||||
\definecolor{orange}{HTML}{D97706}
|
||||
\definecolor{orangefill}{HTML}{FFF3E0}
|
||||
\definecolor{grayline}{HTML}{73808C}
|
||||
|
||||
\begin{document}
|
||||
\begin{tikzpicture}[
|
||||
font=\sffamily\scriptsize,
|
||||
>=Latex,
|
||||
flow/.style={-{Latex[length=2.0mm]}, line width=0.65pt, draw=navy},
|
||||
aux/.style={-{Latex[length=1.8mm]}, line width=0.6pt, draw=teal},
|
||||
box/.style={draw=navy, rounded corners=1.5pt, line width=0.65pt,
|
||||
fill=bluefill, minimum height=7mm, align=center, inner sep=2.3pt},
|
||||
localbox/.style={box, draw=teal, fill=tealfill},
|
||||
residualbox/.style={box, draw=orange, fill=orangefill},
|
||||
labelbox/.style={align=center, text=navy},
|
||||
note/.style={font=\sffamily\tiny, text=grayline, align=center}
|
||||
font=\sffamily\scriptsize, >=Latex,
|
||||
flow/.style={-{Latex[length=1.8mm]},line width=0.65pt,draw=navy},
|
||||
localflow/.style={flow,draw=teal},
|
||||
box/.style={draw=navy,rounded corners=1.5pt,line width=0.65pt,
|
||||
fill=bluefill,minimum height=8mm,align=center,inner sep=2.2pt},
|
||||
localbox/.style={box,draw=teal,fill=tealfill},
|
||||
note/.style={font=\sffamily\tiny,text=grayline,align=center}
|
||||
]
|
||||
|
||||
\node[labelbox] (av) at (0,2.65) {causal full-scene AV\\prefix $\mathcal X_{\leq t_n}$};
|
||||
\node[box, right=7mm of av, minimum width=24mm] (qwen) {frozen\\Qwen2.5-Omni};
|
||||
\node[box, right=8mm of qwen, minimum width=12mm] (g) {global\\$G_{t_n}$};
|
||||
|
||||
\node[labelbox] (audio) at (0,1.45) {mixed audio\\$\leq t_n$};
|
||||
\node[localbox, right=7mm of audio, minimum width=24mm] (cpc) {frozen causal CPC\\25-Hz processing};
|
||||
\node[localbox, right=8mm of cpc, minimum width=12mm] (u) {shared\\$U_{t_n}$};
|
||||
|
||||
\node[labelbox] (faces) at (0,0.15) {visible face crops\\$\leq t_n$};
|
||||
\node[localbox, right=7mm of faces, minimum width=24mm] (asd) {frozen causal AV/ASD\\25-Hz processing};
|
||||
\node[localbox, right=8mm of asd, minimum width=20mm] (persons) {anonymous set\\$\mathcal P_{t_n}=\{[V_i;s_i]\}$};
|
||||
|
||||
\node[box, minimum width=21mm] (base) at (8.25,2.05) {$G+U$ base head\\$Z^{\rm base}_{t_n}$};
|
||||
\node[localbox, minimum width=22mm] (attn) at (8.25,0.70) {masked person MHA\\query: $G_{t_n}+U_{t_n}$\\\scriptsize memory: $\mathcal P_{t_n}+$ NULL};
|
||||
\node[residualbox, minimum width=19mm] (res) at (10.65,0.70) {zero-init residual\\$\Delta Z_{t_n}$};
|
||||
\node[circle, draw=navy, fill=white, line width=0.7pt,
|
||||
minimum size=5.5mm, inner sep=0pt] (sum) at (11.55,2.05) {$+$};
|
||||
\node[box, minimum width=21mm] (out) at (13.15,2.05) {$10\!\times\!3$ future activity\\$0.2$--$2.0$ s};
|
||||
|
||||
\draw[flow] (av) -- (qwen);
|
||||
\draw[flow] (qwen) -- (g);
|
||||
\draw[aux] (audio) -- (cpc);
|
||||
\draw[aux] (cpc) -- (u);
|
||||
\draw[aux] (faces) -- (asd);
|
||||
\draw[aux] (asd) -- (persons);
|
||||
\draw[flow] (g.east) -| ($(base.west)+(-0.25,0.12)$) -- (base.west);
|
||||
\draw[aux] (u.east) -| ($(base.west)+(-0.25,-0.12)$) -- (base.west);
|
||||
\draw[flow] (g.east) -- ++(0.3,0) |- (attn.north west);
|
||||
\draw[aux] (u.east) -- ++(0.5,0) |- (attn.west);
|
||||
\draw[aux] (persons.east) -- (attn.west);
|
||||
\draw[flow] (base) -- (sum);
|
||||
\draw[aux] (attn) -- (res);
|
||||
\draw[flow, draw=orange] (res.north) |- (sum.south);
|
||||
\draw[flow] (sum) -- (out);
|
||||
|
||||
\node[note, below=1.5mm of base] {trained first, then frozen};
|
||||
\node[note, below=1.2mm of res] {trainable adaptation};
|
||||
|
||||
\draw[grayline, line width=0.45pt] (-0.15,-0.62) -- (14.35,-0.62);
|
||||
\node[anchor=west, font=\sffamily\scriptsize\bfseries, text=navy]
|
||||
at (-0.05,-0.92) {Causal streaming at cutoff $t_n$};
|
||||
|
||||
\node[anchor=east, text=navy] at (2.25,-1.45) {Omni input};
|
||||
\draw[fill=bluefill, draw=navy, line width=0.55pt] (2.45,-1.67) rectangle (4.35,-1.23);
|
||||
\draw[fill=bluefill, draw=navy, line width=0.55pt] (4.35,-1.67) rectangle (6.25,-1.23);
|
||||
\node[text=navy] at (3.40,-1.45) {cached 2-s blocks};
|
||||
\node[text=navy] at (5.30,-1.45) {$\cdots$};
|
||||
\draw[fill=orangefill, draw=orange, line width=0.65pt] (6.25,-1.67) rectangle (9.15,-1.23);
|
||||
\node[text=orange] at (7.70,-1.45) {current partial block};
|
||||
\draw[dashed, draw=orange, line width=0.8pt] (9.15,-2.90) -- (9.15,-1.02);
|
||||
\node[anchor=south, text=orange, font=\sffamily\tiny\bfseries] at (9.15,-1.02) {$t_n$};
|
||||
\node[note, anchor=west] at (9.40,-1.45) {latest observed input included\\up to 8-s rolling budget};
|
||||
|
||||
\node[anchor=east, text=teal] at (2.25,-2.15) {local encoder};
|
||||
\foreach \x in {2.55,2.85,...,8.85} {
|
||||
\draw[draw=teal, line width=0.55pt] (\x,-2.33) -- (\x,-1.97);
|
||||
\node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy]
|
||||
at (-0.15,3.15) {Model: global context + recent evidence};
|
||||
\node[align=center,text=navy] (av) at (0.65,2.45)
|
||||
{full-scene AV\\observed prefix};
|
||||
\node[box] (omni) at (3.05,2.45) {frozen Qwen2.5-Omni\\8-s maximum context};
|
||||
\node[box] (global) at (5.65,2.45) {last-four-layer\\projection $G_n$};
|
||||
\node[align=center,text=teal] (audio) at (0.65,1.28)
|
||||
{recent 1-s\\mixed audio};
|
||||
\node[localbox] (ae) at (3.05,1.28) {frozen Qwen audio\\valid-token mean};
|
||||
\node[localbox] (ap) at (5.65,1.28) {local projection\\$\widetilde A_n$ + validity};
|
||||
\node[align=center,text=teal] (video) at (0.65,0.08)
|
||||
{recent 1-s\\person crops};
|
||||
\node[localbox,dashed] (ve) at (3.05,0.08)
|
||||
{frozen Qwen visual\\recent crop pooling};
|
||||
\node[localbox,dashed] (vp) at (5.65,0.08)
|
||||
{optional projection\\$\widetilde V_n$ + validity};
|
||||
\node[box,minimum width=20mm] (fusion) at (8.35,1.28)
|
||||
{feature concatenation\\trainable MLP};
|
||||
\node[box,minimum width=23mm] (out) at (11.15,1.28)
|
||||
{$10\times3$ future activity\\0.2--2.0 s};
|
||||
\draw[flow] (av) -- (omni);
|
||||
\draw[flow] (omni) -- (global);
|
||||
\draw[localflow] (audio) -- (ae);
|
||||
\draw[localflow] (ae) -- (ap);
|
||||
\draw[localflow,dashed] (video) -- (ve);
|
||||
\draw[localflow,dashed] (ve) -- (vp);
|
||||
\draw[flow] (global.east) -- ++(0.35,0) |- (fusion.north);
|
||||
\draw[localflow] (ap) -- (fusion);
|
||||
\draw[localflow,dashed] (vp.east) -- ++(0.35,0) |- (fusion.south);
|
||||
\draw[flow] (fusion) -- (out);
|
||||
\node[note] at (10.95,0.22)
|
||||
{background / wearer / aggregate others\\no semantic event or identity output};
|
||||
\draw[grayline,line width=0.45pt] (-0.15,-0.65) -- (12.5,-0.65);
|
||||
\node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy]
|
||||
at (-0.15,-0.98) {Stream: committed history + temporary current tail};
|
||||
\node[anchor=east,text=navy] at (2.05,-1.65) {Omni media};
|
||||
\foreach \a/\b/\lab in {2.3/3.9/2 s,3.9/5.5/2 s,5.5/7.1/2 s} {
|
||||
\draw[fill=bluefill,draw=navy,line width=0.55pt]
|
||||
(\a,-1.87) rectangle (\b,-1.43);
|
||||
\node[text=navy] at ({(\a+\b)/2},-1.65) {\lab};
|
||||
}
|
||||
\draw[draw=teal, line width=0.55pt] (2.55,-2.15) -- (9.15,-2.15);
|
||||
\node[note, anchor=west, text=teal] at (9.40,-2.15) {25 Hz internal states};
|
||||
|
||||
\node[anchor=east, text=navy] at (2.25,-2.72) {forecast grid};
|
||||
\draw[draw=navy, line width=0.65pt] (2.55,-2.72) -- (9.15,-2.72);
|
||||
\foreach \x/\lab in {3.15/0.2,4.65/0.4,6.15/0.6,7.65/0.8,9.15/$t_n$} {
|
||||
\draw[draw=navy, line width=0.75pt] (\x,-2.92) -- (\x,-2.52);
|
||||
\node[below=0.5mm, font=\sffamily\tiny, text=navy] at (\x,-2.92) {\lab};
|
||||
\draw[-{Latex[length=1.4mm]}, draw=teal, line width=0.55pt] (\x,-2.32) -- (\x,-2.50);
|
||||
\draw[fill=orangefill,draw=orange,line width=0.65pt]
|
||||
(7.1,-1.87) rectangle (8.5,-1.43);
|
||||
\node[text=orange,font=\sffamily\tiny] at (7.8,-1.65) {partial tail};
|
||||
\draw[fill=white,draw=grayline,dashed,line width=0.55pt]
|
||||
(8.5,-1.87) rectangle (8.7,-1.43);
|
||||
\node[note,anchor=west] at (9.0,-1.65)
|
||||
{three completed blocks + observed tail\\8-s maximum; future portion excluded};
|
||||
\node[note,text=navy] at (4.7,-2.13) {reuse completed media payloads};
|
||||
\node[note,text=orange] at (7.8,-2.13) {recompute at each cutoff};
|
||||
\node[anchor=east,text=teal] at (2.05,-2.62) {local evidence};
|
||||
\draw[draw=teal,line width=1.8pt] (7.7,-2.62) -- (8.5,-2.62);
|
||||
\node[note,anchor=west,text=teal] at (9.0,-2.62)
|
||||
{recent 1-s audio / optional vision\\refreshed on the common 5-Hz grid};
|
||||
\node[anchor=east,text=navy] at (2.05,-3.18) {decision grid};
|
||||
\draw[draw=navy,line width=0.6pt] (2.3,-3.18) -- (8.5,-3.18);
|
||||
\foreach \x in {2.42,2.58,...,8.34} {
|
||||
\draw[draw=navy,line width=0.5pt] (\x,-3.28) -- (\x,-3.08);
|
||||
}
|
||||
\node[note, anchor=west, text=navy] at (9.40,-2.72) {5 Hz fusion and prediction\\every fifth local right edge};
|
||||
|
||||
\draw[draw=navy,line width=0.9pt] (8.5,-3.3) -- (8.5,-3.06);
|
||||
\draw[dashed,draw=orange,line width=0.75pt]
|
||||
(8.5,-3.35) -- (8.5,-1.15);
|
||||
\node[anchor=south,text=orange,font=\sffamily\tiny\bfseries]
|
||||
at (8.5,-1.15) {cutoff $t_n$};
|
||||
\node[note,anchor=west,text=navy] at (9.0,-3.18)
|
||||
{one prediction every 200 ms\\same cutoff and target index};
|
||||
\end{tikzpicture}
|
||||
\end{document}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user