diff --git a/manuscript/main/Makefile b/manuscript/main/Makefile index 4695cef..e671c8d 100644 --- a/manuscript/main/Makefile +++ b/manuscript/main/Makefile @@ -1,8 +1,4 @@ LATEXMK ?= latexmk -TEMPLATE_DIR := $(abspath ../ICASSP2026_Paper_Templates) - -export TEXINPUTS := $(TEMPLATE_DIR): -export BSTINPUTS := $(TEMPLATE_DIR): .PHONY: all clean diff --git a/manuscript/main/README.md b/manuscript/main/README.md index 54f3cae..3f2e3d1 100644 --- a/manuscript/main/README.md +++ b/manuscript/main/README.md @@ -3,7 +3,7 @@ This is the editable paper directory. It uses the unmodified official ICASSP 2026 LaTeX assets stored in `../ICASSP2026_Paper_Templates/`. When the ICASSP 2027 author kit is released, replace or add the official template and update -`TEMPLATE_DIR` in the Makefile only after checking the new instructions. +`template_dir` in `latexmkrc` only after checking the new instructions. Compile from this directory: @@ -14,12 +14,19 @@ make The PDF and all intermediate files are written to `build/`; only source files belong in Git. +VS Code's LaTeX Workshop extension is supported when the repository is opened +as the workspace. Open `main.tex` and run **LaTeX Workshop: Build LaTeX +project**. The workspace setting writes output to `build/`, while `latexmkrc` +provides the template and bibliography-style search paths to both LaTeX +Workshop and command-line builds. + Current source layout: ```text main/ main.tex references.bib + latexmkrc Makefile README.md build/ # generated, ignored diff --git a/manuscript/main/latexmkrc b/manuscript/main/latexmkrc new file mode 100644 index 0000000..860d59d --- /dev/null +++ b/manuscript/main/latexmkrc @@ -0,0 +1,9 @@ +my $path_separator = $^O eq 'MSWin32' ? ';' : ':'; +my $template_dir = '../ICASSP2026_Paper_Templates'; + +$ENV{'TEXINPUTS'} = + $template_dir . $path_separator . ($ENV{'TEXINPUTS'} // ''); +$ENV{'BSTINPUTS'} = + $template_dir . $path_separator . ($ENV{'BSTINPUTS'} // ''); + +$out_dir = 'build'; diff --git a/manuscript/main/main.tex b/manuscript/main/main.tex index c14a86a..0480a6e 100644 --- a/manuscript/main/main.tex +++ b/manuscript/main/main.tex @@ -169,7 +169,7 @@ $\mathcal P_\tau=\{(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau}$, a shared visual encoder and masked set operator produce \begin{equation} \vect v_\tau=\mathrm{SetPool} - \left(\{E_v(\vect x_{\tau,i},\vect b_{\tau,i})}_{i=1}^{N_\tau} + \left(\{E_v(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau} \cup\{\vect v_\varnothing\}\right), \label{eq:set} \end{equation} @@ -216,19 +216,47 @@ per-person speech label enters the deployable model. \todo{Add an active-speaker or event auxiliary loss only if it improves the frozen primary protocol and its label/input boundary is fully reported.} -\section{Experiments} -\label{sec:experiments} +\section{Experiments} +\label{sec:experiments} -\textbf{Data and split.} -We follow the EgoSpeak EasyCom split: Sessions 4--12 for training-side model -development and Sessions 1--3 for final reporting \cite{donley2021easycom}. -Thresholds, windows, checkpoints, loss weights, and early stopping are selected -without using the official test sessions. Already observed test results are -treated as exploratory internal evidence and are not included in the paper. -\todo{Report duration, frame/event counts, class prevalence, visibility and -box coverage, source discontinuities, and the frozen calibration fold.} +\subsection{Dataset and Data Preparation} -\textbf{Baselines.} +We evaluate on EasyCom following the EgoSpeak split, using Sessions 4--12 for training and development and Sessions 1--3 for final evaluation \cite{donley2021easycom}. All hyperparameters, event windows, operating thresholds, and checkpoints are selected without access to the test sessions. The input stream is converted to strictly causal examples at 5~Hz. Each decision point is paired with ten future 0.2-s activity bins for background, camera wearer, and aggregate non-wearer speech. Provided person boxes are used only to construct the visible-person inputs; participant identity, close-mic audio, transcripts, and future observations are not available to the model. \todo{Report dataset duration, number of training/test examples and events, and the frozen development/calibration split.} + +\subsection{Model Configuration and Training} + +Our main backbone is Qwen2.5-Omni-3B. We compare a frozen global Omni readout, matched PEFT/LoRA adaptation, global features with recent local audio, a parameter- and history-matched temporal local branch, and the proposed causal local audio--visual adapter. Qwen2.5-Omni-7B is used only as a capacity-scaling control. All learned variants use the same causal context, prediction head, training examples, and class-balanced binary cross-entropy objective in Eq.~\eqref{eq:loss}. Unless otherwise stated, model selection is performed on the development data and final numbers are averaged over three random seeds. \todo{Freeze context length, local window/rates, optimizer, epochs, LoRA configuration, trainable parameter counts, and inference latency.} + +\subsection{Evaluation Metrics} + +We evaluate both dense future-activity prediction and event-level floor +control. For comparability with EgoSpeak, we report per-frame AP at +$0.2$--$2.0$~s horizons and their mean mAP. + +For event-level evaluation, a \emph{wearer entry} is a transition from wearer +silence to speech, while a \emph{wearer yield} is a transition from speech to +silence. An entry prediction is considered correct if it is committed within +the predefined response window of a ground-truth wearer onset. We report entry +recall, false entries per minute of eligible wearer-silent time, and signed +onset latency. Yield events are evaluated analogously using recall and latency. + +Multiparty interaction introduces a competing-risk condition. After another +speaker releases the floor, the current wearer-entry opportunity is terminated +if another non-wearer participant starts speaking before the wearer does. +Predictions after this competing onset are therefore not matched to the +previous opportunity. This prevents an other-to-other handover from being +treated as an open floor for the wearer. + +Operating thresholds are selected on the development split and frozen for test +evaluation. Our primary behavioral metric is false wearer entries per minute at +a matched wearer-entry recall of $0.8$; dense mAP and wearer-yield performance +are reported as complementary measures. + + +\section{Results and Analysis} +\label{sec:results} + +\subsection{Baselines} The confirmatory comparison contains: an EgoSpeak reproduction; the frozen Qwen2.5-Omni-3B global state $G$; matched LoRA/PEFT; $G$ plus recent local audio; a parameter- and history-matched temporal local branch; and the proposed @@ -237,40 +265,6 @@ not a second paper storyline. \todo{Freeze all baseline implementations and ensure the EgoSpeak comparison uses identical targets, split, causal validity, and evaluation.} -\textbf{Event protocol.} -Dense ten-horizon AP/mAP is retained for EgoSpeak comparability. The implemented -competing-risk evaluator counts each eligible aggregate-other boundary once, -ends a response window before the next other onset, and ends a yield window -before the next other offset. Pending the remaining P0 real-data audit and -scenario freeze, the intended primary behavior metric is false wearer entries per -minute of multi-other opportunity at a training-side selected wearer-onset -recall, with $R=.8$ as the headline operating point. The intended analysis also -reports onset latency and wearer-yield recall/latency, following the -event-oriented emphasis on recall, false alarms, and latency -\cite{jiang2026turnbench}. The protocol will censor a response window after an -other offset at the next other onset, and a yield window after an other onset -at the next other offset. - -\todo{P0 blocker: run the tested competing-risk evaluator on real predictions, -manually audit sampled timelines, and freeze pairing before treating any event -score as a paper result. Specify overlap, -minimum duration, timeouts, duplicate detections, source boundaries, offscreen -cases, and event counts. Until this is complete, call existing values -timing-derived proxies rather than final event results.} - -\textbf{Controls and statistics.} -The main intervention compares correctly aligned local evidence with -time-shifted evidence, removed vision, global-frame visual pooling, and the -capacity-matched temporal branch. A same-rate control is required for a -dual-rate claim. Track/no-ID/broken-continuity controls are added only if a -track-aware variant first wins on training-side data. We report three seeds, -mean$\pm$sample standard deviation, and paired cluster-bootstrap intervals over -sessions or conversation blocks. \todo{Freeze seeds, bootstrap unit/count, -optimizer, epochs, hardware, throughput, P50/P95 latency, and trainable -parameters.} - -\section{Results and Analysis} -\label{sec:results} \begin{table}[t] \caption{Confirmatory EasyCom results. Lower multi-other false entries are