Add wide evaluation table and qualitative placeholder
This commit is contained in:
@@ -339,9 +339,12 @@ non-wearer release. It resolves positively when the wearer enters, or
|
||||
negatively when the same non-wearer resumes or a different non-wearer takes the
|
||||
next floor. From the first causal 5-Hz state that observes the release, the
|
||||
entry score is the maximum projected wearer probability before that competing
|
||||
resolution, capped at 2.0~s. Positive entries without a reviewer-selected
|
||||
release are reported as unanchored and excluded from this matched-opportunity
|
||||
analysis; they remain available to the separate raw-onset analysis.
|
||||
resolution, capped at 2.0~s. A selected-release positive is excluded if another
|
||||
non-wearer onset occurs before or exactly at the wearer entry, or if wearer
|
||||
activity already occurred after the selected release. Positive entries without
|
||||
a reviewer-selected release are reported as unanchored and excluded from this
|
||||
matched-opportunity analysis; they remain available to the separate raw-onset
|
||||
analysis.
|
||||
|
||||
An INT opportunity begins when a non-wearer starts while the wearer holds the
|
||||
floor. It resolves positively when the reviewed behavior is an INT-yield, and
|
||||
@@ -352,8 +355,9 @@ strictly at the competing non-wearer offset. All prediction values are read
|
||||
diagonally from one causal source row of the target-aligned forecast tensor.
|
||||
No input after the observed action outcome is used. For current INT-positive
|
||||
annotations, which do not select a competing onset explicitly, the
|
||||
implementation uses the latest supplied non-wearer interval crossing the
|
||||
wearer offset and surfaces cases with multiple eligible intervals for audit.
|
||||
implementation evaluates only a unique supplied non-wearer interval crossing
|
||||
the wearer offset; cases with multiple eligible intervals are excluded and
|
||||
counted for audit rather than assigned an arbitrary onset.
|
||||
|
||||
The headline entry hard negatives are same-speaker mid-turn pauses and other-
|
||||
to-other transfers. Wearer backchannels are not pooled into headline false-
|
||||
@@ -394,42 +398,57 @@ latency claim.
|
||||
|
||||
\subsection{Main Comparison}
|
||||
|
||||
Table~\ref{tab:main} compares the proposed causal local audiovisual adapter
|
||||
with an EgoSpeak reproduction, the frozen Qwen2.5-Omni-3B global state $G$,
|
||||
matched PEFT, $G$ plus recent local audio, and a parameter- and history-matched
|
||||
temporal local branch. Qwen2.5-Omni-7B is included only as a capacity-scaling
|
||||
control. All methods use the same split, causal inputs, targets, and evaluation
|
||||
protocol. Event thresholds are selected on training-side calibration data and
|
||||
frozen before test evaluation. \todo{Freeze all baseline implementations and
|
||||
verify identical targets, causal validity, and evaluation.}
|
||||
Table~\ref{tab:main} places the released EgoSpeak checkpoint beside the
|
||||
currently available Omni screening runs and reserves a row for the final causal
|
||||
local audiovisual model. Historical unmatched runs are identified explicitly
|
||||
and cannot support confirmatory attribution. The final comparison will use the
|
||||
same split, causal inputs, targets, training/selection protocol, and three
|
||||
random seeds; event thresholds will be selected on training-side calibration
|
||||
data and frozen before test evaluation. \todo{Freeze all baseline
|
||||
implementations and verify identical targets, causal validity, and evaluation.}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Confirmatory EasyCom results. Entry and INT false-positive rates use
|
||||
the opportunity-conditioned activity score at matched recall $R=0.8$; E/I AP
|
||||
is the corresponding opportunity AP. Event results will be populated only
|
||||
after protocol-v2 reruns and human adjudication.}
|
||||
\begin{table*}[t]
|
||||
\caption{EasyCom Sessions~1--3 dense future-activity and raw-boundary
|
||||
screening. All values are percentages. ``Avg.'' averages dense mAP over the ten
|
||||
horizons; On/Off are mean raw wearer onset/offset AP and are timing diagnostics,
|
||||
not semantic ENTRY/INT scores. $\dagger$ denotes historical seed-0 runs with an
|
||||
unmatched training/selection protocol; $\ddagger$ denotes a three-seed screen.
|
||||
Missing horizon curves and the final matched comparison will be filled after
|
||||
the matched dense/boundary reruns.}
|
||||
\label{tab:main}
|
||||
\centering
|
||||
\small
|
||||
\resizebox{\columnwidth}{!}{%
|
||||
\begin{tabular}{lccccc}
|
||||
\scriptsize
|
||||
\setlength{\tabcolsep}{2.6pt}
|
||||
\resizebox{\textwidth}{!}{%
|
||||
\begin{tabular}{@{}lccccccccccccc@{}}
|
||||
\hline
|
||||
Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$
|
||||
& Entry FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I AP $\uparrow$\\
|
||||
& \multicolumn{10}{c}{Dense mAP at prediction horizon (s) $\uparrow$}
|
||||
& & \multicolumn{2}{c}{Raw boundary AP $\uparrow$}\\
|
||||
\cline{2-11}\cline{13-14}
|
||||
Method & .2 & .4 & .6 & .8 & 1.0 & 1.2 & 1.4 & 1.6 & 1.8 & 2.0
|
||||
& Avg. & On & Off\\
|
||||
\hline
|
||||
EgoSpeak MiniROADA A+V & 74.12 & 61.16 & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
$G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
$G+$matched temporal local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
$G+$causal local AV (ours) & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
EgoSpeak MiniROADA A+V
|
||||
& 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82
|
||||
& 61.19 & 24.49 & 38.02\\
|
||||
Omni $G{+}A^{\dagger}$
|
||||
& 81.72 & -- & -- & -- & -- & -- & -- & -- & -- & --
|
||||
& 65.11 & 25.96 & 45.25\\
|
||||
Omni $G{+}A{+}$pooled-$V^{\dagger}$
|
||||
& 82.03 & -- & -- & -- & -- & -- & -- & -- & -- & --
|
||||
& 65.40 & 26.02 & 45.37\\
|
||||
Current-frame F1$^{\ddagger}$
|
||||
& $81.78{\pm}0.16$ & -- & -- & -- & -- & -- & -- & -- & -- & --
|
||||
& $64.95{\pm}0.09$ & -- & --\\
|
||||
$G+$causal local AV (ours)
|
||||
& -- & -- & -- & -- & -- & -- & -- & -- & -- & --
|
||||
& -- & -- & --\\
|
||||
\hline
|
||||
\end{tabular}}
|
||||
\end{table}
|
||||
\end{table*}
|
||||
|
||||
\textbf{Dense forecasting.}
|
||||
The EgoSpeak baseline reaches 74.12\% 0.2-s mAP and 61.16\% mean mAP; its raw
|
||||
The EgoSpeak baseline reaches 74.20\% 0.2-s mAP and 61.19\% mean mAP; its raw
|
||||
wearer-onset/offset mean APs are 24.49/38.02\%. Opportunity-conditioned values
|
||||
remain pending a protocol-v2 rerun and human review; continuous wearer-score
|
||||
plots do not alone establish semantic floor-yield selectivity.
|
||||
@@ -459,6 +478,34 @@ beyond additional temporal capacity}.
|
||||
\todo{If this difference is not statistically supported, state instead that
|
||||
the local temporal branch accounts for most of the gain.}
|
||||
|
||||
\subsection{Qualitative Floor-Event Analysis}
|
||||
|
||||
We will compare EgoSpeak with the final matched model on human-confirmed
|
||||
examples selected by pre-declared event subtype rather than post-hoc score
|
||||
extremes. Each case will align participant-resolved activity, minimal transcript
|
||||
context, the opportunity anchor and competing-risk censor, and the two models'
|
||||
causally emitted wearer forecasts. The comparison will include a genuine wearer
|
||||
entry, an other-to-other false-entry opportunity, and an INT-yield paired with a
|
||||
non-takeover overlap; a shared failure case will be retained when space permits.
|
||||
|
||||
\begin{figure*}[t]
|
||||
\centering
|
||||
\fbox{\parbox[c][0.115\textheight][c]{0.965\textwidth}{
|
||||
\centering
|
||||
\textbf{Qualitative comparison placeholder}\\[0.7em]
|
||||
\begin{tabular}{ccc}
|
||||
Case A: wearer entry & Case B: other$\rightarrow$other HN
|
||||
& Case C: INT-yield / failed takeover\\
|
||||
participant timeline + transcript & opportunity anchor + censor
|
||||
& EgoSpeak vs. proposed forecast\\
|
||||
\end{tabular}
|
||||
}}
|
||||
\caption{Planned qualitative floor-event comparison. Final panels will use
|
||||
human-confirmed protocol-v2 cases and show the same causal source time for both
|
||||
models; the figure is a layout placeholder and contains no result claim.}
|
||||
\label{fig:qualitative_floor_events}
|
||||
\end{figure*}
|
||||
|
||||
\subsection{Information Opportunity}
|
||||
|
||||
Before designing the deployable adapter, we measured whether local floor
|
||||
@@ -569,7 +616,7 @@ future-media intervention test. Report within-backbone 5-Hz versus 1-Hz
|
||||
deltas plus cache/no-cache throughput and memory; do not interpret unmatched
|
||||
Qwen--MiniCPM raw scores as a pure backbone effect.}
|
||||
|
||||
\subsection{Limitations and Qualitative Analysis}
|
||||
\subsection{Limitations}
|
||||
|
||||
The results support \todo{precise supported claim}, but several limitations
|
||||
remain.
|
||||
@@ -579,10 +626,9 @@ tracking assumption}.
|
||||
Third, \todo{failure condition: occlusion, overlapping speakers, rapid floor
|
||||
exchange, etc.} remains challenging.
|
||||
|
||||
\todo{Add one compact, pre-selected qualitative timeline only if space permits:
|
||||
one multi-other false-entry improvement paired with one preserved real wearer
|
||||
onset or one failure. Examples illustrate aggregate rates and never establish
|
||||
the claim by themselves.}
|
||||
\todo{Replace Fig.~\ref{fig:qualitative_floor_events} with compact, pre-selected
|
||||
human-confirmed timelines. Examples illustrate aggregate rates and never
|
||||
establish the claim by themselves.}
|
||||
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
Reference in New Issue
Block a user