197 lines
10 KiB
TeX
197 lines
10 KiB
TeX
% ARCHIVED INTERNAL NOTES: not included in the four-page submission draft.
|
|
% Source: main.tex before the 2026-09-17 local-evidence restructure.
|
|
% Provisional annotation workload and protocol notes, not model results.
|
|
% Table references refer to the previous manuscript layout.
|
|
|
|
% Upstream 2026-09-17 merge: semantic evaluation remains deferred in main.tex.
|
|
% Protocol-v1 opportunity AP is superseded: ENTRY cannot cross another other
|
|
% onset/intervening wearer activity; multi-competitor INT must not use an
|
|
% arbitrary onset. Re-run protocol v2 and audit labels before semantic claims.
|
|
% Retained upstream semantic qualitative plan (not compiled in main.tex):
|
|
\subsection{Qualitative Floor-Event Analysis}
|
|
|
|
We will compare EgoSpeak with the final matched model on human-confirmed
|
|
examples selected by pre-declared event subtype rather than post-hoc score
|
|
extremes. Each case will align participant-resolved activity, minimal transcript
|
|
context, the opportunity anchor and competing-risk censor, and the two models'
|
|
causally emitted wearer forecasts. The comparison will include a genuine wearer
|
|
entry, an other-to-other false-entry opportunity, and an INT-yield paired with a
|
|
non-takeover overlap; a shared failure case will be retained when space permits.
|
|
|
|
\begin{figure*}[t]
|
|
\centering
|
|
\fbox{\parbox[c][0.115\textheight][c]{0.965\textwidth}{
|
|
\centering
|
|
\textbf{Qualitative comparison placeholder}\\[0.7em]
|
|
\begin{tabular}{ccc}
|
|
Case A: wearer entry & Case B: other$\rightarrow$other HN
|
|
& Case C: INT-yield / failed takeover\\
|
|
participant timeline + transcript & opportunity anchor + censor
|
|
& EgoSpeak vs. proposed forecast\\
|
|
\end{tabular}
|
|
}}
|
|
\caption{Planned qualitative floor-event comparison. Final panels will use
|
|
human-confirmed protocol-v2 cases and show the same causal source time for both
|
|
models; the figure is a layout placeholder and contains no result claim.}
|
|
\label{fig:qualitative_floor_events}
|
|
\end{figure*}
|
|
|
|
\begin{table}[h]
|
|
\caption{LLM pre-label workload before human adjudication. Agreement is
|
|
between two independent passes.}
|
|
\label{tab:behavior_annotation_stats}
|
|
\centering
|
|
\scriptsize
|
|
\begin{tabular}{@{}lrr@{}}
|
|
\hline
|
|
Measure & EasyCom S1--4 & Ego4D AV val \\
|
|
\hline
|
|
Candidates & 2,128 & 3,608 \\
|
|
AI positive & 367 & 541 \\
|
|
AI hard negative & 1,017 & 1,174 \\
|
|
AI excluded & 744 & 1,893 \\
|
|
Two-pass agreement & 93.4\% & 82.1\% \\
|
|
Human-confirmed & 0 & 0 \\
|
|
\hline
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\appendix
|
|
\section{LLM-Assisted Behavioral Event Annotation}
|
|
\label{app:event_annotation}
|
|
|
|
EasyCom's raw speech boundaries are acoustic annotations rather than floor
|
|
labels. We preserve them unchanged for dense forecasting and create a separate
|
|
behavioral-event layer. Its design principle is to anchor an observed wearer
|
|
behavior first and then retrieve the conversational opportunity that could make
|
|
that behavior meaningful. False-positive opportunities are mined independently
|
|
because, by definition, many contain no wearer transition.
|
|
|
|
\subsection{Positive-event mining}
|
|
|
|
\textbf{Wearer activity transitions.}
|
|
We preserve every raw wearer $0\!\rightarrow\!1$ and $1\!\rightarrow\!0$
|
|
boundary as a semantic review candidate. Candidate construction does not merge
|
|
short silence gaps, infer within-turn resumption, or decide whether overlapping
|
|
speech constitutes takeover. It only records participant-resolved temporal and
|
|
transcript facts. The source intervals and dense targets remain unchanged.
|
|
|
|
\textbf{Wearer entry.}
|
|
Every raw wearer onset at $t_b$ is proposed as one wearer-entry review item.
|
|
We search backward within the retrieval window for non-wearer offsets and store
|
|
each one with its response delay and factual indicators of intervening wearer
|
|
speech, intervening other onsets, and concurrent activity. These are possible
|
|
context anchors rather than deterministically admissible causes. The LLM and
|
|
human reviewer decide whether the boundary is a genuine new entry, a
|
|
within-turn resumption, a wearer backchannel, or an excluded non-event. A
|
|
preceding release is not required for a positive entry: the wearer may enter
|
|
after a release, from silence, or while another speaker remains active. When a
|
|
release clearly prompts the entry, the LLM or human may select one supplied
|
|
option; neither may invent or move a timestamp.
|
|
|
|
\textbf{INT-yield.}
|
|
Every raw wearer offset $t_y$ is proposed as an INT-yield review item. The
|
|
generator records the current wearer activity-run start, all nearby non-wearer
|
|
onsets and offsets, activity immediately before and after $t_y$, whether an
|
|
other speaker is pre-existing or begins during wearer speech, strict and
|
|
50-ms-tolerant boundary crossing, overlap duration, the next speech onset,
|
|
transcripts, and target-of-speech metadata. Relations such as
|
|
$t_w<t_o<t_y<t_o^{\mathrm{off}}$ are descriptive evidence only: they neither
|
|
admit nor reject a candidate and do not establish takeover. The LLM and human
|
|
annotator answer whether the wearer stopped because a non-wearer genuinely took
|
|
over the floor. Natural completion, post-completion activity, and within-turn
|
|
pauses are therefore resolved semantically rather than pre-filtered.
|
|
When a wearer activity run is itself only a backchannel within a non-wearer's
|
|
retained turn, its onset supplies the false-entry opportunity; its trailing
|
|
offset is excluded rather than counted again as a false-yield opportunity.
|
|
|
|
\subsection{False-positive opportunity mining}
|
|
|
|
Hard-negative regions are retrieved independently of positive wearer
|
|
transitions. For entry, a same-speaker pause followed by resumption and an
|
|
other-to-other handover while the wearer stays silent are false-entry
|
|
opportunities. For INT, a short non-wearer backchannel or a non-takeover overlap
|
|
during which the wearer retains and continues the floor is a false-yield
|
|
opportunity. The timing rules provide candidate types, not final semantic
|
|
labels.
|
|
|
|
\subsection{Semantic pre-labeling and human adjudication}
|
|
|
|
The annotation assistant jointly receives immutable timestamps,
|
|
participant-resolved activity, neighboring boundaries, gap and overlap
|
|
measurements, transcripts, and target-of-speech metadata when available. It is
|
|
instructed not to equate raw onset with floor entry, offset with yield, or silence
|
|
with an entry opportunity,
|
|
or overlap with interruption, and not to invent unavailable prosody, gaze, or
|
|
intent. EasyCom's \texttt{[L]} marker supplies laughter evidence only and is not
|
|
by itself a sustained floor take or takeover. Preceding release options are
|
|
optional metadata for a positive wearer-entry pre-label.
|
|
|
|
We use temperature-zero DeepSeek pre-labeling with a primary and a skeptical
|
|
pass (\texttt{deepseek-flash} for the EasyCom snapshot and
|
|
\texttt{deepseek-v4-flash} for the Ego4D AV snapshot). Disagreement on verdict,
|
|
event type, or hard-negative subtype produces an \textsc{Exclude} suggestion.
|
|
Every candidate then receives mandatory
|
|
human adjudication: accept the complete pre-label and optional release metadata,
|
|
relabel it, or exclude an
|
|
unscorable region. Only human-confirmed \textsc{Wearer-Entry},
|
|
\textsc{INT-Yield}, and hard-negative opportunities enter evaluation;
|
|
\textsc{Exclude} candidates do not. Thus \textsc{Exclude} is the sole
|
|
non-scoring verdict, but every exclusion retains a closed-vocabulary reason and
|
|
derived aggregation group. We report valid-but-out-of-scope behavior (e.g.,
|
|
natural wearer completion coincident with feedback), insufficient semantic
|
|
evidence, and data-quality failures separately; model-pass disagreement belongs
|
|
to insufficient evidence and is never counted as corrupt data.
|
|
Annotation-only transcripts, identities, target-of-speech metadata, LLM output,
|
|
and future observations are never supplied to the evaluated forecasting model.
|
|
|
|
% Snapshot source: reports/easycom_event_ai_prelabel_stats_v16/summary.json
|
|
% and reports/ego4d_av_event_candidates_2026-09-15/ego4d_av_val_candidates_ai.jsonl.
|
|
\begin{table}[t]
|
|
\caption{Descriptive LLM pre-label audit before human adjudication. Counts in
|
|
parentheses are percentages of all candidates in that column; agreement uses
|
|
all candidates as its denominator. These values describe annotation workload,
|
|
not ground truth or model performance.}
|
|
\label{tab:llm_prelabel_audit}
|
|
\centering
|
|
\scriptsize
|
|
\begin{tabular}{@{}lrr@{}}
|
|
\hline
|
|
Pre-label outcome & EasyCom S1--4 & Ego4D AV val \\
|
|
\hline
|
|
All candidates & 2,128 & 3,608 \\
|
|
Positive: wearer entry & 324 (15.2\%) & 503 (13.9\%) \\
|
|
Positive: INT-yield & 43 (2.0\%) & 38 (1.1\%) \\
|
|
HN: entry family & 892 (41.9\%) & 940 (26.1\%) \\
|
|
HN: INT family & 125 (5.9\%) & 234 (6.5\%) \\
|
|
Excluded: valid/out of scope & 602 (28.3\%) & 775 (21.5\%) \\
|
|
Excluded: insufficient evidence & 142 (6.7\%) & 1,098 (30.4\%) \\
|
|
Excluded: data quality & 0 (0.0\%) & 20 (0.6\%) \\
|
|
Two-pass semantic agreement & 1,987 (93.4\%)& 2,961 (82.1\%) \\
|
|
Human-confirmed & 0 & 0 \\
|
|
\hline
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Table~\ref{tab:llm_prelabel_audit} expands the unreviewed AI summary in
|
|
Table~\ref{tab:behavior_annotation_stats}; neither table reports final human
|
|
annotation results.
|
|
The EasyCom column covers only Sessions 1--4, while the Ego4D AV column covers
|
|
the 3,608 validation candidates; the 32,768 Ego4D AV training candidates have
|
|
not yet received LLM pre-labels. All 5,736 reported rows remain unreviewed.
|
|
Because the candidate sets deliberately include mined hard-negative regions,
|
|
their class proportions are not estimates of natural event prevalence. The
|
|
higher-level counts are descriptive only and were not used for forecasting
|
|
threshold or model selection. The
|
|
higher Ego4D insufficient-evidence rate (30.4\% versus 6.7\%) and lower
|
|
two-pass agreement (82.1\% versus 93.4\%) identify a larger human-review burden;
|
|
they do not establish a cross-dataset accuracy difference without adjudicated
|
|
reference labels.
|
|
|
|
\todo{Before freezing this annotation protocol, audit randomly sampled
|
|
continuous timeline segments without candidate hints to estimate candidate
|
|
recall for entry, INT, and each hard-negative family. Independently double-label
|
|
a pre-registered subset, report its size and agreement statistic, freeze all
|
|
candidate parameters and prompt/model identifiers on Sessions 4--12, and
|
|
report final candidate, control, exclusion, and adjudicated-label counts.}
|