diff --git a/manuscript/main/main.tex b/manuscript/main/main.tex index 0d4a062..d9f35fa 100644 --- a/manuscript/main/main.tex +++ b/manuscript/main/main.tex @@ -240,7 +240,9 @@ converted into two parallel representations. First, they are downsampled to 5~Hz to construct the EgoSpeak-compatible ten-horizon targets for background, camera-wearer, and aggregate non-wearer activity. Second, the original participant timeline is retained to derive adjudicated interaction events for -behavioral evaluation. +behavioral evaluation. We additionally apply the annotation workflow to the +3,608 Ego4D AV validation candidates as a protocol stress test; Ego4D is not +used to train, tune, or compare the forecasting models reported here. \textbf{Behavioral event annotation.} We use every observed wearer activity transition as a high-recall semantic @@ -259,35 +261,26 @@ wearer response that nevertheless constitute false-positive opportunities: same-speaker pauses and other-to-other transitions for entry, and backchannels or non-takeover overlaps for INT. Only human-confirmed positives and hard negatives enter evaluation. Appendix~\ref{app:event_annotation} gives the full protocol. +Table~\ref{tab:behavior_annotation_stats} reports the current annotation +workload compactly; the EasyCom snapshot covers Sessions 1--4 only and is not +the final train/test annotation inventory. -\begin{table}[t] -\caption{Behavioral-event annotation statistics after human adjudication. -Counts must be filled only after the candidate protocol and annotation set are -frozen. Scoring-eligible totals include confirmed positives and hard negatives -only.} +\begin{table}[h] +\caption{LLM pre-label workload before human adjudication. Agreement is +between two independent passes.} \label{tab:behavior_annotation_stats} \centering -\small -\begin{tabular}{lcc} +\scriptsize +\begin{tabular}{@{}lrr@{}} \hline -Annotation outcome & Train/dev & Test \\ +Measure & EasyCom S1--4 & Ego4D AV val \\ \hline -\multicolumn{3}{l}{\emph{Wearer entry}}\\ -Positive: wearer entry & \todo{--} & \todo{--} \\ -HN: same-speaker pause & \todo{--} & \todo{--} \\ -HN: other-to-other transition & \todo{--} & \todo{--} \\ -HN: wearer backchannel & \todo{--} & \todo{--} \\ -\hline -\multicolumn{3}{l}{\emph{INT-yield}}\\ -Positive: INT-yield & \todo{--} & \todo{--} \\ -HN: other backchannel & \todo{--} & \todo{--} \\ -HN: non-takeover overlap & \todo{--} & \todo{--} \\ -HN: wearer holds floor & \todo{--} & \todo{--} \\ -\hline -Excluded: valid but out of scope & \todo{--} & \todo{--} \\ -Excluded: insufficient evidence & \todo{--} & \todo{--} \\ -Excluded: data quality & \todo{--} & \todo{--} \\ -\textbf{Scoring-eligible total} & \todo{--} & \todo{--} \\ +Candidates & 2,128 & 3,608 \\ +AI positive & 367 & 541 \\ +AI hard negative & 1,017 & 1,174 \\ +AI excluded & 744 & 1,893 \\ +Two-pass agreement & 93.4\% & 82.1\% \\ +Human-confirmed & 0 & 0 \\ \hline \end{tabular} \end{table} @@ -658,9 +651,11 @@ intent. EasyCom's \texttt{[L]} marker supplies laughter evidence only and is not by itself a sustained floor take or takeover. Preceding release options are optional metadata for a positive wearer-entry pre-label. -We use \texttt{deepseek-v4-flash} at temperature zero with a primary and a -skeptical pass. Disagreement on verdict, event type, or hard-negative subtype -produces an \textsc{Exclude} suggestion. Every candidate then receives mandatory +We use temperature-zero DeepSeek pre-labeling with a primary and a skeptical +pass (\texttt{deepseek-flash} for the EasyCom snapshot and +\texttt{deepseek-v4-flash} for the Ego4D AV snapshot). Disagreement on verdict, +event type, or hard-negative subtype produces an \textsc{Exclude} suggestion. +Every candidate then receives mandatory human adjudication: accept the complete pre-label and optional release metadata, relabel it, or exclude an unscorable region. Only human-confirmed \textsc{Wearer-Entry}, @@ -674,6 +669,49 @@ to insufficient evidence and is never counted as corrupt data. Annotation-only transcripts, identities, target-of-speech metadata, LLM output, and future observations are never supplied to the evaluated forecasting model. +% Snapshot source: reports/easycom_event_ai_prelabel_stats_v16/summary.json +% and reports/ego4d_av_event_candidates_2026-09-15/ego4d_av_val_candidates_ai.jsonl. +\begin{table}[t] +\caption{Descriptive LLM pre-label audit before human adjudication. Counts in +parentheses are percentages of all candidates in that column; agreement uses +all candidates as its denominator. These values describe annotation workload, +not ground truth or model performance.} +\label{tab:llm_prelabel_audit} +\centering +\scriptsize +\begin{tabular}{@{}lrr@{}} +\hline +Pre-label outcome & EasyCom S1--4 & Ego4D AV val \\ +\hline +All candidates & 2,128 & 3,608 \\ +Positive: wearer entry & 324 (15.2\%) & 503 (13.9\%) \\ +Positive: INT-yield & 43 (2.0\%) & 38 (1.1\%) \\ +HN: entry family & 892 (41.9\%) & 940 (26.1\%) \\ +HN: INT family & 125 (5.9\%) & 234 (6.5\%) \\ +Excluded: valid/out of scope & 602 (28.3\%) & 775 (21.5\%) \\ +Excluded: insufficient evidence & 142 (6.7\%) & 1,098 (30.4\%) \\ +Excluded: data quality & 0 (0.0\%) & 20 (0.6\%) \\ +Two-pass semantic agreement & 1,987 (93.4\%)& 2,961 (82.1\%) \\ +Human-confirmed & 0 & 0 \\ +\hline +\end{tabular} +\end{table} + +Table~\ref{tab:llm_prelabel_audit} expands the unreviewed AI summary in +Table~\ref{tab:behavior_annotation_stats}; neither table reports final human +annotation results. +The EasyCom column covers only Sessions 1--4, while the Ego4D AV column covers +the 3,608 validation candidates; the 32,768 Ego4D AV training candidates have +not yet received LLM pre-labels. All 5,736 reported rows remain unreviewed. +Because the candidate sets deliberately include mined hard-negative regions, +their class proportions are not estimates of natural event prevalence. The +higher-level counts are descriptive only and were not used for forecasting +threshold or model selection. The +higher Ego4D insufficient-evidence rate (30.4\% versus 6.7\%) and lower +two-pass agreement (82.1\% versus 93.4\%) identify a larger human-review burden; +they do not establish a cross-dataset accuracy difference without adjudicated +reference labels. + \todo{Before freezing this annotation protocol, audit randomly sampled continuous timeline segments without candidate hints to estimate candidate recall for entry, INT, and each hard-negative family. Independently double-label