Update evaluate protocol

This commit is contained in:
Xin Wang
2026-09-14 16:04:49 +08:00
parent 7292fea7fb
commit f4a7c01552

View File

@@ -231,25 +231,31 @@ camera-wearer, and aggregate non-wearer activity. Second, the original
participant timeline is retained to derive committed interaction events for
behavioral evaluation.
We consider two wearer-relative events. An \emph{EOT-response} denotes a
non-wearer floor release followed by the wearer taking the floor, whereas an
\emph{INT-yield} denotes a non-wearer takeover while the wearer is speaking,
followed by the wearer relinquishing the floor. Candidate events are generated
automatically from participant-level onset/offset annotations. Mid-turn pauses,
other-to-other handovers, backchannels, and non-takeover overlaps are retained
as hard-negative regions rather than positive events. Annotation-corrupted or
unresolvable boundaries are excluded from scoring. Ambiguous candidates are
manually adjudicated.
An \emph{EOT-response} is an observed wearer take-turn after a non-wearer floor
release; an \emph{INT-yield} is an observed wearer floor release after a
non-wearer takeover during wearer speech.
\textbf{Hard-negative region definition.}
Committed positive events and hard-negative regions are derived separately
from the participant-resolved speech timeline. Positive EOT-response and
INT-yield events are anchored by observed wearer behavior. In contrast,
hard-negative regions do not require any wearer transition. For example, a
mid-turn pause followed by the same non-wearer resuming speech, or an
other-to-other floor transfer with the wearer remaining silent throughout,
defines an EOT hard-negative region: a take-turn commitment within this region
is counted as a false positive. Likewise, brief backchannels or non-takeover
overlaps during wearer speech define INT hard-negative regions. Ambiguous or
annotation-corrupted regions are excluded from scoring.
Participant identity is used only to construct these evaluation annotations
and is never provided to the prediction model. Provided person boxes are used
only for the causal visible-person input; transcripts, target-of-speech labels,
close-microphone audio, and future observations are unavailable to the model.
\todo{Report duration and event statistics: EOT positives/hard negatives,
INT positives/hard negatives, excluded candidates, and manual-adjudication
count for train/development and test. Freeze the event-construction and
adjudication protocol before reporting semantic EOT/INT results; until then,
retain the timing-derived proxy terminology.}
INT positives/hard negatives, excluded candidates, and audited-region counts
for train/development and test. Freeze the event-construction and region-audit
protocol before reporting semantic EOT/INT results; until then, retain the
timing-derived proxy terminology.}
\subsection{Model Configuration and Training}
@@ -276,12 +282,14 @@ EgoSpeak, reporting AP at the 0.2--2.0~s horizons and their mean mAP.
Behavioral evaluation follows a committed-event protocol. For both
EOT-response and INT-yield, each gold event is matched to at most one committed
prediction and is scored by recall and signed detection latency. A prediction
inside a task-specific hard-negative region is counted as a false positive.
EOT hard negatives include mid-turn continuations and competing
other-to-other floor transfers; INT hard negatives include backchannels and
overlaps in which the wearer retains the floor. Regions excluded during
adjudication do not contribute to either the positive or negative denominator.
prediction and is scored by recall and signed detection latency.
\textbf{False-positive scoring rule.}
A false positive is defined by the model committing inside a labeled
hard-negative region, regardless of whether the wearer subsequently exhibits
an onset or offset. Thus, false-positive evaluation includes conversational
situations in which the correct wearer behavior is to remain silent or retain
the floor.
We report the recall--false-positive trade-off and a common operating point at
matched recall $R=0.8$. The primary behavioral quantities are therefore EOT