Update evaluate protocol
This commit is contained in:
@@ -231,25 +231,31 @@ camera-wearer, and aggregate non-wearer activity. Second, the original
|
||||
participant timeline is retained to derive committed interaction events for
|
||||
behavioral evaluation.
|
||||
|
||||
We consider two wearer-relative events. An \emph{EOT-response} denotes a
|
||||
non-wearer floor release followed by the wearer taking the floor, whereas an
|
||||
\emph{INT-yield} denotes a non-wearer takeover while the wearer is speaking,
|
||||
followed by the wearer relinquishing the floor. Candidate events are generated
|
||||
automatically from participant-level onset/offset annotations. Mid-turn pauses,
|
||||
other-to-other handovers, backchannels, and non-takeover overlaps are retained
|
||||
as hard-negative regions rather than positive events. Annotation-corrupted or
|
||||
unresolvable boundaries are excluded from scoring. Ambiguous candidates are
|
||||
manually adjudicated.
|
||||
An \emph{EOT-response} is an observed wearer take-turn after a non-wearer floor
|
||||
release; an \emph{INT-yield} is an observed wearer floor release after a
|
||||
non-wearer takeover during wearer speech.
|
||||
|
||||
\textbf{Hard-negative region definition.}
|
||||
Committed positive events and hard-negative regions are derived separately
|
||||
from the participant-resolved speech timeline. Positive EOT-response and
|
||||
INT-yield events are anchored by observed wearer behavior. In contrast,
|
||||
hard-negative regions do not require any wearer transition. For example, a
|
||||
mid-turn pause followed by the same non-wearer resuming speech, or an
|
||||
other-to-other floor transfer with the wearer remaining silent throughout,
|
||||
defines an EOT hard-negative region: a take-turn commitment within this region
|
||||
is counted as a false positive. Likewise, brief backchannels or non-takeover
|
||||
overlaps during wearer speech define INT hard-negative regions. Ambiguous or
|
||||
annotation-corrupted regions are excluded from scoring.
|
||||
|
||||
Participant identity is used only to construct these evaluation annotations
|
||||
and is never provided to the prediction model. Provided person boxes are used
|
||||
only for the causal visible-person input; transcripts, target-of-speech labels,
|
||||
close-microphone audio, and future observations are unavailable to the model.
|
||||
\todo{Report duration and event statistics: EOT positives/hard negatives,
|
||||
INT positives/hard negatives, excluded candidates, and manual-adjudication
|
||||
count for train/development and test. Freeze the event-construction and
|
||||
adjudication protocol before reporting semantic EOT/INT results; until then,
|
||||
retain the timing-derived proxy terminology.}
|
||||
INT positives/hard negatives, excluded candidates, and audited-region counts
|
||||
for train/development and test. Freeze the event-construction and region-audit
|
||||
protocol before reporting semantic EOT/INT results; until then, retain the
|
||||
timing-derived proxy terminology.}
|
||||
|
||||
\subsection{Model Configuration and Training}
|
||||
|
||||
@@ -276,12 +282,14 @@ EgoSpeak, reporting AP at the 0.2--2.0~s horizons and their mean mAP.
|
||||
|
||||
Behavioral evaluation follows a committed-event protocol. For both
|
||||
EOT-response and INT-yield, each gold event is matched to at most one committed
|
||||
prediction and is scored by recall and signed detection latency. A prediction
|
||||
inside a task-specific hard-negative region is counted as a false positive.
|
||||
EOT hard negatives include mid-turn continuations and competing
|
||||
other-to-other floor transfers; INT hard negatives include backchannels and
|
||||
overlaps in which the wearer retains the floor. Regions excluded during
|
||||
adjudication do not contribute to either the positive or negative denominator.
|
||||
prediction and is scored by recall and signed detection latency.
|
||||
|
||||
\textbf{False-positive scoring rule.}
|
||||
A false positive is defined by the model committing inside a labeled
|
||||
hard-negative region, regardless of whether the wearer subsequently exhibits
|
||||
an onset or offset. Thus, false-positive evaluation includes conversational
|
||||
situations in which the correct wearer behavior is to remain silent or retain
|
||||
the floor.
|
||||
|
||||
We report the recall--false-positive trade-off and a common operating point at
|
||||
matched recall $R=0.8$. The primary behavioral quantities are therefore EOT
|
||||
|
||||
Reference in New Issue
Block a user