213 lines
10 KiB
Markdown
213 lines
10 KiB
Markdown
# EasyCom wearer-entry/INT pre-label and review pipeline
|
|
|
|
The workflow has three explicit stages. Speech activity timestamps are
|
|
read-only throughout; reviewed events never modify the dense 5 Hz activity
|
|
ground truth.
|
|
|
|
```text
|
|
EasyCom participant speech intervals
|
|
-> all wearer boundaries + independent rule regions + structured factual evidence
|
|
-> DeepSeek semantic pre-label (optional but recommended)
|
|
-> human Accept / Relabel / Exclude
|
|
```
|
|
|
|
## 1. Generate candidates
|
|
|
|
Develop the rules on training-side Sessions 4--12:
|
|
|
|
```bash
|
|
python3 prepare_easycom_eot_int_review.py \
|
|
--dataset-root /path/to/easycom \
|
|
--sessions Session_4 Session_5 Session_6 Session_7 Session_8 Session_9 Session_10 Session_11 Session_12 \
|
|
--output-dir reports/easycom_floor_event_candidates
|
|
```
|
|
|
|
The generator emits two independent files:
|
|
|
|
- `easycom_eot_int_candidates.jsonl` (legacy-compatible filename) contains every
|
|
raw wearer `0->1` boundary (`WEARER_ONSET`, `WEARER_ENTRY_CANDIDATE`), every
|
|
raw wearer `1->0` boundary (`WEARER_OFFSET`, `YIELD_CANDIDATE`), and hard-negative
|
|
regions that need no wearer transition:
|
|
`SAME_SPEAKER_PAUSE`, `OTHER_TO_OTHER`, `OTHER_BACKCHANNEL`, and
|
|
`NON_TAKEOVER_OVERLAP`;
|
|
- `easycom_eot_int_controls.jsonl` is retained for file-format compatibility
|
|
and legacy inputs. The current boundary protocol does not deterministically
|
|
demote wearer onsets or offsets into this file.
|
|
|
|
No short-gap merge or takeover heuristic controls wearer-boundary admission.
|
|
For each raw onset, `temporal_evidence` stores the previous wearer offset,
|
|
silence duration, every other-speaker interval intersecting that gap, interval
|
|
coverage, whether it continues across the onset, transcript, and
|
|
`Target_of_Speech`. DeepSeek and the human reviewer decide whether the onset is
|
|
a genuine entry, within-turn resumption, backchannel, or excluded non-event.
|
|
|
|
Entry construction searches backward for nearby non-wearer releases as optional
|
|
context. Every option carries factual flags for intervening wearer activity,
|
|
intervening other onsets, and activity at the wearer onset; the generator does
|
|
not declare one a valid conversational cause. Each wearer onset produces exactly
|
|
one primary row. When present, `opportunity_options` contains the only release
|
|
anchors that DeepSeek or a human may optionally select; neither may invent a
|
|
timestamp.
|
|
|
|
All wearer onsets and offsets carry the rule-stage value `candidate`; timing
|
|
alone is not a semantic positive label. For each offset, the JSON records the
|
|
wearer activity-run start, nearby other intervals, pre-existing and new-other
|
|
flags, activity immediately before/after the boundary, strict and configurable
|
|
50-ms-tolerant crossing, overlap duration, and subsequent speech onsets.
|
|
`wearer_start < other.start < wearer_offset < other.end` is evidence rather than
|
|
an INT prerequisite. Hard-negative scans are rule hints and still require AI
|
|
and human review. The JSONL retains candidate source, original event time,
|
|
participant-level activity, raw durations/gaps/overlap evidence, transcript,
|
|
and `Target_of_Speech` when present. `Target_of_Speech` is annotation assistance
|
|
only and cannot be a deployable model input.
|
|
|
|
All heuristic durations are configurable rather than frozen paper rules:
|
|
|
|
```text
|
|
--relation-window-seconds
|
|
--same-speaker-pause-max-seconds
|
|
--other-handover-max-seconds
|
|
--short-vocalization-max-seconds
|
|
--continuation-min-seconds
|
|
--int-offset-tolerance-seconds
|
|
```
|
|
|
|
Use `--max-candidates 1` for a one-event smoke test. `summary.json` records the
|
|
exact configuration and candidate counts. `easycom_session_timelines.json`
|
|
stores the read-only participant speech lanes used by the full-session web
|
|
timeline; it is not sent to DeepSeek.
|
|
|
|
## 2. Add DeepSeek pre-labels
|
|
|
|
Prompt v14 first reconstructs the exact participant-resolved chronology and
|
|
then uses transcript content and `Target_of_Speech` to determine conversational
|
|
function. It treats the EasyCom transcript marker `[L]` as laughter-only evidence,
|
|
not as a lexical floor take. Wearer laughter during another participant's
|
|
retained turn is directed toward `ENTRY_WEARER_BACKCHANNEL`; non-wearer laughter
|
|
overlapping a retained wearer turn is directed toward `INT_OTHER_BACKCHANNEL`.
|
|
Ambiguous laughter falls back to `EXCLUDE`, and `[L]` alone is neither evidence
|
|
of a floor take nor evidence of a visible smile. Preceding release options are
|
|
optional entry metadata. No entry/within-turn or INT/takeover semantic decision
|
|
is made before the prompt. A positive `WEARER_ENTRY` may optionally select one
|
|
supplied release; entries from silence or during other activity use `null`.
|
|
|
|
`needs_human_review` is not requested from DeepSeek: every pre-label enters the
|
|
human queue by protocol, so the adjudicator assigns that boolean locally. This
|
|
prevents a string-valued model field from invalidating and retrying an otherwise
|
|
valid batch.
|
|
|
|
First inspect exactly what would be uploaded:
|
|
|
|
```bash
|
|
python3 adjudicate_easycom_eot_int_candidates.py \
|
|
--input reports/easycom_floor_event_candidates/easycom_eot_int_candidates.jsonl \
|
|
--output reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
|
|
--model deepseek-v4-flash \
|
|
--max-events 1 \
|
|
--dry-run
|
|
```
|
|
|
|
After setting `DEEPSEEK_API_KEY` and obtaining authorization to upload the
|
|
candidate excerpts:
|
|
|
|
```bash
|
|
python3 adjudicate_easycom_eot_int_candidates.py \
|
|
--input reports/easycom_floor_event_candidates/easycom_eot_int_candidates.jsonl \
|
|
--output reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
|
|
--model deepseek-v4-flash \
|
|
--allow-external-transcript-upload
|
|
```
|
|
|
|
DeepSeek sees candidate source, structured timing, per-speaker activity, and
|
|
transcript together. It returns one of `POSITIVE`, `HARD_NEGATIVE`, or
|
|
`EXCLUDE`; positives additionally receive
|
|
`WEARER_ENTRY` or `INT_YIELD`, and hard negatives receive a named subtype. It
|
|
cannot modify timestamps. The output preserves the original rule candidate,
|
|
the selected preceding-release UID when applicable, and the complete AI
|
|
decision/provenance. `EXCLUDE` is the sole non-scoring outcome, but it is never
|
|
an untyped bucket. Every AI and human exclusion stores one closed-vocabulary
|
|
`exclude_reason` and the derived `exclude_reason_group`. `OUT_OF_SCOPE_*` means
|
|
the annotations are valid but the behavior is outside the scored target;
|
|
`INSUFFICIENT_EVIDENCE` means the supplied valid material cannot resolve
|
|
conversational function; and `DATA_*` identifies unusable annotation,
|
|
recording, context, artifact, or timestamp quality. Model-pass disagreement is
|
|
recorded as `INSUFFICIENT_EVIDENCE`, never as data corruption.
|
|
|
|
Hard negatives use behavior-specific subtypes. ENTRY distinguishes a same-other
|
|
mid-turn pause, a different-other transition, and a wearer backchannel that
|
|
does not take the floor. INT first distinguishes an other backchannel, then a
|
|
non-backchannel overlap after which the wearer demonstrably holds or resumes
|
|
the floor, and finally another interpretable overlap that fails to become a
|
|
sustained takeover. `OTHER_HARD_NEGATIVE` is reserved for valid false-positive
|
|
opportunities outside these named behaviors; it is never an uncertainty
|
|
fallback. Prompt v15 gives adjacency-pair completion priority over duration:
|
|
a short but complete answer to an addressed question remains a positive wearer
|
|
entry, and floor entry does not require the wearer to retain the next turn.
|
|
Later resumption by the questioner or another speaker therefore does not turn
|
|
the completed answer into a wearer backchannel. `ENTRY_WEARER_BACKCHANNEL` is
|
|
reserved for feedback that does not answer a question, accept or reject a
|
|
proposal, provide requested information, or otherwise complete an adjacency
|
|
pair.
|
|
|
|
During live adjudication, the CLI displays a dependency-free progress bar over
|
|
all candidate/pass work, including the current pass, batch, percentage, and
|
|
elapsed time. Redirected logs emit periodic progress lines instead of carriage
|
|
return animation.
|
|
|
|
Every current `WEARER_ONSET` and `WEARER_OFFSET` row is sent for semantic
|
|
adjudication regardless of its timing pattern. Explicit control rows from older
|
|
protocol files remain excluded unless controls are intentionally reviewed.
|
|
|
|
## 3. Human review
|
|
|
|
Run the web workspace on the AI-enriched JSONL:
|
|
|
|
```bash
|
|
python3 review_floor_events.py \
|
|
--dataset-root /path/to/easycom \
|
|
--events reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
|
|
--output reports/easycom_event_reviews/reviews.jsonl
|
|
```
|
|
|
|
Open `http://127.0.0.1:8765`. The page shows a session candidate overview,
|
|
participant activity near the event, video/audio, transcript, structured timing,
|
|
and the AI verdict/confidence/reason. A decision guide first states the behavioral
|
|
question, then presents the exact AI verdict/subtype with a plain-language
|
|
meaning; the Accept control repeats that exact label so its effect is explicit.
|
|
The common path is one operation:
|
|
|
|
- **Accept** keeps the AI verdict/type and optional preceding-release metadata;
|
|
- **Relabel** expands verdict and subtype controls; a positive entry may
|
|
optionally select one of the supplied release anchors;
|
|
- **Exclude** requires a concrete out-of-scope or data-quality reason.
|
|
|
|
The current primary-review UI deliberately has no second-review action. Older
|
|
saved `second_review` records remain readable for backward compatibility.
|
|
|
|
Boundary correction is hidden under an optional disclosure and is limited to
|
|
one 200 ms bin. The original timestamp is always retained. Each save atomically
|
|
rewrites JSONL and CSV while preserving the rule candidate, AI decision, human
|
|
decision, relabel flag, exclusion state, confidence, and provenance. The current
|
|
UI does not ask for a reviewer name; the export retains an empty `reviewer` field
|
|
only for compatibility with older review files.
|
|
|
|
The primary review queue filters only explicit legacy control rows. To audit
|
|
those controls intentionally,
|
|
pass `--events .../easycom_eot_int_controls.jsonl --include-controls` and use a
|
|
separate review output file. The old `--include-offset-controls` spelling
|
|
remains an alias.
|
|
|
|
Protocol wording for the paper appendix:
|
|
|
|
> Every observed wearer speech onset and offset is retained as a semantic review
|
|
> candidate. Deterministic construction records only participant-resolved timing,
|
|
> transcript, and target-of-speech facts; it does not infer within-turn
|
|
> resumption, floor entry, interruption, or takeover.
|
|
|
|
## Freeze discipline
|
|
|
|
Develop the guide and resolve disagreements only on Sessions 4--12. Freeze
|
|
categories and thresholds before reviewing official test Sessions 1--3. Use two
|
|
independent human review files plus a separate adjudication output for paper-
|
|
quality labels. DeepSeek is pre-label metadata, not an independent reviewer.
|