wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make EasyCom event review wearer anchor first

Browse Source
This commit is contained in:
Xin Wang
2026-09-14 23:30:31 +08:00
parent 06b5e6c385
commit 65347dd75d
12 changed files with 336 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files

69
adjudicate_easycom_eot_int_candidates.py
View File

@@ -26,7 +26,7 @@ import urllib.request
from prepare_easycom_eot_int_review import requires_semantic_adjudication
PROMPT_VERSION = "easycom-wearer-centric-eot-int-prelabel-v6"
PROMPT_VERSION = "easycom-wearer-anchor-first-eot-int-prelabel-v7"
DEFAULT_MODEL = "deepseek-v4-flash"
DEFAULT_BASE_URL = "https://api.deepseek.com"
VERDICTS = {"POSITIVE", "HARD_NEGATIVE", "NOT_APPLICABLE", "EXCLUDE", "UNSURE"}
@@ -93,7 +93,7 @@ Evidence rules:
2. Never alter or propose an event timestamp. Do not infer prosody, gaze, intention, or acoustic cues that were not supplied.
3. A wearer onset is not automatically a floor take; a wearer offset is not automatically a yield; an other onset is not automatically a takeover. A pause is not automatically EOT, overlap is not automatically interruption, and words such as "yeah" are not automatically backchannels. Check who speaks sustainably afterward and who resumes.
4. Use UNSURE when the evidence does not establish conversational function. Never force a label for coverage.
5. Every supplied WEARER_ONSET has two immutable anchors: a non-wearer release opportunity and a later floor-level wearer entry. Do not reject it solely because response_delay_seconds is long. Decide whether the release remained available, whether the wearer contribution is a genuine response, or whether it is a semantic hard negative such as feedback.
5. Every supplied WEARER_ONSET was found by starting from an observed floor-level wearer entry and looking backward. Its immutable behavior anchor is fixed. Select the true preceding release only from temporal_evidence.opportunity_options; never invent a trigger or timestamp. A POSITIVE EOT_RESPONSE must return that option's uid as selected_opportunity_uid. Do not reject an option solely because its response delay is long. If none is a defensible trigger, return another verdict and null.
6. Every supplied WEARER_OFFSET has already passed both deterministic INT prerequisites: a new non-wearer onset occurs strictly inside the current wearer floor episode (t_w < t_o < t_y), and that same non-wearer interval crosses the wearer yield boundary within the recorded annotation tolerance. Decide only whether it is a genuine takeover or a semantic hard negative. Pre-existing other activity, post-completion other onset, within-turn wearer pauses, and completion-to-silence are controls outside this queue.
7. Transcript event markers are evidence, not lexical turns. In EasyCom, a transcription consisting of [L] denotes laughter-only vocalization. Do not treat [L] as a sustained floor take, an EOT response, or an interruption takeover merely because its activity interval creates an onset, offset, or overlap candidate. Specifically:
- wearer [L] during a non-wearer's retained turn is normally HARD_NEGATIVE / EOT_WEARER_BACKCHANNEL, not EOT_RESPONSE;
@@ -113,7 +113,7 @@ POSITIVE event_type must be exactly EOT_RESPONSE or INT_YIELD. EOT_RESPONSE mean
HARD_NEGATIVE subtype must be exactly one of EOT_MIDTURN_PAUSE, EOT_OTHER_TO_OTHER, EOT_WEARER_BACKCHANNEL, INT_OTHER_BACKCHANNEL, INT_NON_TAKEOVER_OVERLAP, INT_WEARER_HOLDS_FLOOR, or OTHER_HARD_NEGATIVE.
Return exactly one JSON object with this shape and no markdown:
{"items":[{"uid":"...","verdict":"UNSURE","event_type":null,"subtype":null,"confidence":0.25,"reason":"brief evidence-based sentence"}]}
{"items":[{"uid":"...","verdict":"UNSURE","event_type":null,"subtype":null,"selected_opportunity_uid":null,"confidence":0.25,"reason":"brief evidence-based sentence"}]}
Return every supplied uid exactly once. Copy uid byte-for-byte and never add a backslash before an underscore. confidence must be a number from 0 to 1."""
@@ -211,7 +211,11 @@ def _transport_rows(rows: Sequence[dict[str, Any]]) -> tuple[list[dict[str, Any]
return transported, original_by_transport
def validate_decisions(payload: Any, expected_uids: Sequence[str]) -> list[dict[str, Any]]:
def validate_decisions(
payload: Any,
expected_uids: Sequence[str],
allowed_opportunities: dict[str, set[str]] | None = None,
) -> list[dict[str, Any]]:
if not isinstance(payload, dict) or not isinstance(payload.get("items"), list):
raise ValueError("Response must be an object with an items list")
expected = set(expected_uids)
@@ -224,6 +228,7 @@ def validate_decisions(payload: Any, expected_uids: Sequence[str]) -> list[dict[
verdict = item.get("verdict")
event_type = item.get("event_type")
subtype = item.get("subtype")
selected_opportunity_uid = item.get("selected_opportunity_uid")
confidence = item.get("confidence")
reason = item.get("reason")
if uid not in expected or uid in decisions:
@@ -238,6 +243,14 @@ def validate_decisions(payload: Any, expected_uids: Sequence[str]) -> list[dict[
raise ValueError(f"HARD_NEGATIVE requires a valid subtype for {uid}")
if verdict != "HARD_NEGATIVE" and subtype is not None:
raise ValueError(f"Only HARD_NEGATIVE may have subtype for {uid}")
allowed = set() if allowed_opportunities is None else allowed_opportunities.get(str(uid), set())
if verdict == "POSITIVE" and event_type == "EOT_RESPONSE" and allowed:
if selected_opportunity_uid not in allowed:
raise ValueError(
f"EOT_RESPONSE must select a supplied opportunity uid for {uid}"
)
elif selected_opportunity_uid is not None:
raise ValueError(f"selected_opportunity_uid is only valid for EOT_RESPONSE: {uid}")
if isinstance(confidence, bool) or not isinstance(confidence, (int, float)) or not 0 <= confidence <= 1:
raise ValueError(f"confidence must be in [0, 1] for {uid}")
if not isinstance(reason, str) or not reason.strip():
@@ -247,6 +260,7 @@ def validate_decisions(payload: Any, expected_uids: Sequence[str]) -> list[dict[
"verdict": str(verdict),
"event_type": event_type,
"subtype": subtype,
"selected_opportunity_uid": selected_opportunity_uid,
"confidence": float(confidence),
"reason": reason.strip(),
# Every AI decision enters the human queue. This is deliberately
@@ -295,7 +309,20 @@ def _post(
envelope = json.loads(response.read().decode("utf-8"))
content = envelope["choices"][0]["message"]["content"]
parsed = json.loads(content)
decisions = validate_decisions(parsed, list(original_by_transport))
allowed_opportunities = {}
for row in transported:
evidence = row.get("temporal_evidence") or {}
options = evidence.get("opportunity_options") or []
allowed_opportunities[row["uid"]] = {
str(option["uid"])
for option in options
if isinstance(option, dict) and option.get("uid") is not None
}
if not allowed_opportunities[row["uid"]] and evidence.get("opportunity_source_uid"):
allowed_opportunities[row["uid"]].add(str(evidence["opportunity_source_uid"]))
decisions = validate_decisions(
parsed, list(original_by_transport), allowed_opportunities
)
for decision in decisions:
decision["uid"] = original_by_transport[decision["uid"]]
return decisions, envelope.get("usage", {})
@@ -376,23 +403,47 @@ def add_suggestions(
output: list[dict[str, Any]] = []
for source in rows:
decisions = by_uid[source["uid"]]
labels = {(d["verdict"], d["event_type"], d["subtype"]) for d in decisions}
labels = {
(d["verdict"], d["event_type"], d["subtype"], d["selected_opportunity_uid"])
for d in decisions
}
unanimous = len(labels) == 1
if unanimous:
verdict, event_type, subtype = next(iter(labels))
verdict, event_type, subtype, selected_opportunity_uid = next(iter(labels))
confidence = min(float(item["confidence"]) for item in decisions)
reason = " | ".join(item["reason"] for item in decisions)
needs_human_review = any(item["needs_human_review"] for item in decisions)
else:
verdict, event_type, subtype = "UNSURE", None, None
verdict, event_type, subtype, selected_opportunity_uid = "UNSURE", None, None, None
confidence = min(float(item["confidence"]) for item in decisions)
reason = "The independent DeepSeek passes disagreed."
needs_human_review = True
opportunity_options = (source.get("temporal_evidence") or {}).get("opportunity_options") or []
selected_opportunity = next(
(
option for option in opportunity_options
if isinstance(option, dict) and option.get("uid") == selected_opportunity_uid
),
None,
)
selected_opportunity_seconds = (
None if selected_opportunity is None else selected_opportunity.get("anchor_seconds")
)
behavior_seconds = (source.get("temporal_evidence") or {}).get(
"behavior_anchor_seconds", source.get("anchor_dense_seconds")
)
enriched = dict(source)
enriched["llm_suggestion"] = {
"verdict": verdict,
"event_type": event_type,
"subtype": subtype,
"selected_opportunity_uid": selected_opportunity_uid,
"selected_opportunity_seconds": selected_opportunity_seconds,
"valid_prediction_interval_seconds": (
None
if selected_opportunity_seconds is None
else [selected_opportunity_seconds, behavior_seconds]
),
"confidence": confidence,
"reason": reason,
"needs_human_review": needs_human_review,
@@ -402,6 +453,8 @@ def add_suggestions(
ai_verdict=verdict,
ai_event_type=event_type,
ai_subtype=subtype,
ai_selected_opportunity_uid=selected_opportunity_uid,
ai_selected_opportunity_seconds=selected_opportunity_seconds,
ai_confidence=confidence,
ai_reason=reason,
human_status="UNREVIEWED",