2.3 KiB
EasyCom event AI pre-label audit
These are DeepSeek pre-labels awaiting mandatory human adjudication, not ground truth and not final manuscript annotation counts.
Inputs contain 2,128 unique candidates from Session_1, Session_2, Session_3, Session_4. Protocol: easycom-wearer-entry-int-candidates-v8; prompt: easycom-wearer-boundary-semantic-prelabel-v15.
Cohort overview
| Cohort | Candidates | Positive | Hard negative | Exclude | Prelabel scoring-eligible | Two-pass semantic/full agreement | Human reviewed |
|---|---|---|---|---|---|---|---|
| Session_1 | 496 | 116 | 182 | 198 | 298 (60.08%) | 458/496 (92.34%) / 453/496 (91.33%) | 0 |
| Session_2 | 490 | 71 | 252 | 167 | 323 (65.92%) | 454/490 (92.65%) / 452/490 (92.24%) | 0 |
| Session_3 | 545 | 117 | 184 | 244 | 301 (55.23%) | 489/545 (89.72%) / 486/545 (89.17%) | 0 |
| Session_4 | 597 | 84 | 381 | 132 | 465 (77.89%) | 575/597 (96.31%) / 571/597 (95.64%) | 0 |
| Test pooled (Sessions 1-3) | 1531 | 304 | 618 | 609 | 922 (60.22%) | 1401/1531 (91.51%) / 1391/1531 (90.86%) | 0 |
| Session 4 audit (train split) | 597 | 84 | 381 | 132 | 465 (77.89%) | 575/597 (96.31%) / 571/597 (95.64%) | 0 |
| All supplied (descriptive only) | 2128 | 388 | 999 | 741 | 1387 (65.18%) | 1976/2128 (92.86%) / 1962/2128 (92.20%) | 0 |
The official EasyCom test sessions (1--3) are pooled only for descriptive reporting; these AI pre-labels must not be used to tune thresholds, taxonomy, prompts, or model choices. Session 4 is shown separately as a train-split audit and is not the complete Sessions 4--12 training corpus.
Detailed counts are in taxonomy_counts.csv and candidate_source_counts.csv. manuscript_ai_prelabel_table.tex mirrors the manuscript row order but is deliberately labeled as an AI audit; retain the manuscript TODOs until human adjudication is frozen.
Reproduce
python3 summarize_easycom_event_prelabels.py \
--input reports/easycom_entry_int_session1_v14/easycom_eot_int_deepseek_official.jsonl \
--input reports/easycom_entry_int_session2_v14/easycom_eot_int_deepseek_official.jsonl \
--input reports/easycom_entry_int_session3_v14/easycom_eot_int_deepseek_official.jsonl \
--input reports/easycom_entry_int_session4_v14/easycom_eot_int_deepseek_official.jsonl \
--output-dir reports/easycom_event_ai_prelabel_stats_v15