Add manuscript-aligned EasyCom EOT INT review workflow
This commit is contained in:
97
docs/easycom_event_review_tool.md
Normal file
97
docs/easycom_event_review_tool.md
Normal file
@@ -0,0 +1,97 @@
|
||||
# EasyCom EOT/INT review workspace
|
||||
|
||||
`review_floor_events.py` is a local annotation workspace for reviewing the
|
||||
manuscript-aligned regions produced by `prepare_easycom_eot_int_review.py`.
|
||||
It can also open the older timing/transcript proxy reports from
|
||||
`derive_easycom_interaction_events.py` and `audit_easycom_floor_events.py`, but
|
||||
those older reports do not have the directionality or hard-negative region
|
||||
contract required by the current manuscript. Raw EasyCom media stays read-only;
|
||||
human decisions are stored separately as JSONL and CSV.
|
||||
|
||||
## Prepare manuscript-aligned candidates
|
||||
|
||||
Develop the annotation guide on training-side Sessions 4--12:
|
||||
|
||||
```bash
|
||||
python3 prepare_easycom_eot_int_review.py \
|
||||
--dataset-root /path/to/easycom \
|
||||
--sessions Session_4 Session_5 Session_6 Session_7 Session_8 Session_9 Session_10 Session_11 Session_12 \
|
||||
--output-dir reports/easycom_floor_event_candidates
|
||||
```
|
||||
|
||||
For a one-event pilot, add `--max-candidates 1`. The generator follows the
|
||||
manuscript direction exactly:
|
||||
|
||||
- EOT-response begins at an aggregate-other offset while the wearer is silent.
|
||||
- INT-yield begins at an aggregate-other onset while the wearer is active.
|
||||
- An observed wearer transition before the competing other transition is a
|
||||
timing-positive candidate.
|
||||
- A competing other transition, or a fully observed no-transition window, is a
|
||||
timing hard-negative candidate.
|
||||
- A source boundary without a complete outcome/window is marked for exclusion.
|
||||
|
||||
Automatic labels remain candidates until independent human semantic review.
|
||||
|
||||
## Optional DeepSeek suggestion
|
||||
|
||||
DeepSeek suggestions are triage metadata and never count as a human label. The
|
||||
script sends only the candidate transcript excerpt and timing fields. Inspect
|
||||
the exact request without network access first:
|
||||
|
||||
```bash
|
||||
python3 adjudicate_easycom_eot_int_candidates.py \
|
||||
--input reports/easycom_floor_event_candidates/easycom_eot_int_candidates.jsonl \
|
||||
--output reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
|
||||
--model deepseek-v4-flash \
|
||||
--max-events 1 \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
After setting `DEEPSEEK_API_KEY`, make the authorized request:
|
||||
|
||||
```bash
|
||||
python3 adjudicate_easycom_eot_int_candidates.py \
|
||||
--input reports/easycom_floor_event_candidates/easycom_eot_int_candidates.jsonl \
|
||||
--output reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
|
||||
--model deepseek-v4-flash \
|
||||
--max-events 1 \
|
||||
--allow-external-transcript-upload
|
||||
```
|
||||
|
||||
Point the review server at the DeepSeek-enriched JSONL to show its suggestion
|
||||
beside the video. Keep suggestions hidden from an initial independent human
|
||||
pass if anchoring bias is a concern.
|
||||
|
||||
## Start the local workspace
|
||||
|
||||
```bash
|
||||
python3 review_floor_events.py \
|
||||
--dataset-root /path/to/easycom \
|
||||
--events reports/easycom_floor_event_candidates/easycom_eot_int_candidates.jsonl \
|
||||
--output reports/easycom_event_reviews/reviewer_1.jsonl
|
||||
```
|
||||
|
||||
Open `http://127.0.0.1:8765`. The page resolves the source MP4 and transcript,
|
||||
seeks to a six-second event context, and supports session/split/task/status
|
||||
filtering.
|
||||
|
||||
Each save atomically rewrites both JSONL and CSV. The review explicitly records
|
||||
`positive`, `hard_negative`, `exclude`, or `uncertain`; the hard-negative
|
||||
subtype; boundary validity; onset function; conversational routing; offset
|
||||
outcome; three 200 ms anchor/region corrections; confidence; and reviewer ID.
|
||||
|
||||
## Freeze discipline
|
||||
|
||||
Develop the annotation guide and resolve category disagreements on training
|
||||
Sessions 4--12. Freeze the guide before reviewing official test Sessions 1--3.
|
||||
Do not use test reviews to change candidate windows, label categories, or the
|
||||
R=.8 threshold calibration rule.
|
||||
|
||||
For paper-quality labels, use two independent reviewers with separate
|
||||
`--output` files and adjudicate disagreements in a third output. A non-empty
|
||||
reviewer ID is required on every saved row. DeepSeek is not one of the two
|
||||
reviewers.
|
||||
|
||||
This workspace produces audited labels. It does not yet convert them into the
|
||||
final committed-event matching metrics; that integration follows only after
|
||||
the schema and real-data pilot are frozen.
|
||||
Reference in New Issue
Block a user