Add provisional LLM review materializer
This commit is contained in:
@@ -29,6 +29,44 @@ acoustic boundary can occur anywhere inside a 200 ms target bin, do not
|
||||
reinterpret it as an exact continuous-time lead without separately reporting
|
||||
the quantization convention.
|
||||
|
||||
## Provisional LLM-only pipeline smoke
|
||||
|
||||
Before human adjudication is complete, unanimous two-pass LLM suggestions can
|
||||
be materialized into the review-file schema for a pipeline smoke test:
|
||||
|
||||
```bash
|
||||
python3 materialize_provisional_llm_reviews.py \
|
||||
--events reports/easycom_entry_int_session{1,2,3,4}_v14/easycom_eot_int_deepseek_official*.jsonl \
|
||||
--output reports/easycom_entry_int_sessions1_4_v14/easycom_eot_int_provisional_llm_reviews.jsonl \
|
||||
--acknowledge-provisional-llm-auto-accept
|
||||
```
|
||||
|
||||
The converter skips pass disagreements, pending API failures, and candidates
|
||||
without an LLM suggestion. It writes JSONL, CSV, and a summary JSON. Every
|
||||
materialized row carries `review_source=PROVISIONAL_LLM_AUTO_ACCEPT` and
|
||||
`human_reviewed=false`; the nested `human_status=REVIEWED` value exists only
|
||||
for review-schema compatibility and must never be described as human review.
|
||||
The discouraged `--include-nonunanimous` option is available only for debugging
|
||||
the fallback path.
|
||||
|
||||
The evaluator rejects these rows by default. A provisional run requires an
|
||||
explicit opt-in and marks its output protocol and semantic scope accordingly:
|
||||
|
||||
```bash
|
||||
python3 evaluate_annotated_floor_events.py \
|
||||
--reviews reports/easycom_entry_int_sessions1_4_v14/easycom_eot_int_provisional_llm_reviews.jsonl \
|
||||
--run egospeak=results/egospeak_miniroada_floor_eval \
|
||||
--sessions Session_4 \
|
||||
--allow-provisional-llm-reviews \
|
||||
--print-event-examples 5 \
|
||||
--output results/easycom_egospeak_event_smoke/session4_summary.json
|
||||
```
|
||||
|
||||
This output is suitable for alignment debugging and preliminary analysis, not
|
||||
for a manuscript claim of human-adjudicated ground truth. Official test
|
||||
Sessions 1--3 still require disjoint training-side calibration unless the
|
||||
separate exploratory in-sample override is used.
|
||||
|
||||
## Development smoke run
|
||||
|
||||
Use only training-side sessions while the annotation and region rules are
|
||||
|
||||
Reference in New Issue
Block a user