Enhance evaluation runner with transcript handling and new case formats

- Updated the evaluation runner to support live dialogue printing and transcript saving in both Markdown and JSON formats.
- Introduced new data classes for managing transcripts and their components, improving structure and readability.
- Modified existing YAML case files to align with the new direct client mode and updated stage codes.
- Added new test cases for the direct client mode, ensuring comprehensive coverage of user interactions.
This commit is contained in:
Xin Wang
2026-07-29 11:13:07 +08:00
parent e6437ab332
commit 4e621e2f0c
5 changed files with 406 additions and 34 deletions

View File

@@ -2,27 +2,29 @@
The evaluation runner calls a live FastAPI service backed by FastGPT. For now,
each conversation step checks only the SSE `stage_code.nextStageCode` value.
During the run it also prints the live dialogue and saves transcripts.
## Case format
Add YAML files anywhere under `evals/cases/`:
```yaml
name: browser_addon_example
clientMode: browser_addon
name: direct_initial_stage
clientMode: direct
steps:
- input: hi
- input: 新对话
expect:
stageCode: browser_addon.1001
stageCode: "1001"
- input: 继续
- input: 继续办理
expect:
stageCode: browser_addon.1002
stageCode: "1002"
```
Steps in one file share the same generated `sessionId`. Different cases always
use different session IDs.
`clientMode` in the case file is the API request setting (`direct` or
`browser_addon`). Steps in one file share the same generated `sessionId`.
Different cases always use different session IDs.
## Run
@@ -41,8 +43,32 @@ python evals/run.py --base-url http://192.168.1.10:8000
Run selected cases:
```bash
python evals/run.py --case direct
python evals/run.py --case browser_addon
```
## Dialogue and transcripts
While a case runs, each step prints the user input, bot reply (`text_delta`),
and stage-code check result.
By default, after the run finishes, transcripts are written under
`evals/results/<timestamp>/`:
- `*.md` — readable conversation log
- `*.json` — structured step results
Save to a specific directory:
```bash
python evals/run.py --output-dir evals/results/manual-run --case direct
```
Print live dialogue only, without writing files:
```bash
python evals/run.py --no-save --case direct
```
The process exits with code `1` when an expectation fails and code `2` when
case configuration is invalid.