f40643dd87
- Updated prompts to enhance clarity and consistency in user interactions, particularly focusing on safety prioritization and accurate information collection. - Introduced new YAML evaluation cases to test various user input scenarios, ensuring robust handling of short, ambiguous, or irrelevant responses. - Enhanced guidelines for transitioning to human assistance, emphasizing strict conditions for triggering state changes based on user responses.
76 lines
1.7 KiB
Markdown
76 lines
1.7 KiB
Markdown
# Workflow evaluations
|
|
|
|
The evaluation runner calls a live FastAPI service backed by FastGPT. For now,
|
|
each conversation step checks only the SSE `stage_code.nextStageCode` value.
|
|
During the run it also prints the live dialogue and saves transcripts.
|
|
|
|
## Case format
|
|
|
|
Add YAML files anywhere under `evals/cases/` (e.g. `direct/`, `browser_addon/`, `no_false/` for false-interrupt / false-transfer checks):
|
|
|
|
```yaml
|
|
name: direct_initial_stage
|
|
clientMode: direct
|
|
|
|
steps:
|
|
- input: 新对话
|
|
expect:
|
|
stageCode: "1001"
|
|
|
|
- input: 继续办理
|
|
expect:
|
|
stageCode: "1002"
|
|
```
|
|
|
|
`clientMode` in the case file is the API request setting (`direct` or
|
|
`browser_addon`). Steps in one file share the same generated `sessionId`.
|
|
Different cases always use different session IDs.
|
|
|
|
## Run
|
|
|
|
Start the API, then execute:
|
|
|
|
```bash
|
|
python evals/run.py
|
|
```
|
|
|
|
Use another API address:
|
|
|
|
```bash
|
|
python evals/run.py --base-url http://192.168.1.10:8000
|
|
```
|
|
|
|
Run selected cases:
|
|
|
|
```bash
|
|
python evals/run.py --case direct
|
|
python evals/run.py --case browser_addon
|
|
python evals/run.py --case no_false
|
|
```
|
|
|
|
## Dialogue and transcripts
|
|
|
|
While a case runs, each step prints the user input, bot reply (`text_delta`),
|
|
and stage-code check result.
|
|
|
|
By default, after the run finishes, transcripts are written under
|
|
`evals/results/<timestamp>/`:
|
|
|
|
- `*.md` — readable conversation log
|
|
- `*.json` — structured step results
|
|
|
|
Save to a specific directory:
|
|
|
|
```bash
|
|
python evals/run.py --output-dir evals/results/manual-run --case direct
|
|
```
|
|
|
|
Print live dialogue only, without writing files:
|
|
|
|
```bash
|
|
python evals/run.py --no-save --case direct
|
|
```
|
|
|
|
The process exits with code `1` when an expectation fails and code `2` when
|
|
case configuration is invalid.
|