Move workflow evaluations to standalone runner
This commit is contained in:
48
evals/README.md
Normal file
48
evals/README.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# Workflow evaluations
|
||||
|
||||
The evaluation runner calls a live FastAPI service backed by FastGPT. For now,
|
||||
each conversation step checks only the SSE `stage_code.nextStageCode` value.
|
||||
|
||||
## Case format
|
||||
|
||||
Add YAML files anywhere under `evals/cases/`:
|
||||
|
||||
```yaml
|
||||
name: browser_addon_example
|
||||
clientMode: browser_addon
|
||||
|
||||
steps:
|
||||
- input: hi
|
||||
expect:
|
||||
stageCode: browser_addon.1001
|
||||
|
||||
- input: 继续
|
||||
expect:
|
||||
stageCode: browser_addon.1002
|
||||
```
|
||||
|
||||
Steps in one file share the same generated `sessionId`. Different cases always
|
||||
use different session IDs.
|
||||
|
||||
## Run
|
||||
|
||||
Start the API, then execute:
|
||||
|
||||
```bash
|
||||
python evals/run.py
|
||||
```
|
||||
|
||||
Use another API address:
|
||||
|
||||
```bash
|
||||
python evals/run.py --base-url http://192.168.1.10:8000
|
||||
```
|
||||
|
||||
Run selected cases:
|
||||
|
||||
```bash
|
||||
python evals/run.py --case browser_addon
|
||||
```
|
||||
|
||||
The process exits with code `1` when an expectation fails and code `2` when
|
||||
case configuration is invalid.
|
||||
Reference in New Issue
Block a user