Xin Wang
935f2fbd1f
Refactor assistant configuration management and update documentation
...
- Removed legacy agent profile settings from the .env.example and README, streamlining the configuration process.
- Introduced a new local YAML configuration adapter for assistant settings, allowing for easier management of assistant profiles.
- Updated backend integration documentation to clarify the behavior of assistant config sourcing based on backend URL settings.
- Adjusted various service implementations to directly utilize API keys from the new configuration structure.
- Enhanced test coverage for the new local YAML adapter and its integration with backend services.
2026-03-05 21:24:15 +08:00
Xin Wang
7d4af18815
Add output.audio.played message handling and update documentation
...
- Introduced `output.audio.played` message type for client acknowledgment of audio playback completion.
- Updated `DuplexPipeline` to track client playback state and handle playback completion events.
- Enhanced session handling to route `output.audio.played` messages to the pipeline.
- Revised API documentation to include details about the new message type and its fields.
- Updated schema documentation to reflect the addition of `output.audio.played` in the message flow.
2026-03-04 10:01:34 +08:00
Xin Wang
3aa9e0f432
Enhance DuplexPipeline to support follow-up context for manual opener tool calls
...
- Introduced logic to trigger a follow-up turn when the manual opener greeting is empty.
- Updated `_execute_manual_opener_tool_calls` to return structured tool call and result data.
- Added `_build_manual_opener_follow_up_context` method to construct context for follow-up turns.
- Modified `_handle_turn` to accept system context for improved conversation management.
- Enhanced tests to validate the new follow-up behavior and ensure proper context handling.
2026-03-02 14:27:44 +08:00
Xin Wang
00b88c5afa
Add manual opener tool calls to Assistant model and API
...
- Introduced `manual_opener_tool_calls` field in the Assistant model to support custom tool calls.
- Updated AssistantBase and AssistantUpdate schemas to include the new field.
- Implemented normalization and migration logic for handling manual opener tool calls in the API.
- Enhanced runtime metadata to include manual opener tool calls in responses.
- Updated tests to validate the new functionality and ensure proper handling of tool calls.
- Refactored tool ID normalization to support legacy tool names for backward compatibility.
2026-03-02 12:34:42 +08:00
Xin Wang
b5cdb76e52
Implement initial generated opener logic in DuplexPipeline to utilize tool-capable assistant turns when tools are available. Update tests to verify the correct behavior of the generated opener under various conditions, ensuring proper handling of user input and task management.
2026-03-02 02:47:30 +08:00
Xin Wang
4d553de34d
Refactor assistant greeting logic to conditionally use system prompt for generated openers. Update related tests to verify new behavior and ensure correct metadata handling in API responses. Enhance UI to reflect changes in opener management based on generated opener settings.
2026-03-02 02:38:45 +08:00
Xin Wang
1561056a3d
Add voice_choice_prompt and text_choice_prompt tools to API and UI. Implement state management and parameter definitions for user selection prompts, enhancing user interaction and experience.
2026-03-02 00:49:31 +08:00
Xin Wang
aae41d4512
Clear stale ASR capture on end of utterance in DuplexPipeline. Add test to verify behavior when conversation state changes, ensuring proper handling of ASR capture variables.
2026-02-28 12:32:35 +08:00
Xin Wang
531cf6080a
Update DuplexPipeline tool wait timeout to 60 seconds and modify DebugDrawer to improve tool call ID handling. Ensure better integration and functionality across components.
2026-02-27 17:38:36 +08:00
Xin Wang
229243e832
Add wait_for_response functionality to ToolResource and related components. Update API models, schemas, and routers to support new parameter. Enhance UI components to manage wait_for_response state, ensuring proper integration across the application.
2026-02-27 16:54:39 +08:00
Xin Wang
95c6e93a9c
Add text_msg_prompt tool to DuplexPipeline and Assistants. Update DebugDrawer to handle text message prompts, including parameter validation and state management for displaying messages. Ensure integration with existing tools and maintain functionality across components.
2026-02-27 16:47:49 +08:00
Xin Wang
cdd8275e35
Add voice_message_prompt tool to API and UI components. Update DuplexPipeline, Assistants, and DebugDrawer to support new tool functionality, including parameter validation and speech synthesis integration. Ensure existing tools are preserved during seeding process in the database.
2026-02-27 16:04:49 +08:00
Xin Wang
b035e023c4
Implement runtime tool ID and display name mapping in DuplexPipeline. Enhance Assistants and ToolLibrary components to utilize new mappings for improved tool identification and display. Update DebugDrawer to reflect changes in tool display names during interactions.
2026-02-27 15:50:43 +08:00
Xin Wang
5f768edf68
Add parameter schema and defaults to ToolResource model and schemas. Implement runtime tool resolution in assistants and tools routers, ensuring proper handling of tool parameters. Update tests to validate new functionality and ensure correct integration of parameter handling in the API.
2026-02-27 14:44:28 +08:00
Xin Wang
d942c85eff
Add new tools to DuplexPipeline: calculator, code_interpreter, turn_on_camera, turn_off_camera, increase_volume, and decrease_volume. Implement fallback schema for unknown string tools and assign default client executors for specific tools. Update tests to validate new functionality and ensure correct tool handling in the pipeline.
2026-02-27 13:59:37 +08:00
Xin Wang
71cbfa2b48
Enhance DuplexPipeline and AssistantsPage for improved interruption handling. Introduce _OPENER_PRE_ROLL_MS constant for a head start on mic capture, and adjust interruption sensitivity settings from 500ms to 180ms across relevant components to optimize responsiveness during assistant interactions.
2026-02-27 11:51:15 +08:00
Xin Wang
f1b60bef22
Update ASR delta throttle timing in DuplexPipeline from 300ms to 500ms to improve processing efficiency and responsiveness.
2026-02-27 10:23:06 +08:00
Xin Wang
403b4b93c7
Add ASR capture timeout handling in DuplexPipeline and enhance EOU detection logic. Introduce _ASR_CAPTURE_MAX_MS constant and manage capture state timing to ensure timely end of utterance processing, even during silence. Update EouDetector to allow silence-only EOU when VAD state is lost.
2026-02-27 09:59:54 +08:00
Xin Wang
fb95e2abe2
Add opener audio functionality to Assistant model and related schemas, enabling audio generation and playback features. Update API routes and frontend components to support opener audio management, including status retrieval and generation controls.
2026-02-26 14:31:50 +08:00
Xin Wang
da83c8ec8a
Implement initial greeting emission in DuplexPipeline after session activation, ensuring proper event ordering for frontend notifications.
2026-02-26 14:07:46 +08:00
Xin Wang
562341a72c
add dashscope tts
2026-02-26 03:02:48 +08:00
Xin Wang
72ed7d0512
Unify db api
2026-02-26 01:58:39 +08:00
Xin Wang
ff3a03b1ad
Use openai compatible as vendor
2026-02-12 18:44:55 +08:00
Xin Wang
35bd83767e
Cleanup engine
2026-02-12 17:42:21 +08:00
Xin Wang
aabf2ce8b9
Fix asr begin error
2026-02-12 16:52:42 +08:00
Xin Wang
543528239e
Tune engine vad config
2026-02-12 16:29:55 +08:00
Xin Wang
a92a56b845
Presist opener to history
2026-02-12 15:59:36 +08:00
Xin Wang
399c9c97b1
Add tool call log
2026-02-12 15:44:01 +08:00
Xin Wang
edcbc2cec7
Add first turn option
2026-02-12 15:23:32 +08:00
Xin Wang
cbebfe1c7a
Fix opener not trigger when tts disabled
2026-02-12 14:55:03 +08:00
Xin Wang
a7ef8858de
Fix frontend opener showing
2026-02-12 14:46:16 +08:00
Xin Wang
ef13ddb6b2
Text drawer use generated opener
2026-02-12 14:40:22 +08:00
Xin Wang
d41db6418c
Add bot not interrupt and generated opener
2026-02-12 13:51:27 +08:00
Xin Wang
80e1d24443
Organize tool scheme
2026-02-11 11:22:56 +08:00
Xin Wang
9304927fe9
Import tool setting
2026-02-11 11:04:05 +08:00
Xin Wang
180a69ca67
Fix delta final priority problem in backend
2026-02-11 10:25:46 +08:00
Xin Wang
15523d9ec2
Add tts/text output schema
2026-02-11 09:50:46 +08:00
Xin Wang
2d7fc2b700
Add server tool timeout protection
2026-02-10 19:17:45 +08:00
Xin Wang
6cac24918d
Now we have server tool and client tool
2026-02-10 19:13:54 +08:00
Xin Wang
4b8da32787
Update code for tool call
2026-02-10 16:28:20 +08:00
Xin Wang
539cf2fda2
Move rag to llm service
2026-02-10 13:47:08 +08:00
Xin Wang
6b4391c423
Implement KB features with codex
2026-02-10 07:35:08 +08:00
Xin Wang
6a42c47700
Prettier EOU log
2026-02-09 19:56:47 +08:00
Xin Wang
3537a865f7
Merge branch 'master' of https://gitea.xiaowang.eu.org/wx44wx/AI-VideoAssistant
2026-02-09 19:18:24 +08:00
Xin Wang
b3e1c3d380
Use priority queue for vad event
2026-02-09 19:17:50 +08:00
Xin Wang
b34d500479
Merge branch 'master' of https://gitea.xiaowang.eu.org/wx44wx/AI-VideoAssistant
2026-02-09 18:29:56 +08:00
Xin Wang
0135f718f3
Fix asr boundary bug
2026-02-09 18:28:43 +08:00
Xin Wang
29d0b931eb
better asr log
2026-02-09 18:22:00 +08:00
Xin Wang
b57d9f3a65
Fix race asr problem
2026-02-09 18:01:40 +08:00
Xin Wang
088943166c
Update asr interim
2026-02-09 17:24:15 +08:00
