wx44wx/AI-VideoAssistant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
260 Commits 2 Branches 0 Tags
57264ad83170f8b5c12e5ab3f12ba83d47e0067f
Commit Graph

130 Commits

Author SHA1 Message Date
Xin Wang
bfe165daae Add DashScope ASR model support and enhance related components 
- Introduced DashScope as a new ASR model in the database initialization.
- Updated ASRModel schema to include vendor information.
- Enhanced ASR router to support DashScope-specific functionality, including connection testing and preview capabilities.
- Modified frontend components to accommodate DashScope as a selectable vendor with appropriate default settings.
- Added tests to validate DashScope ASR model creation, updates, and connectivity.
- Updated backend API to handle DashScope-specific base URLs and vendor normalization.
 2026-03-09 07:37:00 +08:00
Xin Wang
da38157638 Add ASR interim results support in Assistant model and API 
- Introduced `asr_interim_enabled` field in the Assistant model to control interim ASR results.
- Updated AssistantBase and AssistantUpdate schemas to include the new field.
- Modified the database schema to add the `asr_interim_enabled` column.
- Enhanced runtime metadata to reflect interim ASR settings.
- Updated API endpoints and tests to validate the new functionality.
- Adjusted documentation to include details about interim ASR results configuration.
 2026-03-06 12:58:54 +08:00
Xin Wang
eecde9f0fb Integrate React Query for data management and enhance Debug Preferences 
- Added React Query for managing API calls related to assistants and voices.
- Introduced `useAssistantsQuery` and `useVoicesQuery` hooks for fetching data.
- Implemented mutations for creating, updating, and deleting voices using React Query.
- Integrated a global `QueryClient` for managing query states and configurations.
- Refactored components to utilize the new query hooks, improving data handling and performance.
- Added a Zustand store for managing debug preferences, including WebSocket URL and audio settings.
 2026-03-02 22:50:57 +08:00
Xin Wang
70b4043f9b Enhance DebugDrawer to support voice prompts in text prompt dialogs 
- Added `promptType` and `voiceText` properties to `DebugTextPromptDialogState`.
- Updated state management for text prompt dialogs to handle voice prompts.
- Modified dialog activation logic to play voice prompts when applicable.
- Adjusted UI to reflect the type of prompt being displayed (text or voice).
- Ensured proper handling of prompt closure messages based on prompt type.
 2026-03-02 15:10:03 +08:00
Xin Wang
fb017f9952 Refactor selectedToolSchemas logic in DebugDrawer to simplify tool ID normalization. Removed redundant inclusion of DEBUG_CLIENT_TOOLS, enhancing code clarity and performance. 2026-03-02 12:40:00 +08:00
Xin Wang
00b88c5afa Add manual opener tool calls to Assistant model and API 
- Introduced `manual_opener_tool_calls` field in the Assistant model to support custom tool calls.
- Updated AssistantBase and AssistantUpdate schemas to include the new field.
- Implemented normalization and migration logic for handling manual opener tool calls in the API.
- Enhanced runtime metadata to include manual opener tool calls in responses.
- Updated tests to validate the new functionality and ensure proper handling of tool calls.
- Refactored tool ID normalization to support legacy tool names for backward compatibility.
 2026-03-02 12:34:42 +08:00
Xin Wang
4d553de34d Refactor assistant greeting logic to conditionally use system prompt for generated openers. Update related tests to verify new behavior and ensure correct metadata handling in API responses. Enhance UI to reflect changes in opener management based on generated opener settings. 2026-03-02 02:38:45 +08:00
Xin Wang
31b3969b96 Enhance ToolLibrary by adding sourceKey to ToolParameterDraft and updating related functions for improved schema management. Introduce normalization functions for object schemas and defaults, and refactor buildToolParameterConfig to utilize these enhancements. Update state management in ToolLibraryPage to accommodate new schema handling and defaults integration. 2026-03-02 02:18:28 +08:00
Xin Wang
3626297211 Implement schema editor functionality in ToolLibrary, allowing users to manage tool parameters with JSON schema validation. Add a drawer for schema editing, enhance state management for schema-related errors, and integrate schema defaults into tool parameter configuration. Update UI to include a button for opening the schema drawer. 2026-03-02 01:54:54 +08:00
Xin Wang
1561056a3d Add voice_choice_prompt and text_choice_prompt tools to API and UI. Implement state management and parameter definitions for user selection prompts, enhancing user interaction and experience. 2026-03-02 00:49:31 +08:00
Xin Wang
3a5d27d6c3 Implement runtime configuration debugging in DebugDrawer by adding a new function to format session metadata and WebSocket configuration. Update the display logic to enhance clarity and user experience, including renaming UI elements for better context. 2026-03-01 23:14:08 +08:00
Xin Wang
6a46ec69f4 Enhance WebSocket session management by requiring assistant_id as a query parameter for connection. Update API reference documentation to reflect changes in message flow and metadata validation rules, including the introduction of whitelists for allowed metadata fields and restrictions on sensitive keys. Refactor client examples to align with the new session initiation process. 2026-03-01 14:10:38 +08:00
Xin Wang
8b59569b99 Add choice_prompt tool to Assistants and DebugDrawer. Implement state management for choice prompts, including normalization of options and handling user selections. Enhance UI interactions for improved user experience. 2026-02-28 10:39:33 +08:00
Xin Wang
e40899613f Refactor DebugDrawer to use a ref for text prompt dialog state management. Update closeTextPromptDialog function to utilize the ref for improved handling of dialog state, ensuring consistent behavior during interactions. 2026-02-27 18:22:45 +08:00
Xin Wang
915d2f4bd8 Refactor DebugDrawer to utilize a ref for client tool state management. Introduce isClientToolEnabledLive function for real-time tool state checks, ensuring accurate handling of tool enablement in debug settings. 2026-02-27 18:10:07 +08:00
Xin Wang
4d9f083e20 Add Switch component to UI and integrate it into DebugDrawer for tool state management. Update Assistants page to utilize the new Switch for enabling/disabling tools, enhancing user interaction and component functionality. 2026-02-27 17:55:35 +08:00
Xin Wang
531cf6080a Update DuplexPipeline tool wait timeout to 60 seconds and modify DebugDrawer to improve tool call ID handling. Ensure better integration and functionality across components. 2026-02-27 17:38:36 +08:00
Xin Wang
229243e832 Add wait_for_response functionality to ToolResource and related components. Update API models, schemas, and routers to support new parameter. Enhance UI components to manage wait_for_response state, ensuring proper integration across the application. 2026-02-27 16:54:39 +08:00
Xin Wang
95c6e93a9c Add text_msg_prompt tool to DuplexPipeline and Assistants. Update DebugDrawer to handle text message prompts, including parameter validation and state management for displaying messages. Ensure integration with existing tools and maintain functionality across components. 2026-02-27 16:47:49 +08:00
Xin Wang
cdd8275e35 Add voice_message_prompt tool to API and UI components. Update DuplexPipeline, Assistants, and DebugDrawer to support new tool functionality, including parameter validation and speech synthesis integration. Ensure existing tools are preserved during seeding process in the database. 2026-02-27 16:04:49 +08:00
Xin Wang
b035e023c4 Implement runtime tool ID and display name mapping in DuplexPipeline. Enhance Assistants and ToolLibrary components to utilize new mappings for improved tool identification and display. Update DebugDrawer to reflect changes in tool display names during interactions. 2026-02-27 15:50:43 +08:00
Xin Wang
0f1165af64 Enhance Dialog component to accept contentClassName prop for customizable styling. Update ToolLibraryPage to utilize new prop for improved modal layout. Refactor text in ToolLibrary for clarity. 2026-02-27 15:12:01 +08:00
Xin Wang
487634c494 Implement tool parameter management in ToolLibrary, including types, default value handling, and schema generation. Introduce functions for creating, updating, and removing tool parameters, enhancing the configuration capabilities for tools. Update state management to reflect new parameter structure. 2026-02-27 15:04:52 +08:00
Xin Wang
5f768edf68 Add parameter schema and defaults to ToolResource model and schemas. Implement runtime tool resolution in assistants and tools routers, ensuring proper handling of tool parameters. Update tests to validate new functionality and ensure correct integration of parameter handling in the API. 2026-02-27 14:44:28 +08:00
Xin Wang
d942c85eff Add new tools to DuplexPipeline: calculator, code_interpreter, turn_on_camera, turn_off_camera, increase_volume, and decrease_volume. Implement fallback schema for unknown string tools and assign default client executors for specific tools. Update tests to validate new functionality and ensure correct tool handling in the pipeline. 2026-02-27 13:59:37 +08:00
Xin Wang
8d453e10da Refactor AssistantsPage to utilize createPortal for template suggestion dropdowns, enhancing rendering performance and ensuring correct positioning relative to the document body. This change improves user experience by maintaining dropdown visibility during text input. 2026-02-27 13:27:13 +08:00
Xin Wang
e108ee66cb Enhance AssistantsPage template suggestion functionality by adding control parameter to updateTemplateSuggestionState. This allows for dynamic positioning of suggestion dropdowns based on caret position, improving user experience during text input. Update relevant event handlers to pass control element for accurate suggestion placement. 2026-02-27 12:18:48 +08:00
Xin Wang
6178cc05bb Add system-level dynamic variables support in session management. Implement methods to generate and apply built-in variables for current session time, UTC time, and timezone. Update documentation to reflect new variables and enhance tests for dynamic variable handling in the UI components. 2026-02-27 12:08:18 +08:00
Xin Wang
71cbfa2b48 Enhance DuplexPipeline and AssistantsPage for improved interruption handling. Introduce _OPENER_PRE_ROLL_MS constant for a head start on mic capture, and adjust interruption sensitivity settings from 500ms to 180ms across relevant components to optimize responsiveness during assistant interactions. 2026-02-27 11:51:15 +08:00
Xin Wang
6a9b5fcff4 Add error handling for dynamic variables in DebugDrawer component. Introduce state for dynamic variable errors and implement validation logic to manage required keys. Update methods to reset error state and handle errors during dynamic variable operations, enhancing user feedback and session management. 2026-02-27 11:39:53 +08:00
Xin Wang
3272a7a68a Add dynamic variables support in session management and UI components. Implement validation rules for dynamic variables in metadata, including key format and value constraints. Enhance session start handling to manage dynamic variable errors. Update documentation and tests to reflect new functionality. 2026-02-27 11:21:37 +08:00
Xin Wang
0f02de5fc3 Update AssistantsPage to include new icon for audio preview button and adjust text for clarity. Add square icon for stop audio preview button to enhance UI consistency. 2026-02-26 16:35:12 +08:00
Xin Wang
0de6fe529e Add audio preview functionality for assistant opener audio in AssistantsPage. Implement controls for previewing and stopping audio playback, and integrate new API endpoint for fetching PCM buffer. Enhance user interface with updated button states for audio actions. 2026-02-26 16:15:31 +08:00
Xin Wang
fb95e2abe2 Add opener audio functionality to Assistant model and related schemas, enabling audio generation and playback features. Update API routes and frontend components to support opener audio management, including status retrieval and generation controls. 2026-02-26 14:31:50 +08:00
Xin Wang
833cb0d4c4 Add unsaved changes confirmation dialog in AssistantsPage to enhance user experience when opening debug window 2026-02-26 14:17:01 +08:00
Xin Wang
fbbb2e0fee Add assistant snapshot management to track unsaved changes and enhance debug handling 2026-02-26 14:13:47 +08:00
Xin Wang
37b646186d Refactor DebugDrawer to store submitted session metadata and update resolved config view with merged metadata 2026-02-26 11:21:34 +08:00
Xin Wang
1bcf625f86 Enhance runtime metadata fetching by including assistant and app IDs, and defaulting channel to 'web_debug' 2026-02-26 11:04:54 +08:00
Xin Wang
f77f7c7531 Voice library support dashscope 2026-02-26 03:54:52 +08:00
Xin Wang
6744646390 Update frontend debug drawer 2026-02-26 02:23:23 +08:00
Xin Wang
56f8aa2191 Fix talking voice error 2026-02-12 19:39:26 +08:00
Xin Wang
81ed89b84f Vendor can show more 2026-02-12 19:29:24 +08:00
Xin Wang
3c7efce80b Consistent library UI 2026-02-12 19:23:30 +08:00
Xin Wang
ff3a03b1ad Use openai compatible as vendor 2026-02-12 18:44:55 +08:00
Xin Wang
0f9543d8a4 Voice add interface change 2026-02-12 18:01:05 +08:00
Xin Wang
39bcd67eac Update voice vad ui 2026-02-12 15:31:45 +08:00
Xin Wang
82521e7b90 Update assistants setting ui 2026-02-12 15:28:10 +08:00
Xin Wang
edcbc2cec7 Add first turn option 2026-02-12 15:23:32 +08:00
Xin Wang
56ca95c200 Improve UI 2026-02-12 15:09:25 +08:00
Xin Wang
a7ef8858de Fix frontend opener showing 2026-02-12 14:46:16 +08:00