Commit Graph

234 Commits

Author SHA1 Message Date
Xin Wang
526024d603 Enhance assistant configuration documentation with details on persistence and runtime overrides
- Added a new section explaining the two layers of assistant configuration: database persistence and session-level overrides.
- Included a table listing fields that are stored in the database and those that can be overridden during a session.
- Provided code examples demonstrating the merging of baseline configuration with session overrides for clarity.
2026-03-04 10:57:02 +08:00
Xin Wang
a8fa66e9cc Update documentation to reflect changes in WebSocket API message formatting and knowledge base
- Updated the WebSocket API reference to improve clarity by removing unnecessary headings and emphasizing message types.
- Revised the index.md to specify 'chroma' as the knowledge base, enhancing the overview of the platform's architecture.
2026-03-04 10:32:56 +08:00
Xin Wang
aaef370d70 Merge branch 'master' of https://gitea.xiaowang.eu.org/wx44wx/AI-VideoAssistant 2026-03-04 10:01:41 +08:00
Xin Wang
7d4af18815 Add output.audio.played message handling and update documentation
- Introduced `output.audio.played` message type for client acknowledgment of audio playback completion.
- Updated `DuplexPipeline` to track client playback state and handle playback completion events.
- Enhanced session handling to route `output.audio.played` messages to the pipeline.
- Revised API documentation to include details about the new message type and its fields.
- Updated schema documentation to reflect the addition of `output.audio.played` in the message flow.
2026-03-04 10:01:34 +08:00
Xin Wang
530d95eea4 Enhance Docker configuration and update dependencies for Realtime Agent Studio
- Updated Dockerfile for the API to include build tools for C++11 required for native extensions.
- Revised requirements.txt to upgrade several dependencies, including FastAPI and SQLAlchemy.
- Expanded docker-compose.yml to add MinIO service for S3-compatible storage and improved health checks for backend and engine services.
- Enhanced README.md in the Docker directory to provide detailed service descriptions and quick start instructions.
- Updated mkdocs.yml to reflect new navigation structure and added deployment overview documentation.
- Introduced new Dockerfiles for the engine and web services, including development configurations for hot reloading.
2026-03-04 10:01:00 +08:00
Xin Wang
4c05131536 Update documentation and configuration for Realtime Agent Studio
- Revised mkdocs.yml to reflect the new site name and description, enhancing clarity for users.
- Added a changelog.md to document important changes and updates for the project.
- Introduced a roadmap.md to outline development plans and progress for future releases.
- Expanded index.md with a comprehensive overview of the platform, including core features and installation instructions.
- Enhanced concepts documentation with detailed explanations of assistants, engines, and their configurations.
- Updated configuration documentation to provide clear guidance on environment setup and service configurations.
- Added extra JavaScript for improved user experience in the documentation site.
2026-03-02 23:35:22 +08:00
Xin Wang
80fff09b76 Merge branch 'master' of https://gitea.xiaowang.eu.org/wx44wx/AI-VideoAssistant 2026-03-02 22:51:03 +08:00
Xin Wang
eecde9f0fb Integrate React Query for data management and enhance Debug Preferences
- Added React Query for managing API calls related to assistants and voices.
- Introduced `useAssistantsQuery` and `useVoicesQuery` hooks for fetching data.
- Implemented mutations for creating, updating, and deleting voices using React Query.
- Integrated a global `QueryClient` for managing query states and configurations.
- Refactored components to utilize the new query hooks, improving data handling and performance.
- Added a Zustand store for managing debug preferences, including WebSocket URL and audio settings.
2026-03-02 22:50:57 +08:00
Xin Wang
7fbf52078f Update documentation to reflect changes in quickstart navigation and API reference
- Replaced the "通过控制台" and "通过 API" entries in the quickstart section with "资源库配置" for improved clarity.
- Updated the API reference link in index.md to direct users to the main quickstart page instead of the outdated API usage example.
2026-03-02 17:33:32 +08:00
Xin Wang
a003134477 Update documentation to enhance clarity and resource configuration for RAS
- Revised the introduction in index.md to emphasize the need for resource configuration before creating an AI assistant.
- Added a new section detailing the configuration process for ASR, LLM, and TTS resources.
- Updated the quickstart guide to reflect the new resource management steps and included troubleshooting tips for common issues.
- Removed the outdated API guide as it has been integrated into the new resource configuration workflow.
2026-03-02 17:30:48 +08:00
Xin Wang
85315ba6ca Update index.md to clarify RAS's core focus on large voice models
- Revised the description of the Realtime Agent Studio (RAS) to emphasize its foundation on large voice models, enhancing clarity on the platform's capabilities.
2026-03-02 17:01:55 +08:00
Xin Wang
9734b38808 Add task list support and update roadmap in documentation
- Added pymdownx.tasklist extension to mkdocs.yml for enhanced task management.
- Revised the roadmap section in index.md to include additional completed and in-progress tasks, improving project tracking and visibility.
2026-03-02 17:01:24 +08:00
Xin Wang
0a7a3253a6 Add emoji support and enhance documentation in RAS
- Added pymdownx.emoji extension to mkdocs.yml for emoji rendering.
- Updated index.md to include a new dashboard image and revised descriptions for clarity.
- Expanded the features section with detailed descriptions of tools and testing capabilities.
- Introduced a roadmap section outlining completed, in-progress, and to-do features for better project visibility.
2026-03-02 16:50:17 +08:00
Xin Wang
a82100fc79 Merge branch 'master' of https://gitea.xiaowang.eu.org/wx44wx/AI-VideoAssistant 2026-03-02 15:12:04 +08:00
Xin Wang
d0897aca92 Update documentation to reflect rebranding from AI Video Assistant to Realtime Agent Studio (RAS)
- Changed site name and description in mkdocs.yml.
- Revised content in index.md to provide a comprehensive overview of RAS features and capabilities.
- Updated API reference and error documentation to replace AI Video Assistant with RAS.
- Modified deployment and getting started guides to align with the new branding.
- Enhanced quickstart instructions to specify RAS service requirements.
2026-03-02 15:11:33 +08:00
Xin Wang
70b4043f9b Enhance DebugDrawer to support voice prompts in text prompt dialogs
- Added `promptType` and `voiceText` properties to `DebugTextPromptDialogState`.
- Updated state management for text prompt dialogs to handle voice prompts.
- Modified dialog activation logic to play voice prompts when applicable.
- Adjusted UI to reflect the type of prompt being displayed (text or voice).
- Ensured proper handling of prompt closure messages based on prompt type.
2026-03-02 15:10:03 +08:00
Xin Wang
3aa9e0f432 Enhance DuplexPipeline to support follow-up context for manual opener tool calls
- Introduced logic to trigger a follow-up turn when the manual opener greeting is empty.
- Updated `_execute_manual_opener_tool_calls` to return structured tool call and result data.
- Added `_build_manual_opener_follow_up_context` method to construct context for follow-up turns.
- Modified `_handle_turn` to accept system context for improved conversation management.
- Enhanced tests to validate the new follow-up behavior and ensure proper context handling.
2026-03-02 14:27:44 +08:00
Xin Wang
fb017f9952 Refactor selectedToolSchemas logic in DebugDrawer to simplify tool ID normalization. Removed redundant inclusion of DEBUG_CLIENT_TOOLS, enhancing code clarity and performance. 2026-03-02 12:40:00 +08:00
Xin Wang
00b88c5afa Add manual opener tool calls to Assistant model and API
- Introduced `manual_opener_tool_calls` field in the Assistant model to support custom tool calls.
- Updated AssistantBase and AssistantUpdate schemas to include the new field.
- Implemented normalization and migration logic for handling manual opener tool calls in the API.
- Enhanced runtime metadata to include manual opener tool calls in responses.
- Updated tests to validate the new functionality and ensure proper handling of tool calls.
- Refactored tool ID normalization to support legacy tool names for backward compatibility.
2026-03-02 12:34:42 +08:00
Xin Wang
b5cdb76e52 Implement initial generated opener logic in DuplexPipeline to utilize tool-capable assistant turns when tools are available. Update tests to verify the correct behavior of the generated opener under various conditions, ensuring proper handling of user input and task management. 2026-03-02 02:47:30 +08:00
Xin Wang
4d553de34d Refactor assistant greeting logic to conditionally use system prompt for generated openers. Update related tests to verify new behavior and ensure correct metadata handling in API responses. Enhance UI to reflect changes in opener management based on generated opener settings. 2026-03-02 02:38:45 +08:00
Xin Wang
31b3969b96 Enhance ToolLibrary by adding sourceKey to ToolParameterDraft and updating related functions for improved schema management. Introduce normalization functions for object schemas and defaults, and refactor buildToolParameterConfig to utilize these enhancements. Update state management in ToolLibraryPage to accommodate new schema handling and defaults integration. 2026-03-02 02:18:28 +08:00
Xin Wang
3f22e2b875 Merge branch 'master' of https://gitea.xiaowang.eu.org/wx44wx/AI-VideoAssistant 2026-03-02 01:56:47 +08:00
Xin Wang
531688aa6b Enhance API documentation by adding new endpoints for ASR preview, assistant configuration retrieval, and knowledge base management. Update existing assistant and tool definitions for improved clarity and functionality. Remove outdated sections from history records documentation, ensuring a streamlined reference for users. 2026-03-02 01:56:38 +08:00
Xin Wang
3626297211 Implement schema editor functionality in ToolLibrary, allowing users to manage tool parameters with JSON schema validation. Add a drawer for schema editing, enhance state management for schema-related errors, and integrate schema defaults into tool parameter configuration. Update UI to include a button for opening the schema drawer. 2026-03-02 01:54:54 +08:00
Xin Wang
1561056a3d Add voice_choice_prompt and text_choice_prompt tools to API and UI. Implement state management and parameter definitions for user selection prompts, enhancing user interaction and experience. 2026-03-02 00:49:31 +08:00
Xin Wang
3a5d27d6c3 Implement runtime configuration debugging in DebugDrawer by adding a new function to format session metadata and WebSocket configuration. Update the display logic to enhance clarity and user experience, including renaming UI elements for better context. 2026-03-01 23:14:08 +08:00
Xin Wang
3643431565 Enhance WebSocket session configuration by introducing an optional config.resolved event, which provides a public snapshot of the session's configuration. Update the API reference documentation to clarify the conditions under which this event is emitted and the details it includes. Modify session management to respect the new setting for emitting configuration details, ensuring sensitive information remains secure. Update tests to validate the new behavior and ensure compliance with the updated configuration schema. 2026-03-01 23:08:44 +08:00
Xin Wang
2418df80e5 Revamp documentation structure in mkdocs.yml by reorganizing navigation for improved accessibility. Remove outdated content from previous sections and introduce new topics including detailed guides on assistant management, configuration options, and tool integrations. Enhance API reference documentation with comprehensive error codes and WebSocket protocol details. Add new sections for automated testing, data analysis, and knowledge base management, ensuring a cohesive and user-friendly documentation experience. 2026-03-01 22:38:50 +08:00
Xin Wang
6a46ec69f4 Enhance WebSocket session management by requiring assistant_id as a query parameter for connection. Update API reference documentation to reflect changes in message flow and metadata validation rules, including the introduction of whitelists for allowed metadata fields and restrictions on sensitive keys. Refactor client examples to align with the new session initiation process. 2026-03-01 14:10:38 +08:00
Xin Wang
b4fa664d73 Refactor WebSocket authentication handling by removing auth requirements from the hello message. Update related documentation and schemas to reflect the changes in authentication strategy, simplifying the connection process. 2026-02-28 17:33:40 +08:00
Xin Wang
0821d73e7c Add API reference documentation for WebSocket communication. Update mkdocs.yml to include new API reference section. 2026-02-28 14:37:58 +08:00
Xin Wang
a7da109983 Merge branch 'master' of https://gitea.xiaowang.eu.org/wx44wx/AI-VideoAssistant 2026-02-28 12:33:23 +08:00
Xin Wang
aae41d4512 Clear stale ASR capture on end of utterance in DuplexPipeline. Add test to verify behavior when conversation state changes, ensuring proper handling of ASR capture variables. 2026-02-28 12:32:35 +08:00
Xin Wang
c4c473105e Add start-dev.ps1 script to automate the launch of development services in the pycall conda environment. The script initiates the API, Web, and Engine services in separate PowerShell windows, enhancing the development workflow. 2026-02-28 11:26:52 +08:00
Xin Wang
8b59569b99 Add choice_prompt tool to Assistants and DebugDrawer. Implement state management for choice prompts, including normalization of options and handling user selections. Enhance UI interactions for improved user experience. 2026-02-28 10:39:33 +08:00
Xin Wang
e40899613f Refactor DebugDrawer to use a ref for text prompt dialog state management. Update closeTextPromptDialog function to utilize the ref for improved handling of dialog state, ensuring consistent behavior during interactions. 2026-02-27 18:22:45 +08:00
Xin Wang
915d2f4bd8 Refactor DebugDrawer to utilize a ref for client tool state management. Introduce isClientToolEnabledLive function for real-time tool state checks, ensuring accurate handling of tool enablement in debug settings. 2026-02-27 18:10:07 +08:00
Xin Wang
4d9f083e20 Add Switch component to UI and integrate it into DebugDrawer for tool state management. Update Assistants page to utilize the new Switch for enabling/disabling tools, enhancing user interaction and component functionality. 2026-02-27 17:55:35 +08:00
Xin Wang
531cf6080a Update DuplexPipeline tool wait timeout to 60 seconds and modify DebugDrawer to improve tool call ID handling. Ensure better integration and functionality across components. 2026-02-27 17:38:36 +08:00
Xin Wang
229243e832 Add wait_for_response functionality to ToolResource and related components. Update API models, schemas, and routers to support new parameter. Enhance UI components to manage wait_for_response state, ensuring proper integration across the application. 2026-02-27 16:54:39 +08:00
Xin Wang
95c6e93a9c Add text_msg_prompt tool to DuplexPipeline and Assistants. Update DebugDrawer to handle text message prompts, including parameter validation and state management for displaying messages. Ensure integration with existing tools and maintain functionality across components. 2026-02-27 16:47:49 +08:00
Xin Wang
cdd8275e35 Add voice_message_prompt tool to API and UI components. Update DuplexPipeline, Assistants, and DebugDrawer to support new tool functionality, including parameter validation and speech synthesis integration. Ensure existing tools are preserved during seeding process in the database. 2026-02-27 16:04:49 +08:00
Xin Wang
b035e023c4 Implement runtime tool ID and display name mapping in DuplexPipeline. Enhance Assistants and ToolLibrary components to utilize new mappings for improved tool identification and display. Update DebugDrawer to reflect changes in tool display names during interactions. 2026-02-27 15:50:43 +08:00
Xin Wang
0f1165af64 Enhance Dialog component to accept contentClassName prop for customizable styling. Update ToolLibraryPage to utilize new prop for improved modal layout. Refactor text in ToolLibrary for clarity. 2026-02-27 15:12:01 +08:00
Xin Wang
487634c494 Implement tool parameter management in ToolLibrary, including types, default value handling, and schema generation. Introduce functions for creating, updating, and removing tool parameters, enhancing the configuration capabilities for tools. Update state management to reflect new parameter structure. 2026-02-27 15:04:52 +08:00
Xin Wang
5f768edf68 Add parameter schema and defaults to ToolResource model and schemas. Implement runtime tool resolution in assistants and tools routers, ensuring proper handling of tool parameters. Update tests to validate new functionality and ensure correct integration of parameter handling in the API. 2026-02-27 14:44:28 +08:00
Xin Wang
d942c85eff Add new tools to DuplexPipeline: calculator, code_interpreter, turn_on_camera, turn_off_camera, increase_volume, and decrease_volume. Implement fallback schema for unknown string tools and assign default client executors for specific tools. Update tests to validate new functionality and ensure correct tool handling in the pipeline. 2026-02-27 13:59:37 +08:00
Xin Wang
8d453e10da Refactor AssistantsPage to utilize createPortal for template suggestion dropdowns, enhancing rendering performance and ensuring correct positioning relative to the document body. This change improves user experience by maintaining dropdown visibility during text input. 2026-02-27 13:27:13 +08:00
Xin Wang
e108ee66cb Enhance AssistantsPage template suggestion functionality by adding control parameter to updateTemplateSuggestionState. This allows for dynamic positioning of suggestion dropdowns based on caret position, improving user experience during text input. Update relevant event handlers to pass control element for accurate suggestion placement. 2026-02-27 12:18:48 +08:00