- Introduce dynamic variable definitions in AssistantConfig and Assistant models, allowing for flexible prompt customization.
- Implement validation for dynamic variable names and types in the schema.
- Update backend services and routes to handle dynamic variables in assistant configurations and runtime processing.
- Enhance frontend components to support dynamic variable definitions, including a new editor for managing variables.
- Add tests to ensure proper functionality and validation of dynamic variables in various scenarios.
- Introduce new fields for knowledge retrieval configuration in AssistantConfig and Assistant models, including mode, top_n, and score_threshold.
- Implement KnowledgeRetrievalConfig schema with validation for top_n.
- Update backend services and routes to handle knowledge retrieval settings.
- Enhance frontend components to support knowledge retrieval configuration, including a new dialog for advanced settings.
- Add tests for knowledge retrieval configuration validation and description generation.
- Add a guideline to reference frontend/DESIGN.md for interface design considerations.
- Maintain existing emphasis on writing clear thoughts before coding for better understanding and maintainability.
- Add DialogDescription components to provide context in the settings and search dialogs.
- Refactor layout for better spacing and organization of form elements in the settings dialog.
- Update search results display to improve user experience with clearer messaging and styling.
- Introduce footer buttons for closing dialogs, enhancing user interaction.
- Update the main knowledge page to support a list mode.
- Add new pages for creating and editing knowledge bases, utilizing the ComponentsKnowledgePage component with appropriate modes.
- Introduce a new API method for retrieving knowledge base details by ID.
- Enhance the ComponentsKnowledgePage to handle different modes (list, create, edit) and improve user interaction with knowledge bases.
- Add new models for `KnowledgeDocument` and `KnowledgeChunk` to manage document ingestion and chunking.
- Implement S3-compatible storage integration for knowledge documents, allowing for file uploads and retrieval.
- Introduce API endpoints for managing knowledge bases and documents, including creation, deletion, and searching.
- Update frontend components to support knowledge base configuration and document management, improving user interaction.
- Enhance backend services for knowledge processing and retrieval, ensuring robust handling of document statuses and errors.
- Introduce a new `turnConfig` field in `AssistantConfig` and `Assistant` models to manage user interaction settings.
- Implement `TurnConfig`, `BargeInConfig`, `VadConfig`, and `TurnDetectionConfig` schemas to define turn management strategies.
- Update the backend to handle turn configuration in the database and during assistant operations.
- Enhance frontend components with a `TurnConfigEditor` for configuring turn settings, including VAD and barge-in strategies.
- Modify existing pages to integrate turn configuration, improving user experience and interaction capabilities.
- Introduce new fields `dify_api_url` and `dify_api_key` in `AssistantConfig` for Dify API integration.
- Update `requirements.txt` to include `dify-client-python` for Dify SDK support.
- Modify `config_resolver` to handle Dify connection information.
- Add a new `globalNode` type in workflow specifications to provide unified settings across workflows.
- Enhance node specifications with additional constraints and default values for better configuration management.
- Update frontend components to support the new `globalNode` type and its properties, improving workflow editor functionality.
- Add a new guideline emphasizing the importance of articulating thoughts in simple language before coding.
- Maintain existing focus on building with MVP principles for better code maintainability and readability.
- Create a new JSON schema file for documentation settings.
- Add core features documentation detailing assistant configuration and model integration.
- Include an index page introducing the AI 视频助手 and its platform components.
- Document integration methods for model services and external agents.
- Provide a quickstart guide for setting up the local development environment using Docker.
- Add light and dark logo SVG files for branding purposes.
- Introduce a new state variable `callEnded` in `useVoicePreview` to manage call termination more effectively.
- Modify `MobileCallPage` to adjust the `inCall` condition based on `callEnded`, ensuring accurate UI updates during and after calls.
- Update rendering logic to display appropriate messages based on call status, enhancing user feedback during call transitions.
- Refactor button visibility to only show when in an active call, improving user interface clarity.
- Update the effect to reset the view to "camera" when the call status is "idle" or "failed", improving user experience during call interruptions.
- Modify the rendering condition for the MobileCallTranscript to only display when inCall is true, ensuring chat messages are shown appropriately during active calls.
- Implement a new API endpoint for duplicating tools in the backend.
- Add a duplicate_tool function in ComponentsToolsPage to handle tool duplication requests.
- Update the UI to include a dropdown menu option for duplicating tools, improving user experience.
- Refactor the remove function to accept a resource object instead of an ID for better clarity and maintainability.
- Implement a new API endpoint for deleting conversations in the backend.
- Enhance the HistoryPage component to include a dropdown menu for conversation actions, allowing users to delete conversations.
- Update the pagination logic to handle conversation removal and improve user experience during deletion.
- Adjust the loading state and error handling for the delete operation, ensuring smooth interaction.
- Introduce new database models for conversation sessions, messages, and artifacts to support conversation history tracking.
- Implement API routes for listing conversations and retrieving detailed conversation data, enhancing user interaction with historical records.
- Add a conversation recorder service to persist conversation messages in real-time without disrupting ongoing calls.
- Update the frontend to display conversation history, including filtering and sorting options, improving user experience.
- Enhance the pipeline to integrate conversation history recording seamlessly during interactions.
- Introduce a new `NetworkQualityIndicator` component to visually represent network quality status during calls.
- Integrate the `NetworkQualityIndicator` into `AssistantPage` and `MobileCallPage`, enhancing user awareness of connection quality.
- Update `use-voice-preview` hook to manage network quality metrics, improving overall call experience and performance monitoring.
- Introduce a new `MobileCallTranscript` component to display chat messages during calls, improving user interaction.
- Implement message timestamp formatting and automatic scrolling for better usability.
- Add a toggle button to switch between camera view and chat view, enhancing the user experience during calls.
- Refactor the layout of the `MobileCallPage` to accommodate the new chat functionality while maintaining existing video call features.
- Introduce a new `RuntimeTool` model to encapsulate tool data for runtime sessions, including attributes like `id`, `name`, `function_name`, `type`, and `description`.
- Update the `AssistantConfig` model to include a list of reusable tools, allowing for better management of tools within assistant configurations.
- Modify the `config_resolver` service to fetch and resolve tools associated with assistants, ensuring they are available during runtime.
- Refactor tool-related CRUD operations in the `tools` route to support the new runtime execution model, enhancing the overall tool management system.
- Update documentation and comments to reflect changes in tool execution and configuration handling, improving clarity for future development.
- Introduce a new `sync_default_tools` function in `session.py` to ensure essential reusable tools are created without overwriting existing edits.
- Update the `lifespan` context manager in `app.py` to call `sync_default_tools`, enhancing the initialization process for the application.
- This change improves the management of default tools within the system, ensuring they are available for use while preserving user modifications.
- Refactor `CallDeviceSelect` to `CallAudioDeviceSelect`, allowing simultaneous selection of microphone and output devices.
- Introduce dropdown menus for selecting audio input and output devices, improving user experience.
- Update `useVoicePreview` hook to manage audio output devices, including support for selecting and applying output devices.
- Modify `MobileCallPage` to integrate new device selection components, ensuring seamless audio and video call functionality.
- Introduce a new `auth-redirect` module to manage safe return paths for login and redirection.
- Update the login page to utilize `readReturnToFromLocation` for redirecting users post-login.
- Modify the `AuthGate` component to use `loginPathWithReturnTo` for redirecting unauthorized users, enhancing security and user experience.
- Adjust API request handling to incorporate the new redirection logic, ensuring users are directed to the appropriate login page with return paths.
- Introduce a new `Tool` model and `AssistantToolBinding` for managing reusable tools within the application.
- Implement CRUD operations for tools in the new `tools` route, allowing for the creation, retrieval, updating, and deletion of tools.
- Update the `Assistant` model to include a list of tool IDs, enabling assistants to utilize these tools.
- Enhance the backend routes to synchronize tool bindings with assistants, ensuring proper management of tool associations.
- Add frontend components for tool management, including a tool picker in the assistant configuration, improving user experience in tool selection.
- Create a mobile call page to facilitate video calls, integrating camera and microphone selection for enhanced communication capabilities.
- Update API definitions to include tool-related types and operations, ensuring consistency across the application.
- Add a migration script to create the necessary database tables for tools and bindings, supporting the new functionality.
- Extend the `allowedDevOrigins` in `next.config.ts` to include "118.89.89.89" for enhanced development flexibility.
- Add a new `.DS_Store` file in the `scripts` directory, which is a system file created by macOS for folder metadata.
- Introduce a new `auth` module with login, logout, and user verification endpoints for a single admin user.
- Update backend routes to require admin authentication for sensitive operations, enhancing security.
- Modify frontend components to include an authentication provider and gate, ensuring only authorized users can access the application.
- Implement a login page for admin access, improving user experience and security management.
- Update API request handling to redirect unauthorized users to the login page, ensuring proper access control.
- Modify `docker-compose.yaml` to use environment variables for HTTP and HTTPS port mappings, allowing flexibility in port configuration.
- Enhance `README.md` with instructions for using non-standard HTTPS ports, including examples for remote development setups.
- Introduce a new `nginx-remote-dev` service in `docker-compose.yaml` for remote development environments, allowing Nginx to proxy requests to the host machine's frontend and backend services.
- Create a new Nginx configuration file `ai-video.remote-dev.conf` tailored for remote setups, ensuring proper handling of WebSocket and API requests.
- Update `README.md` to include detailed instructions for setting up remote development with Nginx, including certificate generation and service initialization.
- Enhance the `setup-certs.sh` script to support additional hostnames for certificate signing, improving flexibility for remote access.
- Revise documentation to clarify the development paths and requirements for local and remote environments.
- Replace the `config.py` module with a new `settings.py` to streamline environment variable management, focusing on database, CORS, and TURN settings.
- Update references throughout the backend codebase to use the new `settings` module instead of the deprecated `config`.
- Modify the `.env.example` file to reflect the new configuration approach, indicating that model provider credentials should be maintained separately.
- Enhance the `AssistantConfig` model to clarify the source of runtime connection information, ensuring it is injected from model resources rather than relying on defaults from the environment.
- Introduce new user scripts for audio and video management in the Tampermonkey environment, enhancing WebRTC capabilities.
- Remove the showPassword state and associated logic for toggling password visibility.
- Streamline the password input to always use type "password" and directly handle value changes.
- Update placeholder text for clarity while maintaining existing functionality for stored values.
- Integrate Alembic for managing database schema migrations, replacing the previous SQL schema management.
- Update the FastAPI application to synchronize interface definitions at startup.
- Modify the Docker Compose command to run Alembic migrations before starting the API.
- Enhance the Makefile with new commands for database migration and revision management.
- Remove outdated SQL schema and seed files, transitioning to a more dynamic migration approach.
- Add initial migration scripts and configuration for Alembic, ensuring a structured database evolution.
- Extend the AssistantConfig model to include agent_interface_type, agent_values, and agent_secrets for enhanced agent management.
- Update ModelType to include "Agent" for broader capability support.
- Seed new agent models and configurations in the database for Dify, FastGPT, and OpenCode applications.
- Modify the Assistant routes to validate and handle agent capabilities.
- Implement agent resource resolution in the config resolver for runtime configuration.
- Enhance the interface catalog with agent-specific fields and definitions.
- Update frontend components to manage agent configurations, including form handling and state management for agent-related inputs.
- Introduce a new user script `audio-sender-sine-tester.js` that lists current WebRTC audio senders and allows temporary sine wave transmission through selected senders.
- Implement functionality for managing audio sender candidates, tracking audio statistics, and providing real-time logging of audio activities.
- Enhance audio testing capabilities with sine wave generation and track replacement features, improving overall audio management in the WebRTC environment.
- This addition supports better monitoring and testing of audio streams in applications utilizing WebRTC.
- Delete the user scripts `copilot-native-element.js` and `copilot-native-static.js` that implemented native-track bridges for audio and video input selection in WebRTC.
- These scripts provided functionalities for managing remote audio and video tracks, recording capabilities, and chat logging features for AI interactions.
- The removal is part of a refactoring effort to streamline the codebase and improve overall media management.
- Introduce a new user script `video-track-probe.js` that probes WebRTC video input/output tracks and displays active RTP statistics.
- Implement functionality to manage video track candidates, including live video detection and performance metrics.
- Enhance logging features to provide real-time feedback on video track status and statistics.
- Integrate mechanisms for tracking video elements and their associated media streams, improving overall media management in the WebRTC environment.
- Introduce a new user script `audio-track-probe.js` that probes WebRTC audio input/output tracks and displays active RTP statistics.
- Implement functionality to manage audio track candidates, including live audio detection and audio level analysis.
- Enhance logging features to provide real-time feedback on audio track status and metrics.
- Integrate audio context management for improved audio visualization and monitoring capabilities.
- Update the description to clarify that participant audio is fed into AI after both sides are connected.
- Introduce new functions to manage audio track patterns and enhance the logic for determining participant and operator presence.
- Improve gate status messaging to reflect the online status of participants and operators.
- Add volume handling functions for better audio visualization and feedback.
- Refactor existing functions to streamline audio processing and improve overall code clarity.
- Introduce a new user script `copilot-element2-two-party.js` that implements a two-party gated bridge for audio input to AI, ensuring both sides are connected before processing.
- Implement functionality for managing audio and video tracks, including recording capabilities for AI input.
- Enhance chat logging features to provide real-time feedback on AI interactions and recording status.
- Integrate mechanisms for handling media streams, improving overall media management in the WebRTC environment.
- Introduce a new user script `copilot-native-direct-output.js` that facilitates direct AI response track output to business senders.
- Implement functionality for managing audio and video tracks, including recording capabilities for AI input.
- Enhance chat logging features to provide real-time feedback on AI interactions and recording status.
- Integrate mechanisms for handling media streams, improving overall media management in the WebRTC environment.
- Introduce a new user script `copilot-native.js` that implements a native-track bridge for audio and video input selection in WebRTC.
- Implement functionality for managing remote audio and video tracks, including recording capabilities for AI input.
- Enhance chat logging features to provide real-time feedback on AI interactions and recording status.
- Integrate mechanisms for handling media streams, improving overall media management in the WebRTC environment.
- Introduce a new user script `copilot-native-element.js` that implements a native-track bridge for audio and video input selection in WebRTC.
- Implement functionality for managing remote audio and video tracks, including recording capabilities for AI input.
- Enhance chat logging features to provide real-time feedback on AI interactions and recording status.
- Integrate mechanisms for handling media streams, improving overall media management in the WebRTC environment.
- Update vision model authorization logic to rely solely on server-side configuration, removing client offer dependency for vision capabilities.
- Enhance clarity in comments regarding the role of server configurations in enabling vision understanding.
- Update `requirements.txt` to include Pillow for image handling.
- Refactor vision model validation logic in `voice_webrtc.py` to improve error handling for unsupported image input.
- Introduce new functions in `pipeline.py` for image data processing and analysis using vision models.
- Implement `VisionCaptureProcessor` to manage video frame requests for auxiliary vision model analysis.
- Enhance the pipeline to support image input requests and integrate vision model responses into the processing flow.
- Introduce a new user script `copilot-native-static.js` that implements a native-track bridge with static input/output selection rules for WebRTC.
- Implement functionality for managing remote audio and video tracks, including recording capabilities for AI input.
- Enhance chat logging features to provide real-time feedback on AI interactions and recording status.
- Integrate mechanisms for handling media streams and improve overall media management in the WebRTC environment.
- Introduce a new function to capture video streams from playable HTML elements, enhancing the ability to process remote video feeds.
- Implement logic to manage and store captured video streams, ensuring only live tracks are utilized.
- Update existing audio processing functions to integrate video handling, improving overall media management in the WebRTC environment.
- Enhance logging for better debugging and user feedback regarding video stream capture status.
- Add new database management targets: db-up, db-schema, db-seed-interface-definitions, and db-init to streamline database setup and seeding processes.
- Update db-seed and db-reset dependencies to include new initialization steps for better data integrity.
- Introduce new SQL files for schema definition and interface definitions seeding, ensuring a consistent database structure.
- Refactor existing seed scripts to align with new dependencies and improve clarity in database operations.
- Add db-drop-schema target to facilitate schema deletion and recreation for development purposes.
- Introduce db-reset-hard target to streamline the process of resetting the database and seeding data after schema changes.
- Update .PHONY targets for improved clarity and maintainability.
- Remove db-init target from db-seed and db-reset dependencies to simplify the seed process.
- Update .PHONY targets for clarity and maintainability.
- Enhance comments for better understanding of database operations.
- Introduce new fields in AssistantConfig, schemas, and database models to support vision capabilities, including `vision_enabled` and `vision_model_resource_id`.
- Enhance validation logic in routes to ensure proper handling of vision models and their requirements.
- Update the AssistantPage and related frontend components to include options for enabling vision understanding and selecting appropriate vision models.
- Modify database seed scripts to include vision-related data for assistants, ensuring consistent setup.
- Refactor related functions to integrate vision model handling in the audio-visual processing pipeline.
- Add `hasUnsavedChanges` prop to `DebugDrawer` to indicate unsaved changes in the AssistantPage.
- Update `DebugVoicePanel` to display appropriate messages and disable actions when there are unsaved changes.
- Refactor related components to ensure consistent handling of unsaved changes across the AssistantPage, improving user experience during interactions.
- Introduce a new `support_image_input` field in model resources, allowing models to indicate support for image input.
- Update the backend models, schemas, and database seed scripts to accommodate the new field.
- Enhance the AssistantConfig and related routes to handle image input capabilities, ensuring proper validation and error handling.
- Modify the frontend components to include toggles for enabling visual understanding and filtering models based on image input support.
- Implement necessary adjustments in the voice preview and pipeline to integrate video stream handling alongside audio functionalities.
