- Update `requirements.txt` to include Pillow for image handling.
- Refactor vision model validation logic in `voice_webrtc.py` to improve error handling for unsupported image input.
- Introduce new functions in `pipeline.py` for image data processing and analysis using vision models.
- Implement `VisionCaptureProcessor` to manage video frame requests for auxiliary vision model analysis.
- Enhance the pipeline to support image input requests and integrate vision model responses into the processing flow.
- Introduce a new user script `copilot-native-static.js` that implements a native-track bridge with static input/output selection rules for WebRTC.
- Implement functionality for managing remote audio and video tracks, including recording capabilities for AI input.
- Enhance chat logging features to provide real-time feedback on AI interactions and recording status.
- Integrate mechanisms for handling media streams and improve overall media management in the WebRTC environment.
- Introduce a new function to capture video streams from playable HTML elements, enhancing the ability to process remote video feeds.
- Implement logic to manage and store captured video streams, ensuring only live tracks are utilized.
- Update existing audio processing functions to integrate video handling, improving overall media management in the WebRTC environment.
- Enhance logging for better debugging and user feedback regarding video stream capture status.
- Add new database management targets: db-up, db-schema, db-seed-interface-definitions, and db-init to streamline database setup and seeding processes.
- Update db-seed and db-reset dependencies to include new initialization steps for better data integrity.
- Introduce new SQL files for schema definition and interface definitions seeding, ensuring a consistent database structure.
- Refactor existing seed scripts to align with new dependencies and improve clarity in database operations.
- Add db-drop-schema target to facilitate schema deletion and recreation for development purposes.
- Introduce db-reset-hard target to streamline the process of resetting the database and seeding data after schema changes.
- Update .PHONY targets for improved clarity and maintainability.
- Remove db-init target from db-seed and db-reset dependencies to simplify the seed process.
- Update .PHONY targets for clarity and maintainability.
- Enhance comments for better understanding of database operations.
- Introduce new fields in AssistantConfig, schemas, and database models to support vision capabilities, including `vision_enabled` and `vision_model_resource_id`.
- Enhance validation logic in routes to ensure proper handling of vision models and their requirements.
- Update the AssistantPage and related frontend components to include options for enabling vision understanding and selecting appropriate vision models.
- Modify database seed scripts to include vision-related data for assistants, ensuring consistent setup.
- Refactor related functions to integrate vision model handling in the audio-visual processing pipeline.
- Add `hasUnsavedChanges` prop to `DebugDrawer` to indicate unsaved changes in the AssistantPage.
- Update `DebugVoicePanel` to display appropriate messages and disable actions when there are unsaved changes.
- Refactor related components to ensure consistent handling of unsaved changes across the AssistantPage, improving user experience during interactions.
- Introduce a new `support_image_input` field in model resources, allowing models to indicate support for image input.
- Update the backend models, schemas, and database seed scripts to accommodate the new field.
- Enhance the AssistantConfig and related routes to handle image input capabilities, ensuring proper validation and error handling.
- Modify the frontend components to include toggles for enabling visual understanding and filtering models based on image input support.
- Implement necessary adjustments in the voice preview and pipeline to integrate video stream handling alongside audio functionalities.
- Introduce a new user script `copilot.js` for WebRTC that implements PCM ScriptProcessor to address packet loss issues.
- Add features for real-time audio monitoring and improved audio stream handling, including mechanisms for capturing and processing remote audio streams.
- Implement a robust audio context setup with gain nodes and oscillators for enhanced audio management.
- Enhance logging and state management for better debugging and monitoring of audio streams in the WebRTC environment.
- Introduce a new utility class `.scrollbar-subtle` in `globals.css` for a refined scrollbar appearance.
- Update `AssistantPage` to apply the subtle scrollbar styling to overflow containers, enhancing the user interface.
- Reset message state and sequence in `useVoicePreview` hook upon resource release, ensuring clean state management.
- Introduce a new `.env.example` file for environment variable setup, including `PUBLIC_IP`, `TURN_SECRET`, and `TURN_URLS` for WebRTC TURN server configuration.
- Update `docker-compose.yaml` to support TURN server deployment with necessary environment variables and commands.
- Enhance backend configuration and routes to include WebRTC ICE server settings, allowing for STUN/TURN server integration.
- Implement a new service for managing WebRTC ICE server configurations, providing credentials for TURN when configured.
- Modify frontend API to fetch ICE server configurations dynamically, improving support for cross-network voice preview.
- Add a new `useCameraPreview` hook for managing camera access and video stream handling.
- Integrate camera preview capabilities into the AssistantPage, allowing users to toggle video stream visibility.
- Introduce a new toggle for enabling visual understanding, which activates the video stream preview alongside voice functionalities.
- Update the DebugDrawer component to support switching between chat and video views based on the visual understanding setting.
- Refactor related components to accommodate the new camera features and improve user interaction during debugging.
- Add new fields in AssistantConfig for FastGPT connection details, including `fastgpt_api_url`, `fastgpt_api_key`, and `fastgpt_app_id`.
- Update the pipeline to utilize the new FastGPT configuration, ensuring proper integration with external services.
- Introduce type handling for different assistant types, including support for realtime modes and external brain management.
- Refactor frontend components to include hints for FastGPT configuration inputs, improving user guidance during setup.
- Introduce a new parameter `audio_out_end_silence_secs` in the `_base_params` function to control the duration of silence added after the end frame, allowing for smoother call termination.
- Set the default value to 0 to ensure immediate hang-up after the end speech, enhancing user experience during call endings.
- Introduce mechanisms in the pipeline to ensure that the end call process waits for the completion of the end speech before hanging up, improving user experience during call termination.
- Update the useVoicePreview hook to handle server-initiated call endings gracefully, distinguishing between normal and error disconnections.
- Adjust TTS stop frame timeout settings to optimize the timing of call terminations, ensuring timely responses without unnecessary delays.
- Refactor related components to support the new end call logic, enhancing overall workflow management and user interaction.
- Introduce edge transition speech functionality in the WorkflowEngine to provide optional speech during node transitions.
- Update pipeline execution to utilize the new transition speech feature, enhancing user experience by masking delays during transitions.
- Modify frontend components to support transition speech in edge specifications, allowing users to define and edit transition speech for edges.
- Refactor edge handling logic in the WorkflowEditor to accommodate the new transition speech field, improving workflow management capabilities.
- Change the 'addable' property of a specific node type to true, allowing for dynamic addition of nodes.
- Modify the GenericNode component to include a new icon and adjust styles for better visual representation.
- Update node handling logic to prevent deletion of 'startCall' nodes and improve node change handling in the workflow editor.
- Refactor layout and styling in the WorkflowEditor for a more polished user interface.
- Introduce a new WorkflowEngine class to manage workflow graphs, enabling dynamic node-based interactions.
- Update AssistantConfig to include a graph field for workflow definitions, allowing for flexible configuration.
- Modify pipeline execution to support workflow-driven dialogue, integrating node transitions and system prompts based on active nodes.
- Enhance frontend components to visualize active nodes and provide debugging capabilities, including highlighting the current node during interactions.
- Refactor existing components to accommodate new workflow functionalities and improve overall user experience.
- Introduce a new workflow editor component for visualizing and managing workflows, allowing users to add nodes and define connections.
- Implement backend support for node types, including validation and constraints for workflow graphs.
- Add new API endpoints for retrieving node types and their specifications.
- Enhance the AssistantPage to integrate the workflow editor, enabling users to create and edit workflows directly.
- Update frontend components to support new workflow functionalities, including condition edges and generic nodes.
- Refactor existing code to accommodate the new workflow features and improve overall structure.
- Add new fields to AssistantConfig for realtime interface configuration, including types, values, and secrets.
- Introduce StepFunRealtimeService to handle speech-to-speech processing via WebSocket, integrating STT, LLM, and TTS functionalities.
- Refactor pipeline execution to support a new realtime mode, allowing direct text input processing and immediate responses.
- Update model resource testing to include validation for StepFun Realtime connections.
- Enhance service factory to create realtime services based on configuration settings.
- Modify README documentation to reflect new realtime capabilities and usage instructions.
- Introduce event handlers in PassthroughLLMAssistantAggregator for managing LLM text streaming, including start, delta, and end events.
- Implement a new method to finalize text streams, ensuring proper handling of interruptions.
- Update useVoicePreview to support new message types for LLM text streaming, allowing real-time updates to chat messages.
- Enhance message sorting logic to maintain order based on timestamps and sequence numbers, improving user experience during voice interactions.
- Refactor TextInputProcessor to handle immediate and silent text inputs, improving user experience during voice interactions.
- Introduce PassthroughLLMAssistantAggregator to manage LLM responses while preserving context for downstream TTS processing.
- Update event handling for text input and client readiness, ensuring timely updates to the conversation context.
- Modify run_pipeline to integrate new aggregators and streamline message handling, enhancing overall pipeline efficiency.
- Improve message ordering in useVoicePreview to ensure accurate display of chat messages based on timestamps.
- Rename `setSelectedDeviceId` to `selectDevice` in `DebugVoicePanel` and `VoiceSessionControls` for clarity and consistency.
- Update `useVoicePreview` hook to implement the `selectDevice` function, enabling dynamic microphone switching during voice sessions.
- Enhance device selection logic to support real-time audio track replacement without requiring session reconnection.
- Introduce a new model structure for managing interface definitions and model resources, enhancing the backend's capability to handle various service integrations.
- Update the Makefile to reflect changes in database seeding and resource management commands.
- Remove the deprecated credentials management routes and replace them with a unified model registry API.
- Modify existing routes and schemas to align with the new model structure, ensuring seamless integration with the frontend.
- Enhance database seeding scripts to populate new model resources and their configurations.
- Update README documentation to reflect the new architecture and usage instructions for model resources and interface definitions.
- Introduce new Xfyun ASR and TTS services, enabling integration with iFlytek's voice recognition and synthesis capabilities.
- Update AssistantConfig model to include interface types for STT and TTS.
- Enhance credential testing to validate Xfyun credentials.
- Modify service factory to create Xfyun services based on configuration.
- Update README with new configuration details for Xfyun integration.
- Add new frontend components for visualizing audio streams and managing user interactions.
- Add audio input selection to DebugVoicePanel, allowing users to choose their microphone device.
- Update useVoicePreview hook to manage available audio inputs and selected device state.
- Enhance device enumeration and selection handling to ensure a seamless user experience during voice interactions.
- Add a new Docker configuration for the UI in launch.json to facilitate development.
- Refactor pipeline.py to integrate a TranscriptProcessor for managing user and assistant transcripts, including event handlers for real-time updates and message handling.
- Update useVoicePreview.ts to establish a data channel for sending and receiving text messages, improving interaction flow.
- Modify AssistantPage.tsx to support displaying chat messages and sending user input, enhancing the user experience during voice interactions.
- Revise DebugTranscriptPanel to dynamically render chat messages with timestamps, improving the visual representation of conversation history.
- Update CLAUDE.md to reflect changes in the navigation model, emphasizing the use of App Router routes for sidebar sections.
- Refactor layout.tsx to wrap children in AppShell, enhancing the overall layout structure.
- Replace AppShell usage in page.tsx with HomePage component for better separation of concerns.
- Introduce new pages for assistants, components, dashboard, history, profile, and test, each rendering their respective components.
- Revise Sidebar component to utilize Next.js Link for navigation and improve active state handling based on the current pathname.
- Update AssistantPage to support routing-driven modes (list, choose, edit) and streamline form handling for assistant creation and editing.
- Introduce `setup-certs.sh` script for generating trusted local TLS certificates using mkcert.
- Add Nginx configuration files for local and Docker environments to handle HTTPS requests and proxy to backend services.
- Update `docker-compose.yaml` to include Nginx service for unified TLS entry and adjust frontend service ports for local development.
- Create `AGENTS.md` and `README.md` files to document the local HTTPS setup process and usage instructions.
- Modify backend startup commands in `README.md` for consistency with new requirements.
- Add `.gitignore` to exclude generated certificates from version control.
- Update README to reflect the integration of the DebugVoicePanel with WebSocket support for voice interactions.
- Refactor voice_webrtc.py to improve error handling during WebRTC signaling and include assistant_id in the offer payload.
- Add useVoicePreview hook to manage microphone access and WebRTC connections for real-time voice previews.
- Modify AssistantPage to incorporate new visualizer options and pass assistantId to DebugVoicePanel, enhancing user experience during audio interactions.
- Update API model to include new fields for voice, speed, and language, supporting TTS and ASR configurations.
- Implement rgbToHsl and hslToRgb functions for color space conversions.
- Introduce adaptPalette function to adjust colors based on dark/light themes, enhancing visual consistency.
- Add isDarkTheme function to determine the current theme state, improving theme handling across visual components.
- Update SpectrumVisualizer component to enhance the visual representation of audio frequencies with a new layout and smoother animations.
- Modify prop descriptions for clarity and adjust the number of frequency bars for better performance.
- Implement a refined drawing logic that maintains visual consistency across different themes and improves the overall user experience during audio playback.
- Introduce the NebulaVisualizer component, featuring particles that respond to audio input, enhancing the visual experience.
- Refactor AuraVisualizer, SpectrumVisualizer, and WaveVisualizer to utilize the adaptPalette function for improved theme handling.
- Update visualizer logic to enhance responsiveness and visual effects based on audio analysis, ensuring a cohesive user experience across components.
- Introduce the AssistantIdentity component to show the current assistant ID and provide a copy-to-clipboard feature.
- Update multiple sections of AssistantPage to include the AssistantIdentity component, enhancing user interaction with assistant IDs.
- Ensure the component handles the display of the assistant ID and provides feedback when copied successfully.
- Introduce three new audio visualizer components: AuraVisualizer, SpectrumVisualizer, and WaveVisualizer, enhancing the audio interaction experience.
- Replace the deprecated VoiceVisualizer with the new visualizers, ensuring a cohesive visual language across components.
- Update the AssistantPage to support dynamic visualization style switching, improving user engagement during audio interactions.
- Refactor DebugVoicePanel to accommodate the new visualizer props and enhance the overall debugging interface.
- Remove debug mode state management from AssistantPage, simplifying the component structure.
- Update DebugDrawer to eliminate mode selection, focusing on voice interaction features.
- Enhance the VoiceVisualizer component with improved visual effects and responsiveness to audio input.
- Adjust styles and layout for better user experience in the debugging interface.
- Introduce new fields for voice, speed, and language in the AssistantConfig and ProviderCredential models to support TTS and ASR configurations.
- Update the database schema and seeding script to accommodate the new fields, ensuring backward compatibility.
- Implement credential testing endpoints and logic to validate OpenAI-compatible credentials, enhancing user experience and reliability.
- Modify frontend components to include new fields in the credential forms and improve connection testing feedback.
- Refactor related services and API interactions to support the new credential testing feature.
- Introduce a new model field in the OpenCode form to manage language model selection.
- Refactor the form handling logic to improve data loading and error management for OpenCode assistants.
- Update UI components to utilize ResourceSelectField for model and voice configuration, enhancing user experience.
- Clear form fields when creating new OpenCode entries to ensure a fresh start for users.
- Update AssistantPage to use a stored value mask for API keys, improving security and user experience.
- Modify ComponentsModelsPage to display the current API key contextually, enhancing clarity for users.
- Adjust related components to ensure consistent handling of API key visibility and management.
- Update AssistantPage to handle API key input more securely by removing placeholder values and allowing empty submissions to retain existing keys.
- Introduce a new SecretInputField component for API key entry, improving user experience with visibility toggling and contextual hints.
- Modify ComponentsModelsPage to reflect similar API key handling, ensuring users can manage keys effectively while providing feedback on existing configurations.
- Add EditorBackButton for better navigation within the AssistantPage.
- Change 'DeepSeek-V3' to 'DeepSeek-Chat' and update its API key.
- Rename 'OpenAI TTS' to 'SiliconFlow-CosyVoice2-0.5B' and update its details.
- Add new models: 'SiliconFlow-TeleSpeechASR' and 'SiliconFlow-Qwen3-Embedding-4B' with corresponding API keys and configurations.
- Adjust existing entries to ensure consistency in the database seeding process.
- Update Makefile to include new database seed commands for assistants and credentials.
- Refactor assistant model to use explicit fields instead of a config dictionary, improving data integrity and clarity.
- Implement new seeding SQL script for assistants, ensuring dependencies on credentials are respected.
- Modify backend routes and frontend components to accommodate the new assistant structure, including direct field access for prompt, API URL, and keys.
- Enhance the AssistantPage component to handle the new data structure and streamline the save process for different assistant types.
Implement server-side duplication for both assistants and credentials, allowing users to create copies with unique IDs and modified names. Update the respective API routes and frontend components to handle duplication requests, ensuring sensitive information is securely managed. Enhance the AssistantPage and ComponentsModelsPage to support this new feature, including loading and error handling for the duplication process.
Add CRUD functionality for knowledge bases, including routes for listing, creating, updating, and deleting knowledge bases. Update the assistant model to include foreign key references to knowledge bases and modify the assistant configuration to handle external API keys securely. Refactor related services and routes to accommodate these changes, ensuring proper handling of credential resolution and configuration normalization.
Implement a new feature that allows users to create a copy of an existing assistant. The duplicate is inserted after the original, with the name suffixed by "副本" and the updated timestamp. Update the state management to handle the list of assistants accordingly, and add a dropdown menu item for triggering the duplication.
