- Introduce a new user script `copilot-native.js` that implements a native-track bridge for audio and video input selection in WebRTC.
- Implement functionality for managing remote audio and video tracks, including recording capabilities for AI input.
- Enhance chat logging features to provide real-time feedback on AI interactions and recording status.
- Integrate mechanisms for handling media streams, improving overall media management in the WebRTC environment.
- Introduce a new user script `copilot-native-element.js` that implements a native-track bridge for audio and video input selection in WebRTC.
- Implement functionality for managing remote audio and video tracks, including recording capabilities for AI input.
- Enhance chat logging features to provide real-time feedback on AI interactions and recording status.
- Integrate mechanisms for handling media streams, improving overall media management in the WebRTC environment.
- Update vision model authorization logic to rely solely on server-side configuration, removing client offer dependency for vision capabilities.
- Enhance clarity in comments regarding the role of server configurations in enabling vision understanding.
- Update `requirements.txt` to include Pillow for image handling.
- Refactor vision model validation logic in `voice_webrtc.py` to improve error handling for unsupported image input.
- Introduce new functions in `pipeline.py` for image data processing and analysis using vision models.
- Implement `VisionCaptureProcessor` to manage video frame requests for auxiliary vision model analysis.
- Enhance the pipeline to support image input requests and integrate vision model responses into the processing flow.
- Introduce a new user script `copilot-native-static.js` that implements a native-track bridge with static input/output selection rules for WebRTC.
- Implement functionality for managing remote audio and video tracks, including recording capabilities for AI input.
- Enhance chat logging features to provide real-time feedback on AI interactions and recording status.
- Integrate mechanisms for handling media streams and improve overall media management in the WebRTC environment.
- Introduce a new function to capture video streams from playable HTML elements, enhancing the ability to process remote video feeds.
- Implement logic to manage and store captured video streams, ensuring only live tracks are utilized.
- Update existing audio processing functions to integrate video handling, improving overall media management in the WebRTC environment.
- Enhance logging for better debugging and user feedback regarding video stream capture status.
- Add new database management targets: db-up, db-schema, db-seed-interface-definitions, and db-init to streamline database setup and seeding processes.
- Update db-seed and db-reset dependencies to include new initialization steps for better data integrity.
- Introduce new SQL files for schema definition and interface definitions seeding, ensuring a consistent database structure.
- Refactor existing seed scripts to align with new dependencies and improve clarity in database operations.
- Add db-drop-schema target to facilitate schema deletion and recreation for development purposes.
- Introduce db-reset-hard target to streamline the process of resetting the database and seeding data after schema changes.
- Update .PHONY targets for improved clarity and maintainability.
- Remove db-init target from db-seed and db-reset dependencies to simplify the seed process.
- Update .PHONY targets for clarity and maintainability.
- Enhance comments for better understanding of database operations.
- Introduce new fields in AssistantConfig, schemas, and database models to support vision capabilities, including `vision_enabled` and `vision_model_resource_id`.
- Enhance validation logic in routes to ensure proper handling of vision models and their requirements.
- Update the AssistantPage and related frontend components to include options for enabling vision understanding and selecting appropriate vision models.
- Modify database seed scripts to include vision-related data for assistants, ensuring consistent setup.
- Refactor related functions to integrate vision model handling in the audio-visual processing pipeline.
- Add `hasUnsavedChanges` prop to `DebugDrawer` to indicate unsaved changes in the AssistantPage.
- Update `DebugVoicePanel` to display appropriate messages and disable actions when there are unsaved changes.
- Refactor related components to ensure consistent handling of unsaved changes across the AssistantPage, improving user experience during interactions.
- Introduce a new `support_image_input` field in model resources, allowing models to indicate support for image input.
- Update the backend models, schemas, and database seed scripts to accommodate the new field.
- Enhance the AssistantConfig and related routes to handle image input capabilities, ensuring proper validation and error handling.
- Modify the frontend components to include toggles for enabling visual understanding and filtering models based on image input support.
- Implement necessary adjustments in the voice preview and pipeline to integrate video stream handling alongside audio functionalities.
- Introduce a new user script `copilot.js` for WebRTC that implements PCM ScriptProcessor to address packet loss issues.
- Add features for real-time audio monitoring and improved audio stream handling, including mechanisms for capturing and processing remote audio streams.
- Implement a robust audio context setup with gain nodes and oscillators for enhanced audio management.
- Enhance logging and state management for better debugging and monitoring of audio streams in the WebRTC environment.
- Introduce a new utility class `.scrollbar-subtle` in `globals.css` for a refined scrollbar appearance.
- Update `AssistantPage` to apply the subtle scrollbar styling to overflow containers, enhancing the user interface.
- Reset message state and sequence in `useVoicePreview` hook upon resource release, ensuring clean state management.
- Introduce a new `.env.example` file for environment variable setup, including `PUBLIC_IP`, `TURN_SECRET`, and `TURN_URLS` for WebRTC TURN server configuration.
- Update `docker-compose.yaml` to support TURN server deployment with necessary environment variables and commands.
- Enhance backend configuration and routes to include WebRTC ICE server settings, allowing for STUN/TURN server integration.
- Implement a new service for managing WebRTC ICE server configurations, providing credentials for TURN when configured.
- Modify frontend API to fetch ICE server configurations dynamically, improving support for cross-network voice preview.
- Add a new `useCameraPreview` hook for managing camera access and video stream handling.
- Integrate camera preview capabilities into the AssistantPage, allowing users to toggle video stream visibility.
- Introduce a new toggle for enabling visual understanding, which activates the video stream preview alongside voice functionalities.
- Update the DebugDrawer component to support switching between chat and video views based on the visual understanding setting.
- Refactor related components to accommodate the new camera features and improve user interaction during debugging.
- Add new fields in AssistantConfig for FastGPT connection details, including `fastgpt_api_url`, `fastgpt_api_key`, and `fastgpt_app_id`.
- Update the pipeline to utilize the new FastGPT configuration, ensuring proper integration with external services.
- Introduce type handling for different assistant types, including support for realtime modes and external brain management.
- Refactor frontend components to include hints for FastGPT configuration inputs, improving user guidance during setup.
- Introduce a new parameter `audio_out_end_silence_secs` in the `_base_params` function to control the duration of silence added after the end frame, allowing for smoother call termination.
- Set the default value to 0 to ensure immediate hang-up after the end speech, enhancing user experience during call endings.
- Introduce mechanisms in the pipeline to ensure that the end call process waits for the completion of the end speech before hanging up, improving user experience during call termination.
- Update the useVoicePreview hook to handle server-initiated call endings gracefully, distinguishing between normal and error disconnections.
- Adjust TTS stop frame timeout settings to optimize the timing of call terminations, ensuring timely responses without unnecessary delays.
- Refactor related components to support the new end call logic, enhancing overall workflow management and user interaction.
- Introduce edge transition speech functionality in the WorkflowEngine to provide optional speech during node transitions.
- Update pipeline execution to utilize the new transition speech feature, enhancing user experience by masking delays during transitions.
- Modify frontend components to support transition speech in edge specifications, allowing users to define and edit transition speech for edges.
- Refactor edge handling logic in the WorkflowEditor to accommodate the new transition speech field, improving workflow management capabilities.
- Change the 'addable' property of a specific node type to true, allowing for dynamic addition of nodes.
- Modify the GenericNode component to include a new icon and adjust styles for better visual representation.
- Update node handling logic to prevent deletion of 'startCall' nodes and improve node change handling in the workflow editor.
- Refactor layout and styling in the WorkflowEditor for a more polished user interface.
- Introduce a new WorkflowEngine class to manage workflow graphs, enabling dynamic node-based interactions.
- Update AssistantConfig to include a graph field for workflow definitions, allowing for flexible configuration.
- Modify pipeline execution to support workflow-driven dialogue, integrating node transitions and system prompts based on active nodes.
- Enhance frontend components to visualize active nodes and provide debugging capabilities, including highlighting the current node during interactions.
- Refactor existing components to accommodate new workflow functionalities and improve overall user experience.
- Introduce a new workflow editor component for visualizing and managing workflows, allowing users to add nodes and define connections.
- Implement backend support for node types, including validation and constraints for workflow graphs.
- Add new API endpoints for retrieving node types and their specifications.
- Enhance the AssistantPage to integrate the workflow editor, enabling users to create and edit workflows directly.
- Update frontend components to support new workflow functionalities, including condition edges and generic nodes.
- Refactor existing code to accommodate the new workflow features and improve overall structure.
- Add new fields to AssistantConfig for realtime interface configuration, including types, values, and secrets.
- Introduce StepFunRealtimeService to handle speech-to-speech processing via WebSocket, integrating STT, LLM, and TTS functionalities.
- Refactor pipeline execution to support a new realtime mode, allowing direct text input processing and immediate responses.
- Update model resource testing to include validation for StepFun Realtime connections.
- Enhance service factory to create realtime services based on configuration settings.
- Modify README documentation to reflect new realtime capabilities and usage instructions.
- Introduce event handlers in PassthroughLLMAssistantAggregator for managing LLM text streaming, including start, delta, and end events.
- Implement a new method to finalize text streams, ensuring proper handling of interruptions.
- Update useVoicePreview to support new message types for LLM text streaming, allowing real-time updates to chat messages.
- Enhance message sorting logic to maintain order based on timestamps and sequence numbers, improving user experience during voice interactions.
- Refactor TextInputProcessor to handle immediate and silent text inputs, improving user experience during voice interactions.
- Introduce PassthroughLLMAssistantAggregator to manage LLM responses while preserving context for downstream TTS processing.
- Update event handling for text input and client readiness, ensuring timely updates to the conversation context.
- Modify run_pipeline to integrate new aggregators and streamline message handling, enhancing overall pipeline efficiency.
- Improve message ordering in useVoicePreview to ensure accurate display of chat messages based on timestamps.
- Rename `setSelectedDeviceId` to `selectDevice` in `DebugVoicePanel` and `VoiceSessionControls` for clarity and consistency.
- Update `useVoicePreview` hook to implement the `selectDevice` function, enabling dynamic microphone switching during voice sessions.
- Enhance device selection logic to support real-time audio track replacement without requiring session reconnection.
- Introduce a new model structure for managing interface definitions and model resources, enhancing the backend's capability to handle various service integrations.
- Update the Makefile to reflect changes in database seeding and resource management commands.
- Remove the deprecated credentials management routes and replace them with a unified model registry API.
- Modify existing routes and schemas to align with the new model structure, ensuring seamless integration with the frontend.
- Enhance database seeding scripts to populate new model resources and their configurations.
- Update README documentation to reflect the new architecture and usage instructions for model resources and interface definitions.
- Introduce new Xfyun ASR and TTS services, enabling integration with iFlytek's voice recognition and synthesis capabilities.
- Update AssistantConfig model to include interface types for STT and TTS.
- Enhance credential testing to validate Xfyun credentials.
- Modify service factory to create Xfyun services based on configuration.
- Update README with new configuration details for Xfyun integration.
- Add new frontend components for visualizing audio streams and managing user interactions.
- Add audio input selection to DebugVoicePanel, allowing users to choose their microphone device.
- Update useVoicePreview hook to manage available audio inputs and selected device state.
- Enhance device enumeration and selection handling to ensure a seamless user experience during voice interactions.
- Add a new Docker configuration for the UI in launch.json to facilitate development.
- Refactor pipeline.py to integrate a TranscriptProcessor for managing user and assistant transcripts, including event handlers for real-time updates and message handling.
- Update useVoicePreview.ts to establish a data channel for sending and receiving text messages, improving interaction flow.
- Modify AssistantPage.tsx to support displaying chat messages and sending user input, enhancing the user experience during voice interactions.
- Revise DebugTranscriptPanel to dynamically render chat messages with timestamps, improving the visual representation of conversation history.
- Update CLAUDE.md to reflect changes in the navigation model, emphasizing the use of App Router routes for sidebar sections.
- Refactor layout.tsx to wrap children in AppShell, enhancing the overall layout structure.
- Replace AppShell usage in page.tsx with HomePage component for better separation of concerns.
- Introduce new pages for assistants, components, dashboard, history, profile, and test, each rendering their respective components.
- Revise Sidebar component to utilize Next.js Link for navigation and improve active state handling based on the current pathname.
- Update AssistantPage to support routing-driven modes (list, choose, edit) and streamline form handling for assistant creation and editing.
- Introduce `setup-certs.sh` script for generating trusted local TLS certificates using mkcert.
- Add Nginx configuration files for local and Docker environments to handle HTTPS requests and proxy to backend services.
- Update `docker-compose.yaml` to include Nginx service for unified TLS entry and adjust frontend service ports for local development.
- Create `AGENTS.md` and `README.md` files to document the local HTTPS setup process and usage instructions.
- Modify backend startup commands in `README.md` for consistency with new requirements.
- Add `.gitignore` to exclude generated certificates from version control.
- Update README to reflect the integration of the DebugVoicePanel with WebSocket support for voice interactions.
- Refactor voice_webrtc.py to improve error handling during WebRTC signaling and include assistant_id in the offer payload.
- Add useVoicePreview hook to manage microphone access and WebRTC connections for real-time voice previews.
- Modify AssistantPage to incorporate new visualizer options and pass assistantId to DebugVoicePanel, enhancing user experience during audio interactions.
- Update API model to include new fields for voice, speed, and language, supporting TTS and ASR configurations.
- Implement rgbToHsl and hslToRgb functions for color space conversions.
- Introduce adaptPalette function to adjust colors based on dark/light themes, enhancing visual consistency.
- Add isDarkTheme function to determine the current theme state, improving theme handling across visual components.
- Update SpectrumVisualizer component to enhance the visual representation of audio frequencies with a new layout and smoother animations.
- Modify prop descriptions for clarity and adjust the number of frequency bars for better performance.
- Implement a refined drawing logic that maintains visual consistency across different themes and improves the overall user experience during audio playback.
- Introduce the NebulaVisualizer component, featuring particles that respond to audio input, enhancing the visual experience.
- Refactor AuraVisualizer, SpectrumVisualizer, and WaveVisualizer to utilize the adaptPalette function for improved theme handling.
- Update visualizer logic to enhance responsiveness and visual effects based on audio analysis, ensuring a cohesive user experience across components.
- Introduce the AssistantIdentity component to show the current assistant ID and provide a copy-to-clipboard feature.
- Update multiple sections of AssistantPage to include the AssistantIdentity component, enhancing user interaction with assistant IDs.
- Ensure the component handles the display of the assistant ID and provides feedback when copied successfully.
- Introduce three new audio visualizer components: AuraVisualizer, SpectrumVisualizer, and WaveVisualizer, enhancing the audio interaction experience.
- Replace the deprecated VoiceVisualizer with the new visualizers, ensuring a cohesive visual language across components.
- Update the AssistantPage to support dynamic visualization style switching, improving user engagement during audio interactions.
- Refactor DebugVoicePanel to accommodate the new visualizer props and enhance the overall debugging interface.
- Remove debug mode state management from AssistantPage, simplifying the component structure.
- Update DebugDrawer to eliminate mode selection, focusing on voice interaction features.
- Enhance the VoiceVisualizer component with improved visual effects and responsiveness to audio input.
- Adjust styles and layout for better user experience in the debugging interface.
- Introduce new fields for voice, speed, and language in the AssistantConfig and ProviderCredential models to support TTS and ASR configurations.
- Update the database schema and seeding script to accommodate the new fields, ensuring backward compatibility.
- Implement credential testing endpoints and logic to validate OpenAI-compatible credentials, enhancing user experience and reliability.
- Modify frontend components to include new fields in the credential forms and improve connection testing feedback.
- Refactor related services and API interactions to support the new credential testing feature.
- Introduce a new model field in the OpenCode form to manage language model selection.
- Refactor the form handling logic to improve data loading and error management for OpenCode assistants.
- Update UI components to utilize ResourceSelectField for model and voice configuration, enhancing user experience.
- Clear form fields when creating new OpenCode entries to ensure a fresh start for users.
- Update AssistantPage to use a stored value mask for API keys, improving security and user experience.
- Modify ComponentsModelsPage to display the current API key contextually, enhancing clarity for users.
- Adjust related components to ensure consistent handling of API key visibility and management.
- Update AssistantPage to handle API key input more securely by removing placeholder values and allowing empty submissions to retain existing keys.
- Introduce a new SecretInputField component for API key entry, improving user experience with visibility toggling and contextual hints.
- Modify ComponentsModelsPage to reflect similar API key handling, ensuring users can manage keys effectively while providing feedback on existing configurations.
- Add EditorBackButton for better navigation within the AssistantPage.
- Change 'DeepSeek-V3' to 'DeepSeek-Chat' and update its API key.
- Rename 'OpenAI TTS' to 'SiliconFlow-CosyVoice2-0.5B' and update its details.
- Add new models: 'SiliconFlow-TeleSpeechASR' and 'SiliconFlow-Qwen3-Embedding-4B' with corresponding API keys and configurations.
- Adjust existing entries to ensure consistency in the database seeding process.
- Update Makefile to include new database seed commands for assistants and credentials.
- Refactor assistant model to use explicit fields instead of a config dictionary, improving data integrity and clarity.
- Implement new seeding SQL script for assistants, ensuring dependencies on credentials are respected.
- Modify backend routes and frontend components to accommodate the new assistant structure, including direct field access for prompt, API URL, and keys.
- Enhance the AssistantPage component to handle the new data structure and streamline the save process for different assistant types.
