Add CHANGELOG entry describing LLM switcher

This commit is contained in:
Paul Kompfner
2025-08-25 15:03:38 -04:00
parent 98dc891640
commit 8181962236

View File

@@ -13,7 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`LLMContextAggregatorPair`, which will eventually replace `OpenAILLMContext`
(and the other under-the-hood contexts) and the other context aggregators.
The new universal `LLMContext` machinery allows a single context to be shared
between different LLMs, enabling scenarios like LLM failover.
between different LLMs, enabling runtime LLM switching and scenarios like
failover.
From the developer's point of view, switching to using the new universal
context machinery will usually be a matter of going from this:
@@ -36,6 +37,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `OpenAILLMService`
- `GoogleLLMService`
- Added a new `LLMSwitcher` class to enable runtime LLM switching, built atop a
new generic `ServiceSwitcher`.
Switchers take a switching strategy. The first available strategy is
`ServiceSwitcherStrategyManual`.
To switch LLMs at runtime, the LLMs must be sharing one instance of the new
universal `LLMContext` (see above bullet).
```python
# Instantiate your LLM services
llm_openai = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
llm_google = GoogleLLMService(api_key=os.getenv("GOOGLE_API_KEY"))
# Instantiate a switcher
# (ServiceSwitcherStrategyManual default to OpenAI, as it's first in the list)
llm_switcher = LLMSwitcher(
llms=[llm_openai, llm_google], strategy_type=ServiceSwitcherStrategyManual
)
# Create your pipeline
pipeline = Pipeline(
[
transport.input(),
stt,
context_aggregator.user(),
llm_switcher,
tts,
transport.output(),
context_aggregator.assistant(),
]
)
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
# ...
# Whenever is appropriate, switch LLMs!
await task.queue_frames([ManuallySwitchServiceFrame(service=llm_google)])
```
### Fixed
- Fixed a `CartesiaTTSService` issue that was causing the application to hang