Add CHANGELOG entry describing LLM switcher
This commit is contained in:
42
CHANGELOG.md
42
CHANGELOG.md
@@ -13,7 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
`LLMContextAggregatorPair`, which will eventually replace `OpenAILLMContext`
|
||||
(and the other under-the-hood contexts) and the other context aggregators.
|
||||
The new universal `LLMContext` machinery allows a single context to be shared
|
||||
between different LLMs, enabling scenarios like LLM failover.
|
||||
between different LLMs, enabling runtime LLM switching and scenarios like
|
||||
failover.
|
||||
|
||||
From the developer's point of view, switching to using the new universal
|
||||
context machinery will usually be a matter of going from this:
|
||||
@@ -36,6 +37,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- `OpenAILLMService`
|
||||
- `GoogleLLMService`
|
||||
|
||||
- Added a new `LLMSwitcher` class to enable runtime LLM switching, built atop a
|
||||
new generic `ServiceSwitcher`.
|
||||
|
||||
Switchers take a switching strategy. The first available strategy is
|
||||
`ServiceSwitcherStrategyManual`.
|
||||
|
||||
To switch LLMs at runtime, the LLMs must be sharing one instance of the new
|
||||
universal `LLMContext` (see above bullet).
|
||||
|
||||
```python
|
||||
# Instantiate your LLM services
|
||||
llm_openai = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
|
||||
llm_google = GoogleLLMService(api_key=os.getenv("GOOGLE_API_KEY"))
|
||||
|
||||
# Instantiate a switcher
|
||||
# (ServiceSwitcherStrategyManual default to OpenAI, as it's first in the list)
|
||||
llm_switcher = LLMSwitcher(
|
||||
llms=[llm_openai, llm_google], strategy_type=ServiceSwitcherStrategyManual
|
||||
)
|
||||
|
||||
# Create your pipeline
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
transport.input(),
|
||||
stt,
|
||||
context_aggregator.user(),
|
||||
llm_switcher,
|
||||
tts,
|
||||
transport.output(),
|
||||
context_aggregator.assistant(),
|
||||
]
|
||||
)
|
||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
||||
|
||||
# ...
|
||||
# Whenever is appropriate, switch LLMs!
|
||||
await task.queue_frames([ManuallySwitchServiceFrame(service=llm_google)])
|
||||
```
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed a `CartesiaTTSService` issue that was causing the application to hang
|
||||
|
||||
Reference in New Issue
Block a user