Add CHANGELOG entry describing LLM switcher
This commit is contained in:
42
CHANGELOG.md
42
CHANGELOG.md
@@ -13,7 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
`LLMContextAggregatorPair`, which will eventually replace `OpenAILLMContext`
|
`LLMContextAggregatorPair`, which will eventually replace `OpenAILLMContext`
|
||||||
(and the other under-the-hood contexts) and the other context aggregators.
|
(and the other under-the-hood contexts) and the other context aggregators.
|
||||||
The new universal `LLMContext` machinery allows a single context to be shared
|
The new universal `LLMContext` machinery allows a single context to be shared
|
||||||
between different LLMs, enabling scenarios like LLM failover.
|
between different LLMs, enabling runtime LLM switching and scenarios like
|
||||||
|
failover.
|
||||||
|
|
||||||
From the developer's point of view, switching to using the new universal
|
From the developer's point of view, switching to using the new universal
|
||||||
context machinery will usually be a matter of going from this:
|
context machinery will usually be a matter of going from this:
|
||||||
@@ -36,6 +37,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
- `OpenAILLMService`
|
- `OpenAILLMService`
|
||||||
- `GoogleLLMService`
|
- `GoogleLLMService`
|
||||||
|
|
||||||
|
- Added a new `LLMSwitcher` class to enable runtime LLM switching, built atop a
|
||||||
|
new generic `ServiceSwitcher`.
|
||||||
|
|
||||||
|
Switchers take a switching strategy. The first available strategy is
|
||||||
|
`ServiceSwitcherStrategyManual`.
|
||||||
|
|
||||||
|
To switch LLMs at runtime, the LLMs must be sharing one instance of the new
|
||||||
|
universal `LLMContext` (see above bullet).
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Instantiate your LLM services
|
||||||
|
llm_openai = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
|
||||||
|
llm_google = GoogleLLMService(api_key=os.getenv("GOOGLE_API_KEY"))
|
||||||
|
|
||||||
|
# Instantiate a switcher
|
||||||
|
# (ServiceSwitcherStrategyManual default to OpenAI, as it's first in the list)
|
||||||
|
llm_switcher = LLMSwitcher(
|
||||||
|
llms=[llm_openai, llm_google], strategy_type=ServiceSwitcherStrategyManual
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create your pipeline
|
||||||
|
pipeline = Pipeline(
|
||||||
|
[
|
||||||
|
transport.input(),
|
||||||
|
stt,
|
||||||
|
context_aggregator.user(),
|
||||||
|
llm_switcher,
|
||||||
|
tts,
|
||||||
|
transport.output(),
|
||||||
|
context_aggregator.assistant(),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
|
# ...
|
||||||
|
# Whenever is appropriate, switch LLMs!
|
||||||
|
await task.queue_frames([ManuallySwitchServiceFrame(service=llm_google)])
|
||||||
|
```
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
- Fixed a `CartesiaTTSService` issue that was causing the application to hang
|
- Fixed a `CartesiaTTSService` issue that was causing the application to hang
|
||||||
|
|||||||
Reference in New Issue
Block a user