feat: support new async-tool mechanism in AWS Nova Sonic
Add support to AWSNovaSonicLLMService for the new "async tool call" mechanism activated by `cancel_on_interruption=False`, which includes: - delivering results asynchronously - delivering result streams - cancelling running async tools Note that the introduction of the new mechanism had actually caused a regression in AWS Nova Sonic, which previously supported `cancel_on_interruption=False` with the old mechanism (simply avoiding discarding tool calls on interruptions). Support for the other major realtime services (`GeminiLiveLLMService`, `OpenAIRealtimeLLMService`) will follow in a separate PR — Gemini Live in particular needs more work before it can support long-running tool calls reliably.
This commit is contained in:
@@ -5,7 +5,6 @@
|
||||
#
|
||||
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import random
|
||||
from datetime import datetime
|
||||
@@ -46,11 +45,6 @@ async def fetch_weather_from_api(params: FunctionCallParams):
|
||||
if params.arguments["format"] == "fahrenheit"
|
||||
else random.randint(15, 30)
|
||||
)
|
||||
# Simulate a long network delay.
|
||||
# You can continue chatting while waiting for this to complete.
|
||||
# With Nova 2 Sonic (the default model), the assistant will respond
|
||||
# appropriately once the function call is complete.
|
||||
await asyncio.sleep(5)
|
||||
await params.result_callback(
|
||||
{
|
||||
"conditions": "nice",
|
||||
@@ -150,9 +144,7 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
||||
# Register function for function calls
|
||||
# you can either register a single function for all function calls, or specific functions
|
||||
# llm.register_function(None, fetch_weather_from_api)
|
||||
llm.register_function(
|
||||
"get_current_weather", fetch_weather_from_api, cancel_on_interruption=False
|
||||
)
|
||||
llm.register_function("get_current_weather", fetch_weather_from_api)
|
||||
|
||||
# Set up context and context management.
|
||||
context = LLMContext(tools=tools)
|
||||
|
||||
Reference in New Issue
Block a user