Aleix Conchillo Flaqué
b468b2f926
audio: clamp normalized volume
2024-05-22 13:04:09 -07:00
Aleix Conchillo Flaqué
af1c7d0023
interruptions: introduce pyloudnorm to compute loudness
...
https://github.com/csteinmetz1/pyloudnorm
2024-05-22 11:52:07 -07:00
Aleix Conchillo Flaqué
979739c1b7
processors: reset aggergator before pushing
2024-05-22 11:26:08 -07:00
Aleix Conchillo Flaqué
57a568986a
processors: only interrupt asssisstant
...
We were pushing interruption frames in the audio task. This was caussing the
LLMUserResponseAggregator to push the accumulated text and then casuing the LLM
to respond.
2024-05-22 10:15:35 -07:00
Aleix Conchillo Flaqué
825738440e
transports: run threads in their own ThreadPoolExecutor
2024-05-21 18:52:27 -07:00
Aleix Conchillo Flaqué
40ccf46b4b
aggregators: accumulate full responses and take interruptions into account
2024-05-20 11:40:57 -07:00
Aleix Conchillo Flaqué
e4c990c677
aggregators: expose LLM messages
2024-05-20 10:51:37 -07:00
Aleix Conchillo Flaqué
83812f2671
transports(daily): implement DailyOutputTransport.send_message
2024-05-20 10:30:59 -07:00
Kwindla Hultman Kramer
7ffb10d7f5
add to CHANGELOG.md
2024-05-19 12:44:45 -07:00
Kwindla Hultman Kramer
66377954cb
fix up openai vision and gemini implementation
2024-05-19 12:33:57 -07:00
Kwindla Hultman Kramer
e507686cef
oops, fix openai.py
2024-05-19 11:13:39 -07:00
Kwindla Hultman Kramer
cf597a2f6b
add back in debug log line in openai.py
2024-05-19 11:08:38 -07:00
Kwindla Hultman Kramer
b337e984b3
Initial commit of Google Gemini LLM service.
...
Gemini text input works. We translate from OpenAILLMContext format
on the fly in the GoogleLLMService implementation. This commit also
implements image input (vision) in both the GoogleLLMService and in
the OpenAILLMService. Image input is a hack and needs to be revisited.
OpenAI expects images to be uploaded as base64-encoded JPEGs. Google
does not require the base64 encoding. Other than for images, we use
the OpenAI format as our standard, but base64-encoding the images
and then unencoding them in the GoogleLLMService feels wasteful.
2024-05-19 10:35:20 -07:00
Aleix Conchillo Flaqué
c3bfcbd562
aggregators: clear accumulated responses if interruption happens
2024-05-19 10:21:45 -07:00
Aleix Conchillo Flaqué
810dc30d3d
examples: fix examples to use LLMFullResponseEndFrame
2024-05-19 09:39:34 -07:00
Aleix Conchillo Flaqué
435fffe1b0
add LLMFullResponseStartFrame/LLMFullResponseEndFrame
2024-05-18 09:49:38 -07:00
Aleix Conchillo Flaqué
2b8f1c4cda
services(openai): send LLMResponseStartFrame for each completion
2024-05-17 17:47:33 -07:00
Aleix Conchillo Flaqué
0e8c7a9b28
transports(output): create an downstream push frame task
2024-05-17 17:47:24 -07:00
Aleix Conchillo Flaqué
3e13678f23
vad: use exponential smoothed volume to improve speech detection
2024-05-17 17:13:31 -07:00
Aleix Conchillo Flaqué
455ec4f1fd
services(tts): always send received TextFrame downstream
2024-05-17 17:11:11 -07:00
Aleix Conchillo Flaqué
de65028061
vad: reduce default confidence back to 0.5
2024-05-17 14:39:40 -07:00
Aleix Conchillo Flaqué
34762bf604
transports: allows update allow_interruptinos when receiving StartFrame
2024-05-17 14:15:37 -07:00
Aleix Conchillo Flaqué
57121338b1
pipeline(task): cleanup processors only if we need to
2024-05-17 13:53:33 -07:00
Aleix Conchillo Flaqué
a5d246ec0c
vad: use exponential smoothing to avoid sudden changes
2024-05-17 13:53:33 -07:00
Aleix Conchillo Flaqué
f2cefeeedc
utils: move exp_smoothing to utils module
2024-05-17 13:52:18 -07:00
Aleix Conchillo Flaqué
537e72a05f
vad: introduce VADParams so you can tweak things
2024-05-17 13:52:18 -07:00
Aleix Conchillo Flaqué
efa5a061d7
silero: simplify int16 -> float32 conversion
2024-05-17 13:51:06 -07:00
Aleix Conchillo Flaqué
0bef44c2ff
introduce StartInterruptionFrame and StopInterruptionFrame
2024-05-17 13:51:06 -07:00
Aleix Conchillo Flaqué
f62fe059b1
fix issues with Ctrl-C tasks cancellation
2024-05-17 13:51:04 -07:00
Aleix Conchillo Flaqué
f432e2b17e
transports: allow adding a vad analyzer to BaseInputTransport
2024-05-17 13:50:48 -07:00
Aleix Conchillo Flaqué
dc9377fb92
add missing queue task_done()
2024-05-17 13:50:48 -07:00
Aleix Conchillo Flaqué
7384b63b1d
initial interruptions support
2024-05-17 13:50:45 -07:00
Aleix Conchillo Flaqué
add8d3cbaf
transport: create input transports push frame task
2024-05-16 16:54:39 -07:00
Aleix Conchillo Flaqué
0da427e127
transports(daily): don't send camera/audio tracks if not enabled
2024-05-16 08:16:39 -07:00
Aleix Conchillo Flaqué
9447b32f3e
transports(daily): on_app_message doesn't need to be event handler
2024-05-15 17:06:47 -07:00
Aleix Conchillo Flaqué
af10adb7fe
some minor event loop updates
2024-05-15 17:00:43 -07:00
Aleix Conchillo Flaqué
129acf886f
transports(daily): hot fix for receiving transport messages
2024-05-15 17:00:04 -07:00
Aleix Conchillo Flaqué
9e22a8b4ff
transports(daily): add receiving transport messages
2024-05-15 15:59:08 -07:00
Aleix Conchillo Flaqué
28da747f19
transports(daily): fix on_participant_left event
2024-05-15 15:40:31 -07:00
Aleix Conchillo Flaqué
3d6783ddb0
transports: resize output image if it doesn't match camera
2024-05-15 15:36:20 -07:00
Aleix Conchillo Flaqué
349fc526d7
transports(daily): avoid locking if no participant has joined yet
2024-05-15 15:24:58 -07:00
Aleix Conchillo Flaqué
acf6dc0a30
transports: more start and stop fixes
2024-05-15 15:23:03 -07:00
Aleix Conchillo Flaqué
3563e66ff6
transports(daily): add on_participant_left event
2024-05-15 15:20:37 -07:00
Aleix Conchillo Flaqué
86feb1e104
services: fix DailyTransport stop/cleanup ordering
2024-05-14 19:24:55 -07:00
Aleix Conchillo Flaqué
754c1c6775
services: fixed DailyTransport output camera and audio
2024-05-14 19:07:19 -07:00
Aleix Conchillo Flaqué
0b01eb5a11
services: pass **kwargs to TTService
2024-05-14 18:46:03 -07:00
Aleix Conchillo Flaqué
6247b9df39
services: fix STTService and WhisperSTTService
2024-05-14 18:45:40 -07:00
Aleix Conchillo Flaqué
bd5344c892
services: MoondreamService model_id argument is now model
2024-05-14 18:34:10 -07:00
Aleix Conchillo Flaqué
e4fe54cd7f
vad: rename VADAnalyzer arguments
2024-05-14 18:33:17 -07:00
Aleix Conchillo Flaqué
5147f4086e
transports(daily): add DailyTranscriptionSettings to update settings easier
2024-05-14 14:49:30 -07:00
