wx44wx/pipecat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #3617 from pipecat-ai/fix/cjk-sentence-splitting

Browse Source 
Fix sentence splitting for CJK and other non-Latin languages
This commit is contained in:
Mark Backman
2026-02-02 18:16:51 -05:00
committed by GitHub
parent ca49acfaa6 774041e9a1
commit a4e499dc80
4 changed files with 123 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
changelog/3617.fixed.md Normal file
View File

@@ -0,0 +1,5 @@
- Fixed sentence splitting for Japanese, Chinese, Korean, and other non-Latin
languages in TTS pipeline. NLTK's sentence tokenizer does not support CJK
languages, causing text to accumulate until flush instead of being split at
sentence boundaries. Added fallback detection for unambiguous non-Latin
sentence-ending punctuation (e.g., `。`, `？`, `！`).