utils: add support for ellipses in match_endofsentence()

This commit is contained in:
Aleix Conchillo Flaqué
2025-02-21 09:54:11 -08:00
parent 4acf7296e0
commit 12bce2e8c0
3 changed files with 10 additions and 6 deletions

View File

@@ -13,8 +13,8 @@ ENDOFSENTENCE_PATTERN_STR = r"""
(?<!Mr|Ms|Dr) # Negative lookbehind: not preceded by Mr, Ms, Dr (combined bc. length is the same)
(?<!Mrs) # Negative lookbehind: not preceded by "Mrs"
(?<!Prof) # Negative lookbehind: not preceded by "Prof"
[\.\?\!;]| # Match a period, question mark, exclamation point, or semicolon
[。?!;।] # the full-width version (mainly used in East Asian languages such as Chinese, Hindi)
(\.\s*\.\s*\.|[\.\?\!;])| # Match a period, question mark, exclamation point, or semicolon
(\\s*\\s*\。|[。?!;।]) # the full-width version (mainly used in East Asian languages such as Chinese, Hindi)
$ # End of string
"""
ENDOFSENTENCE_PATTERN = re.compile(ENDOFSENTENCE_PATTERN_STR, re.VERBOSE)