Problem with syllable tokenization

[p16i]

Describe the bug
I've observed a behavior that is worth being discussed here. In short, when there are some punctuations, syllable tokenizes would return some incorrect syllables, both from the default engine and ssg.

To Reproduce
Please see: https://colab.research.google.com/drive/12gxSmskjHCQzqV1-Nb4IOaBD-LJ0ARl5?usp=sharing

Expected behavior
imho, the expected result is ['หน้า', 'ที่', ' ', '19', '...'] . To achieve this, we can split the sentence by punctuation first then do syllable tokenization for each part.

What do you think?

[wannaphong]

I think we should report this problem to ssg.

[p16i]

that's true but we also see here that the default engine doesn't split 19... either.

[bact]

Since syllable doesn't involve anything with punctuation (unlike word that can have punctuation like "โควิด-19" "ก.พ.), i think it's safe to split at punctuations first then do the syllable tokenization

[wannaphong]

OK. We will remove syllable_tokenize in 2.4 (move to subword_tokenize)

[bact]

that's true but we also see here that the default engine doesn't split 19... either.

Current newmm engine will group that together, due to this:

pythainlp/pythainlp/tokenize/newmm.py

Lines 104 to 106 in 7497f1b

	m = _PAT_NONTHAI.match(text[begin_pos:])
	if m: # non-Thai token, skip to the end
	end_pos = begin_pos + m.end()

_PAT_NONTHAI is

pythainlp/pythainlp/tokenize/newmm.py

Lines 25 to 33 in 7497f1b

	# match non-Thai tokens
	_PAT_NONTHAI = re.compile(
	r"""(?x)
	[-a-zA-Z]+| # Latin characters
	\d[\d,\.]*| # number
	[ \t]+| # space
	\r?\n # newline
	"""
	)

I agree with @heytitle that we probably need to think about 19... case.

Our current tokenization is focus on Thai word tokenization and probably not cover all the things we should do with punctuation, etc, in general.

[bact]

Fixed for newmm.

For ssg, I filed a new issue at ponrawee/ssg#8