Test major functions with the Big List of Naughty Strings

[bact]

Detailed description

Add test with strings from the Big List of Naughty Strings, to test robustness of the library.

Context

The Big List of Naughty String is "an evolving list of strings which have a high probability of causing issues when used as user-input data." For example, a string with zero-width space (U+200B).

As a text processing library that has to deal with strings of all sorts, both from user-input and from data archive, it is expected that the library should be robust enough to handle variety of character combinations.

Possible implementation

• Run major functions through the string list
• First step, aim the lib not to break and return a correct output type in a reasonable time.
• Correctness is not the goal for now (as there's no definition of correctness yet).

[pavaris-pm]

@bact i think that making a library to be robust enough to handle variety of character combinations is a very great idea!. Did you mean that adding strings from the Big List of Naughty Strings as an additional test case with all related functions inside pythainlp/tests/ directory which may cause an issue when encounter with Naughty Strings like tokenize or translate engine ?

[bact]

@pavaris-pm Exactly. We can start with tokenize tests.

[pavaris-pm]

@pavaris-pm Exactly. We can start with tokenize tests.

Cool! i will try with that first. Do we need to test will all naughty string in that repo? or just some sample of it is ok. Since the Big List of Naughty Strings repo itself has a blns.txt file that divided it into categories like reverse string , special characters, etc. According to that, we can choose about 2-3 strings in each categories in order to test a library with each categories. What do you think? @bact

[bact]

blns.txt contains 496 test strings (I use egrep -cv '#|^\s*$' blns.txt). Maybe not too big to test all?

But if it takes too long time in the test (which may affect productivity) we can focus on relevant categories.

I would say these categories are more relevant

Group 1: (non-)whitespaces and control characters - as they occurred a lot and sometimes our regular expressions may not well covered them:

#       Special Characters
# ASCII punctuation. 
# Non-whitespace C0 controls:
# Non-whitespace C1 controls:
# Whitespace:
# Unicode additional control characters:
# "Byte order marks",

Group 2: string-length related: some non-careful string manipulation may breaks some of these strings. For this group, I think the expected behavior for the testing is for any f(), len(str) should equal to len(f(str)). But I may wrong.

#       Unicode Symbols
#	Two-Byte Characters
#	Strings which contain two-byte letters: 
#	Special Unicode Characters Union
#	Changing length when lowercased
#	Japanese Emoticons
#	Emoji