3 Commits

Author	SHA1	Message	Date
cyyever	1e6b546ea6	Use Python 3.9 syntax in tests (#37343) ... Signed-off-by: cyy <cyyever@outlook.com>	2025-04-08 14:12:08 +02:00
Yih-Dar	1fcaad6df9	Use lru_cache for tokenization tests (#36818) ... * fix * fix * fix * fix --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2025-03-28 15:09:35 +01:00
bastrob	464d986b6c	Add missing Flaubert tokenizer tests (#30492) ... * add flaubert tokenization test, enrich inheritance in FlaubertTokenizer. * fix quality code ci * ensure parameter consistency * fix ci * fix copyright year and flatten vocab list. * fix style	2024-06-05 13:52:16 +02:00