Ita Zaporozhets 3f03c379d2 fix tiktoken convert to pass AddedToken to Tokenizer (#36566) ...

* pass AddedToken to Tokenizer

* ruff

* handle dict for special tokens

* option: test tokenizer from tiktoken same as fast

* ruff

* ruff

2025-03-20 11:26:49 +01:00

..

__init__.py

Move test model folders (#17034)

2022-05-03 14:42:02 +02:00

test_modeling_flax_gpt2.py

[tests] remove flax-pt equivalence and cross tests (#36283)

2025-02-19 15:13:27 +00:00

test_modeling_gpt2.py

Fix typos in tests (#36547)

2025-03-05 15:04:06 -08:00

test_modeling_tf_gpt2.py

Remove static pretrained maps from the library's internals (#29112)

2024-03-25 10:33:38 +01:00

test_tokenization_gpt2_tf.py

update ruff version (#30932)

2024-05-22 06:40:15 +02:00

test_tokenization_gpt2.py

fix tiktoken convert to pass AddedToken to Tokenizer (#36566)

2025-03-20 11:26:49 +01:00