Arthur 085ea7e56c [CodeLlamaTokenizer] Nit, update __init__ to make sure the AddedTokens are not normalized because they are special (#27359) ...

* make sure tokens are properly initialized for codellama slow

* add m ore pretrained models

* style

* test more tokenizers checkpoints

2023-11-09 10:15:10 +01:00

..

__init__.py

[CodeLlama] Add support for CodeLlama (#25740)

2023-08-25 18:57:40 +02:00

test_tokenization_code_llama.py

[CodeLlamaTokenizer] Nit, update __init__ to make sure the AddedTokens are not normalized because they are special (#27359)

2023-11-09 10:15:10 +01:00