Matt d15781597a Allow add_tokens for ESM (#28535) ...

* Allow non-special tokens to be added

* Add test, fix token adding code

* Revert changes to id_to_token and token_to_id

* Update the ESM tokenizer to be a bit more standardized

* Update src/transformers/models/esm/tokenization_esm.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

---------

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

2024-01-19 12:32:05 +00:00

..

__init__.py

Rebase ESM PR and update all file formats (#19055)

2022-09-30 14:16:25 +01:00

test_modeling_esm.py

added support for gradient checkpointing in ESM models (#26386)

2023-09-26 10:15:53 +02:00

test_modeling_esmfold.py

CI with num_hidden_layers=2 🚀🚀🚀 (#25266)

2023-08-02 20:22:36 +02:00

test_modeling_tf_esm.py

Speed up TF tests by reducing hidden layer counts (#24595)

2023-06-30 16:30:33 +01:00

test_tokenization_esm.py

Allow add_tokens for ESM (#28535)

2024-01-19 12:32:05 +00:00