Files
HuggingFace_transformer/docs/source/en/model_doc
st81 8d6b509611 Add token type ids to CodeGenTokenizer (#29265)
* Add create token type ids to CodeGenTokenizer

* Fix inconsistent length of token type ids

* Format source codes

* Fix inconsistent order of methods

* Update docstring

* add test_tokenizer_integration test

* Format source codes

* Add `copied from` comment to CodeGenTokenizerFast

* Add doc of create_token_type_ids_from_sequences

* Make return_token_type_ids False by default

* Make test_tokenizer_integration as slow test

* Add return_token_type_ids to tokenizer init arg

* Add test for tokenizer's init return_token_type_ids

* Format source codes
2024-04-17 12:19:18 +02:00
..
2024-03-11 17:26:38 +00:00
2023-11-06 19:45:03 +00:00
2023-11-10 13:49:10 +00:00
2024-04-16 11:58:55 +02:00
2024-03-15 14:29:11 +01:00
2023-11-23 17:44:08 +00:00
2023-11-23 17:44:08 +00:00
2024-03-12 10:16:21 +00:00
2024-02-21 14:21:28 +01:00
2024-04-15 17:03:03 +01:00
2023-10-30 21:42:19 +01:00
2023-12-20 14:25:07 +05:30
2023-11-13 14:20:54 +01:00
2024-02-19 15:22:29 +01:00
2024-02-19 15:22:29 +01:00
2024-03-13 19:05:20 +00:00
2024-02-23 10:43:31 +01:00
2023-11-06 19:45:03 +00:00
2023-07-13 11:46:54 -04:00
2024-02-19 15:22:29 +01:00
2024-02-19 15:22:29 +01:00
2023-11-06 19:45:03 +00:00
2023-11-23 17:02:16 +00:00
2023-12-15 20:16:47 +01:00
2024-02-19 15:22:29 +01:00