SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Add token type ids to CodeGenTokenizer (#29265)

Browse Source 
* Add create token type ids to CodeGenTokenizer

* Fix inconsistent length of token type ids

* Format source codes

* Fix inconsistent order of methods

* Update docstring

* add test_tokenizer_integration test

* Format source codes

* Add `copied from` comment to CodeGenTokenizerFast

* Add doc of create_token_type_ids_from_sequences

* Make return_token_type_ids False by default

* Make test_tokenizer_integration as slow test

* Add return_token_type_ids to tokenizer init arg

* Add test for tokenizer's init return_token_type_ids

* Format source codes
This commit is contained in:
st81
2024-04-17 19:19:18 +09:00
committed by GitHub
parent 812a5de229
commit 8d6b509611
4 changed files with 127 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/source/en/model_doc/codegen.md
View File

@@ -72,6 +72,7 @@ hello_world()
## CodeGenTokenizer
[[autodoc]] CodeGenTokenizer
- create_token_type_ids_from_sequences
- save_vocabulary
## CodeGenTokenizerFast