SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Tokenizer summary (#5467)

Browse Source 
* Work on tokenizer summary

* Finish tutorial

* Link to it

* Apply suggestions from code review

Co-authored-by: Anthony MOI <xn1t0x@gmail.com>
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* Add vocab definition

Co-authored-by: Anthony MOI <xn1t0x@gmail.com>
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
This commit is contained in:
Sylvain Gugger
2020-07-02 17:07:42 -04:00
committed by GitHub
parent ef0e9d806c
commit 6b735a7253
3 changed files with 247 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/source/index.rst
View File

@@ -142,6 +142,7 @@ conversion utilities for the following models:
preprocessing
training
model_sharing
tokenizer_summary
multilingual
.. toctree::