SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

[Marian] documentation and AutoModel support (#4152)

Browse Source 
- MarianSentencepieceTokenizer - > MarianTokenizer
- Start using unk token.
- add docs page
- add better generation params to MarianConfig
- more conversion utilities
This commit is contained in:
Sam Shleifer
2020-05-10 13:54:57 -04:00
committed by GitHub
parent 9d2f467bfb
commit 3487be75ef
14 changed files with 355 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/source/index.rst
View File

@@ -108,3 +108,4 @@ The library currently contains PyTorch and Tensorflow implementations, pre-train
model_doc/electra
model_doc/dialogpt
model_doc/reformer
model_doc/marian