Files
HuggingFace_transformer/src/transformers/models
Mehrad Moradshahi b880508440 tokenization_marian.py: use current_spm for decoding (#10357)
* Fix Marian decoding

Tokenizer's decode and batch_decode now accepts a new argument (use_source_tokenizer) which indicates whether the source spm should be used to decode ids. This is useful for Marian models specificallly when decoding source input ids.

* Adapt docstrings

Co-authored-by: Sylvain Gugger <sylvain.gugger@gmail.com>
2021-03-08 08:14:31 -05:00
..
2021-03-06 22:14:16 +05:30
2021-03-07 22:31:50 -05:00
2021-01-08 07:40:59 -05:00
2021-01-08 07:40:59 -05:00
2021-01-27 21:25:11 +03:00
2021-01-08 07:40:59 -05:00
2021-01-08 07:40:59 -05:00
2021-01-08 07:40:59 -05:00
2021-03-06 22:14:16 +05:30