SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
Files
f284089ec4cf6da2d659512b09f2cf526dc6b308
HuggingFace_transformer/src/transformers/models
History
Mehrad Moradshahi b880508440 tokenization_marian.py: use current_spm for decoding (#10357) 
* Fix Marian decoding

Tokenizer's decode and batch_decode now accepts a new argument (use_source_tokenizer) which indicates whether the source spm should be used to decode ids. This is useful for Marian models specificallly when decoding source input ids.

* Adapt docstrings

Co-authored-by: Sylvain Gugger <sylvain.gugger@gmail.com>
2021-03-08 08:14:31 -05:00
..
albert
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
auto
Add m2m100 (#10236)
2021-03-06 22:14:16 +05:30
bart
fix BART Summarization example in doc (#10582)
2021-03-08 15:45:06 +05:30
barthez
Authorize last version of tokenizer (#9799)
2021-02-04 14:18:33 -05:00
bert
fix tf doc bug (#10570)
2021-03-07 22:31:50 -05:00
bert_generation
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
bert_japanese
Fast imports part 3 (#9474)
2021-01-08 07:40:59 -05:00
bertweet
Fast imports part 3 (#9474)
2021-01-08 07:40:59 -05:00
blenderbot
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
blenderbot_small
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
bort
ADD BORT (#9813)
2021-01-27 21:25:11 +03:00
camembert
Authorize last version of tokenizer (#9799)
2021-02-04 14:18:33 -05:00
convbert
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
ctrl
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
deberta
Fix embeddings for PyTorch 1.8 (#10549)
2021-03-05 16:18:48 -05:00
deberta_v2
Fix embeddings for PyTorch 1.8 (#10549)
2021-03-05 16:18:48 -05:00
dialogpt
Fix imports in conversion scripts (#9674)
2021-01-19 09:40:15 -05:00
distilbert
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
dpr
[PretrainedFeatureExtractor] + Wav2Vec2FeatureExtractor, Wav2Vec2Processor, Wav2Vec2Tokenizer (#10324)
2021-02-25 17:42:46 +03:00
electra
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
encoder_decoder
Fast imports part 3 (#9474)
2021-01-08 07:40:59 -05:00
flaubert
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
fsmt
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
funnel
Fix embeddings for PyTorch 1.8 (#10549)
2021-03-05 16:18:48 -05:00
gpt2
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
herbert
Fast imports part 3 (#9474)
2021-01-08 07:40:59 -05:00
ibert
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
layoutlm
Fix embeddings for PyTorch 1.8 (#10549)
2021-03-05 16:18:48 -05:00
led
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
longformer
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
lxmert
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
m2m_100
[M2M100] fix positional embeddings (#10590)
2021-03-08 16:06:19 +05:30
marian
tokenization_marian.py: use current_spm for decoding (#10357)
2021-03-08 08:14:31 -05:00
mbart
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
mmbt
Fast imports part 3 (#9474)
2021-01-08 07:40:59 -05:00
mobilebert
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
mpnet
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
mt5
Deprecate prepare_seq2seq_batch (#10287)
2021-02-22 12:36:16 -05:00
openai
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
pegasus
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
phobert
Fast imports part 3 (#9474)
2021-01-08 07:40:59 -05:00
prophetnet
[ProphetNet] Bart-like Refactor (#10501)
2021-03-04 23:27:12 +03:00
rag
[PretrainedFeatureExtractor] + Wav2Vec2FeatureExtractor, Wav2Vec2Processor, Wav2Vec2Tokenizer (#10324)
2021-02-25 17:42:46 +03:00
reformer
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
retribert
Fix embeddings for PyTorch 1.8 (#10549)
2021-03-05 16:18:48 -05:00
roberta
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
squeezebert
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
t5
[T5] Fix speed degradation bug t5 (#10496)
2021-03-03 12:42:41 +03:00
tapas
Fix embeddings for PyTorch 1.8 (#10549)
2021-03-05 16:18:48 -05:00
transfo_xl
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
wav2vec2
tokenization_marian.py: use current_spm for decoding (#10357)
2021-03-08 08:14:31 -05:00
xlm
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
xlm_prophetnet
Add from_slow in fast tokenizers build and fixes some bugs (#9987)
2021-02-04 03:34:23 -05:00
xlm_roberta
Authorize last version of tokenizer (#9799)
2021-02-04 14:18:33 -05:00
xlnet
Refactoring checkpoint names for multiple models (#10527)
2021-03-05 18:06:55 -05:00
__init__.py
Add m2m100 (#10236)
2021-03-06 22:14:16 +05:30