SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
Files
66a5a6fda8fdb71b35d76d98d8908eb2cc586e00
HuggingFace_transformer/src/transformers
History
Ashwin Geet Dsa 66a5a6fda8 fix to ensure that returned tensors after the tokenization is Long (#7039) 
* fix to ensure that returned tensors after the tokenization is Long

* fix to ensure that returned tensors after the tokenization is Long

Co-authored-by: Ashwin Geet Dsa <adsa@grvingt-6.nancy.grid5000.fr>
2020-09-10 11:04:03 -04:00
..
benchmark
Black 20 release
2020-08-26 17:20:22 +02:00
commands
add -y to bypass prompt for transformers-cli upload (#7035)
2020-09-10 04:58:29 -04:00
data
fix to ensure that returned tensors after the tokenization is Long (#7039)
2020-09-10 11:04:03 -04:00
utils
adding TRANSFORMERS_VERBOSITY env var (#6961)
2020-09-09 04:08:01 -04:00
__init__.py
Add TF Funnel Transformer (#7029)
2020-09-10 10:41:56 -04:00
activations.py
Black 20 release
2020-08-26 17:20:22 +02:00
configuration_albert.py
Black 20 release
2020-08-26 17:20:22 +02:00
configuration_auto.py
Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models. (#6594)
2020-09-10 16:40:51 +02:00
configuration_bart.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
configuration_bert_generation.py
Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models. (#6594)
2020-09-10 16:40:51 +02:00
configuration_bert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
configuration_camembert.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
configuration_ctrl.py
Black 20 release
2020-08-26 17:20:22 +02:00
configuration_distilbert.py
Black 20 release
2020-08-26 17:20:22 +02:00
configuration_dpr.py
Black 20 release
2020-08-26 17:20:22 +02:00
configuration_electra.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
configuration_encoder_decoder.py
Black 20 release
2020-08-26 17:20:22 +02:00
configuration_flaubert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
configuration_funnel.py
Funnel transformer (#6908)
2020-09-08 08:08:08 -04:00
configuration_gpt2.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
configuration_longformer.py
Black 20 release
2020-08-26 17:20:22 +02:00
configuration_lxmert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
configuration_marian.py
Kill model archive maps (#4636)
2020-06-02 09:39:33 -04:00
configuration_mbart.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
configuration_mmbt.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
configuration_mobilebert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
configuration_openai.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
configuration_pegasus.py
Pegasus finetune script: add --adafactor (#6811)
2020-08-29 17:43:32 -04:00
configuration_reformer.py
Fixed the default number of attention heads in Reformer Configuration (#6973)
2020-09-07 12:12:22 +02:00
configuration_retribert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
configuration_roberta.py
Black 20 release
2020-08-26 17:20:22 +02:00
configuration_t5.py
Black 20 release
2020-08-26 17:20:22 +02:00
configuration_transfo_xl.py
Black 20 release
2020-08-26 17:20:22 +02:00
configuration_utils.py
[from_pretrained] Allow tokenizer_type ≠ model_type (#6995)
2020-09-09 04:22:59 -04:00
configuration_xlm_roberta.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
configuration_xlm.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
configuration_xlnet.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
convert_albert_original_tf_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_bart_original_pytorch_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_bert_original_tf2_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_bert_original_tf_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_bert_pytorch_checkpoint_to_original_tf.py
GPU text generation: mMoved the encoded_prompt to correct device
2020-01-06 15:11:12 +01:00
convert_dialogpt_original_pytorch_checkpoint_to_pytorch.py
Black 20 release
2020-08-26 17:20:22 +02:00
convert_dpr_original_checkpoint_to_pytorch.py
Add DPR model (#5279)
2020-07-07 08:56:12 -04:00
convert_electra_original_tf_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_funnel_original_tf_checkpoint_to_pytorch.py
Funnel transformer (#6908)
2020-09-08 08:08:08 -04:00
convert_gpt2_original_tf_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_graph_to_onnx.py
Black 20 release
2020-08-26 17:20:22 +02:00
convert_longformer_original_pytorch_lightning_to_pytorch.py
[Longformer For Question Answering] Conversion script, doc, small fixes (#4593)
2020-05-26 14:58:47 +02:00
convert_lxmert_original_tf_checkpoint_to_pytorch.py
Adding the LXMERT pretraining model (MultiModal languageXvision) to HuggingFace's suite of models (#5793)
2020-09-03 04:02:25 -04:00
convert_marian_to_pytorch.py
Black 20 release
2020-08-26 17:20:22 +02:00
convert_mbart_original_checkpoint_to_pytorch.py
mBART Conversion script (#6230)
2020-08-04 09:53:51 -04:00
convert_mobilebert_original_tf_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_openai_original_tf_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_pegasus_tf_to_pytorch.py
Pegasus finetune script: add --adafactor (#6811)
2020-08-29 17:43:32 -04:00
convert_pytorch_checkpoint_to_tf2.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_reformer_trax_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_roberta_original_pytorch_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_t5_original_tf_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_tf_hub_seq_to_seq_bert_to_pytorch.py
Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models. (#6594)
2020-09-10 16:40:51 +02:00
convert_transfo_xl_original_tf_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_xlm_original_pytorch_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
convert_xlnet_original_tf_checkpoint_to_pytorch.py
Conversion scripts shouldn't have relative imports (#6991)
2020-09-07 08:31:06 -04:00
file_utils.py
Add TF Funnel Transformer (#7029)
2020-09-10 10:41:56 -04:00
generation_tf_utils.py
Black 20 release
2020-08-26 17:20:22 +02:00
generation_utils.py
Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models. (#6594)
2020-09-10 16:40:51 +02:00
hf_api.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
hf_argparser.py
Specify config filename (#6626)
2020-08-24 07:27:58 -04:00
integrations.py
Split hp search methods (#6857)
2020-08-31 15:16:39 -04:00
modelcard.py
Output attention takes an s (#6903)
2020-09-02 08:11:45 -04:00
modeling_albert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_auto.py
Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models. (#6594)
2020-09-10 16:40:51 +02:00
modeling_bart.py
replace torch.triu with onnx compatible code (#6929)
2020-09-09 04:56:40 -04:00
modeling_bert_generation.py
Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models. (#6594)
2020-09-10 16:40:51 +02:00
modeling_bert.py
Cannot index None (#6984)
2020-09-07 04:56:08 -04:00
modeling_camembert.py
Black 20 release
2020-08-26 17:20:22 +02:00
modeling_ctrl.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_distilbert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_dpr.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_electra.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_encoder_decoder.py
Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models. (#6594)
2020-09-10 16:40:51 +02:00
modeling_flaubert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_funnel.py
Add TF Funnel Transformer (#7029)
2020-09-10 10:41:56 -04:00
modeling_gpt2.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_longformer.py
[Longformer] Fix longformer documentation (#7016)
2020-09-08 18:51:28 +02:00
modeling_lxmert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_marian.py
[model_cards] Fix tiny typos
2020-08-26 23:16:06 +02:00
modeling_mbart.py
[Doc] add more MBart and other doc (#6490)
2020-08-17 12:30:26 -04:00
modeling_mmbt.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_mobilebert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_openai.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_outputs.py
[Generate] Facilitate PyTorch generate using ModelOutputs (#6735)
2020-09-01 12:38:25 +02:00
modeling_pegasus.py
Regression test for pegasus bugfix (#6606)
2020-08-20 15:34:43 -04:00
modeling_reformer.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_retribert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_roberta.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_t5.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_albert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_auto.py
Add TF Funnel Transformer (#7029)
2020-09-10 10:41:56 -04:00
modeling_tf_bert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_camembert.py
Black 20 release
2020-08-26 17:20:22 +02:00
modeling_tf_ctrl.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_distilbert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_electra.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_flaubert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_funnel.py
Add TF Funnel Transformer (#7029)
2020-09-10 10:41:56 -04:00
modeling_tf_gpt2.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_longformer.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_lxmert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_mobilebert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_openai.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_outputs.py
[Generate] Facilitate PyTorch generate using ModelOutputs (#6735)
2020-09-01 12:38:25 +02:00
modeling_tf_pytorch_utils.py
Fix confusing warnings during TF2 import from PyTorch (#6623)
2020-09-10 05:31:59 -04:00
modeling_tf_roberta.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_t5.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_transfo_xl_utilities.py
Black 20 release
2020-08-26 17:20:22 +02:00
modeling_tf_transfo_xl.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_utils.py
Adding the LXMERT pretraining model (MultiModal languageXvision) to HuggingFace's suite of models (#5793)
2020-09-03 04:02:25 -04:00
modeling_tf_xlm_roberta.py
Black 20 release
2020-08-26 17:20:22 +02:00
modeling_tf_xlm.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_tf_xlnet.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_transfo_xl_utilities.py
Black 20 release
2020-08-26 17:20:22 +02:00
modeling_transfo_xl.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_utils.py
Floating-point operations logging in trainer (#6768)
2020-09-08 10:00:56 -04:00
modeling_xlm_roberta.py
[EncoderDecoder] Add xlm-roberta to encoder decoder (#6878)
2020-09-01 21:56:39 +02:00
modeling_xlm.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
modeling_xlnet.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
optimization_tf.py
Black 20 release
2020-08-26 17:20:22 +02:00
optimization.py
Fix in Adafactor docstrings (#6845)
2020-08-31 10:52:47 -04:00
pipelines.py
feat: allow prefix for any generative model (#5885)
2020-09-07 03:03:45 -04:00
testing_utils.py
[from_pretrained] Allow tokenizer_type ≠ model_type (#6995)
2020-09-09 04:22:59 -04:00
tokenization_albert.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
tokenization_auto.py
Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models. (#6594)
2020-09-10 16:40:51 +02:00
tokenization_bart.py
prepare_seq2seq_batch makes labels/ decoder_input_ids made later. (#6654)
2020-08-28 11:15:17 -04:00
tokenization_bert_generation.py
Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models. (#6594)
2020-09-10 16:40:51 +02:00
tokenization_bert_japanese.py
fix error class instantiation (#6634)
2020-09-02 07:36:32 -04:00
tokenization_bert.py
Add missing arguments for BertWordPieceTokenizer (#5810)
2020-09-07 08:35:41 -04:00
tokenization_camembert.py
Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models. (#6594)
2020-09-10 16:40:51 +02:00
tokenization_ctrl.py
Black 20 release
2020-08-26 17:20:22 +02:00
tokenization_distilbert.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
tokenization_dpr.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
tokenization_electra.py
Cleanup fast tokenizers integration (#3706)
2020-04-18 13:43:57 +02:00
tokenization_flaubert.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
tokenization_funnel.py
Funnel transformer (#6908)
2020-09-08 08:08:08 -04:00
tokenization_gpt2.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
tokenization_longformer.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
tokenization_lxmert.py
Adding the LXMERT pretraining model (MultiModal languageXvision) to HuggingFace's suite of models (#5793)
2020-09-03 04:02:25 -04:00
tokenization_marian.py
prepare_seq2seq_batch makes labels/ decoder_input_ids made later. (#6654)
2020-08-28 11:15:17 -04:00
tokenization_mbart.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
tokenization_mobilebert.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
tokenization_openai.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
tokenization_pegasus.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
tokenization_reformer.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
tokenization_retribert.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
tokenization_roberta.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
tokenization_t5.py
Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models. (#6594)
2020-09-10 16:40:51 +02:00
tokenization_transfo_xl.py
Transformer-XL: Improved tokenization with sacremoses (#6322)
2020-08-28 09:56:17 -04:00
tokenization_utils_base.py
Batch encore plus and overflowing tokens fails when non existing overflowing tokens for a sequence (#6677)
2020-09-09 06:55:17 -04:00
tokenization_utils_fast.py
Black 20 release
2020-08-26 17:20:22 +02:00
tokenization_utils.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
tokenization_xlm_roberta.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
tokenization_xlm.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
tokenization_xlnet.py
[doc] remove the implied defaults to :obj:None, s/True/ :obj:`True/, etc. (#6956)
2020-09-04 18:22:25 -04:00
trainer_tf.py
Format
2020-08-27 14:56:47 +02:00
trainer_utils.py
Fixing FLOPS merge by checking if torch is available (#7013)
2020-09-08 10:51:58 -04:00
trainer.py
Fixing FLOPS merge by checking if torch is available (#7013)
2020-09-08 10:51:58 -04:00
training_args_tf.py
Trainer with grad accum (#6930)
2020-09-07 04:54:00 -04:00
training_args.py
Trainer with grad accum (#6930)
2020-09-07 04:54:00 -04:00