From 9ddf60b694346eef4f151835d80cb79b9decffb8 Mon Sep 17 00:00:00 2001 From: Lysandre Date: Tue, 21 Jan 2020 15:58:25 -0500 Subject: [PATCH] Tips + whitespaces --- docs/source/model_doc/camembert.rst | 35 +++++++++---- docs/source/model_doc/ctrl.rst | 41 ++++++++++----- docs/source/model_doc/distilbert.rst | 52 +++++++++++-------- docs/source/model_doc/gpt.rst | 19 ++++--- docs/source/model_doc/gpt2.rst | 16 +++--- docs/source/model_doc/roberta.rst | 20 ++++++-- docs/source/model_doc/transformerxl.rst | 22 ++++---- docs/source/model_doc/xlm.rst | 31 +++++++++--- docs/source/model_doc/xlmroberta.rst | 29 +++++++---- docs/source/model_doc/xlnet.rst | 56 +++++++++++++-------- src/transformers/configuration_camembert.py | 33 ++---------- src/transformers/file_utils.py | 4 +- src/transformers/modeling_albert.py | 26 ++++------ src/transformers/modeling_bert.py | 28 +++++------ src/transformers/modeling_camembert.py | 6 +-- src/transformers/modeling_ctrl.py | 23 +++++---- src/transformers/modeling_distilbert.py | 15 +++--- src/transformers/modeling_gpt2.py | 25 ++++----- src/transformers/modeling_openai.py | 28 +++++------ src/transformers/modeling_roberta.py | 21 ++++---- src/transformers/modeling_tf_albert.py | 22 ++++---- src/transformers/modeling_tf_bert.py | 22 ++++---- src/transformers/modeling_tf_ctrl.py | 20 ++++---- src/transformers/modeling_tf_distilbert.py | 20 ++++---- src/transformers/modeling_tf_gpt2.py | 22 ++++---- src/transformers/modeling_tf_openai.py | 22 ++++---- src/transformers/modeling_tf_roberta.py | 28 ++++++----- src/transformers/modeling_tf_transfo_xl.py | 14 +++--- src/transformers/modeling_tf_xlm.py | 30 +++++------ src/transformers/modeling_tf_xlnet.py | 24 ++++----- src/transformers/modeling_transfo_xl.py | 12 ++--- src/transformers/modeling_xlm.py | 24 +++++---- src/transformers/modeling_xlm_roberta.py | 8 +-- src/transformers/modeling_xlnet.py | 23 +++++---- 34 files changed, 452 insertions(+), 369 deletions(-) diff --git a/docs/source/model_doc/camembert.rst b/docs/source/model_doc/camembert.rst index 58958dd2fd..e60f7dd174 100644 --- a/docs/source/model_doc/camembert.rst +++ b/docs/source/model_doc/camembert.rst @@ -1,57 +1,70 @@ CamemBERT ---------------------------------------------------- -The CamemBERT model was proposed in `CamemBERT: a Tasty French Language Model`_ +The CamemBERT model was proposed in `CamemBERT: a Tasty French Language Model `__ by Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la -Clergerie, Djamé Seddah, and Benoît Sagot. It is based on Facebook's RoBERTa model released in 2019. +Clergerie, Djamé Seddah, and Benoît Sagot. It is based on Facebook's RoBERTa model released in 2019. It is a model +trained on 138GB of French text. -It is a model trained on 138GB of French text. +The abstract from the paper is the following: -This implementation is the same as RoBERTa. +*Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, +most available models have either been trained on English data or on the concatenation of data in multiple +languages. This makes practical use of such models --in all languages except English-- very limited. Aiming +to address this issue for French, we release CamemBERT, a French version of the Bi-directional Encoders for +Transformers (BERT). We measure the performance of CamemBERT compared to multilingual models in multiple +downstream tasks, namely part-of-speech tagging, dependency parsing, named-entity recognition, and natural +language inference. CamemBERT improves the state of the art for most of the tasks considered. We release the +pretrained model for CamemBERT hoping to foster research and downstream applications for French NLP.* -``CamembertConfig`` +Tips: + +- This implementation is the same as RoBERTa. Refer to the `documentation of RoBERTa <./roberta.html>`__ for usage + examples as well as the information relative to the inputs and outputs. + +CamembertConfig ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.CamembertConfig :members: -``CamembertTokenizer`` +CamembertTokenizer ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.CamembertTokenizer :members: -``CamembertModel`` +CamembertModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.CamembertModel :members: -``CamembertForMaskedLM`` +CamembertForMaskedLM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.CamembertForMaskedLM :members: -``CamembertForSequenceClassification`` +CamembertForSequenceClassification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.CamembertForSequenceClassification :members: -``CamembertForMultipleChoice`` +CamembertForMultipleChoice ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.CamembertForMultipleChoice :members: -``CamembertForTokenClassification`` +CamembertForTokenClassification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.CamembertForTokenClassification diff --git a/docs/source/model_doc/ctrl.rst b/docs/source/model_doc/ctrl.rst index e83007d8dd..a8a04837d7 100644 --- a/docs/source/model_doc/ctrl.rst +++ b/docs/source/model_doc/ctrl.rst @@ -6,51 +6,68 @@ by Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and Ric It's a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus of ~140 GB of text data with the first token reserved as a control code (such as Links, Books, Wikipedia etc.). -This model is a PyTorch `torch.nn.Module `_ sub-class. -Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general -usage and behavior. +The abstract from the paper is the following: -Note: if you fine-tune a CTRL model using the Salesforce code (https://github.com/salesforce/ctrl), -you'll be able to convert from TF to our HuggingFace/Transformers format using the -``convert_tf_to_huggingface_pytorch.py`` script (see `issue #1654 `_). +*Large-scale language models show promising text generation capabilities, but users cannot easily control particular +aspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model, +trained to condition on control codes that govern style, content, and task-specific behavior. Control codes were +derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning +while providing more explicit control over text generation. These codes also allow CTRL to predict which parts of +the training data are most likely given a sequence. This provides a potential method for analyzing large amounts +of data via model-based source attribution.* + +Tips: + +- CTRL makes use of control codes to generate text: it requires generations to be started by certain words, sentences + or links to generate coherent text. Refer to the `original implementation `__ + for more information. +- CTRL is a model with absolute position embeddings so it's usually advised to pad the inputs on + the right rather than the left. +- CTRL was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next + token in a sequence. Leveraging this feature allows CTRL to generate syntactically coherent text as + it can be observed in the `run_generation.py` example script. +- The PyTorch models can take the `past` as input, which is the previously computed key/value attention pairs. Using + this `past` value prevents the model from re-computing pre-computed values in the context of text generation. + See `reusing the past in generative models <../quickstart.html#using-the-past>`_ for more information on the usage + of this argument. -``CTRLConfig`` +CTRLConfig ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.CTRLConfig :members: -``CTRLTokenizer`` +CTRLTokenizer ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.CTRLTokenizer :members: -``CTRLModel`` +CTRLModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.CTRLModel :members: -``CTRLLMHeadModel`` +CTRLLMHeadModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.CTRLLMHeadModel :members: -``TFCTRLModel`` +TFCTRLModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFCTRLModel :members: -``TFCTRLLMHeadModel`` +TFCTRLLMHeadModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFCTRLLMHeadModel diff --git a/docs/source/model_doc/distilbert.rst b/docs/source/model_doc/distilbert.rst index 9ffbf0c2f8..81d8086c15 100644 --- a/docs/source/model_doc/distilbert.rst +++ b/docs/source/model_doc/distilbert.rst @@ -1,86 +1,96 @@ DistilBERT ---------------------------------------------------- -DistilBERT is a small, fast, cheap and light Transformer model -trained by distilling Bert base. It has 40% less parameters than -`bert-base-uncased`, runs 60% faster while preserving over 95% of -Bert's performances as measured on the GLUE language understanding benchmark. +The DistilBERT model was proposed in the blog post +`Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT `__, +and the paper `DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter `__. +DistilBERT is a small, fast, cheap and light Transformer model trained by distilling Bert base. It has 40% less +parameters than `bert-base-uncased`, runs 60% faster while preserving over 95% of Bert's performances as measured on +the GLUE language understanding benchmark. -Here are the differences between the interface of Bert and DistilBert: +The abstract from the paper is the following: + +*As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), +operating these large models in on-the-edge and/or under constrained computational training or inference budgets +remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation +model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger +counterparts. While most prior work investigated the use of distillation for building task-specific models, we +leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a +BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage +the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language +modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train +and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative +on-device study.* + +Tips: - DistilBert doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just separate your segments with the separation token `tokenizer.sep_token` (or `[SEP]`) - DistilBert doesn't have options to select the input positions (`position_ids` input). This could be added if necessary though, just let's us know if you need this option. -For more information on DistilBERT, please refer to our -`detailed blog post`_ -.. _`detailed blog post`: - https://medium.com/huggingface/distilbert-8cf3380435b5 - - -``DistilBertConfig`` +DistilBertConfig ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.DistilBertConfig :members: -``DistilBertTokenizer`` +DistilBertTokenizer ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.DistilBertTokenizer :members: -``DistilBertModel`` +DistilBertModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.DistilBertModel :members: -``DistilBertForMaskedLM`` +DistilBertForMaskedLM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.DistilBertForMaskedLM :members: -``DistilBertForSequenceClassification`` +DistilBertForSequenceClassification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.DistilBertForSequenceClassification :members: -``DistilBertForQuestionAnswering`` +DistilBertForQuestionAnswering ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.DistilBertForQuestionAnswering :members: -``TFDistilBertModel`` +TFDistilBertModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFDistilBertModel :members: -``TFDistilBertForMaskedLM`` +TFDistilBertForMaskedLM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFDistilBertForMaskedLM :members: -``TFDistilBertForSequenceClassification`` +TFDistilBertForSequenceClassification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFDistilBertForSequenceClassification :members: -``TFDistilBertForQuestionAnswering`` +TFDistilBertForQuestionAnswering ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFDistilBertForQuestionAnswering diff --git a/docs/source/model_doc/gpt.rst b/docs/source/model_doc/gpt.rst index 605d083e7f..8c27fcb776 100644 --- a/docs/source/model_doc/gpt.rst +++ b/docs/source/model_doc/gpt.rst @@ -1,6 +1,9 @@ OpenAI GPT ---------------------------------------------------- +Overview +~~~~~~~~~~~~~~~~~~~~~ + OpenAI GPT model was proposed in `Improving Language Understanding by Generative Pre-Training`_ by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. It's a causal (unidirectional) transformer pre-trained using language modeling on a large corpus will long range dependencies, the Toronto Book Corpus. @@ -33,56 +36,56 @@ Tips: `Write With Transformer `__ is a webapp created and hosted by Hugging Face showcasing the generative capabilities of several models. GPT is one of them. -``OpenAIGPTConfig`` +OpenAIGPTConfig ~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.OpenAIGPTConfig :members: -``OpenAIGPTTokenizer`` +OpenAIGPTTokenizer ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.OpenAIGPTTokenizer :members: -``OpenAIGPTModel`` +OpenAIGPTModel ~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.OpenAIGPTModel :members: -``OpenAIGPTLMHeadModel`` +OpenAIGPTLMHeadModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.OpenAIGPTLMHeadModel :members: -``OpenAIGPTDoubleHeadsModel`` +OpenAIGPTDoubleHeadsModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.OpenAIGPTDoubleHeadsModel :members: -``TFOpenAIGPTModel`` +TFOpenAIGPTModel ~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFOpenAIGPTModel :members: -``TFOpenAIGPTLMHeadModel`` +TFOpenAIGPTLMHeadModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFOpenAIGPTLMHeadModel :members: -``TFOpenAIGPTDoubleHeadsModel`` +TFOpenAIGPTDoubleHeadsModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFOpenAIGPTDoubleHeadsModel diff --git a/docs/source/model_doc/gpt2.rst b/docs/source/model_doc/gpt2.rst index 48c1209af9..54ef3cea08 100644 --- a/docs/source/model_doc/gpt2.rst +++ b/docs/source/model_doc/gpt2.rst @@ -35,56 +35,56 @@ Hugging Face showcasing the generative capabilities of several models. GPT-2 is different sizes: small, medium, large, xl and a distilled version of the small checkpoint: distilgpt-2. -``GPT2Config`` +GPT2Config ~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.GPT2Config :members: -``GPT2Tokenizer`` +GPT2Tokenizer ~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.GPT2Tokenizer :members: -``GPT2Model`` +GPT2Model ~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.GPT2Model :members: -``GPT2LMHeadModel`` +GPT2LMHeadModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.GPT2LMHeadModel :members: -``GPT2DoubleHeadsModel`` +GPT2DoubleHeadsModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.GPT2DoubleHeadsModel :members: -``TFGPT2Model`` +TFGPT2Model ~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFGPT2Model :members: -``TFGPT2LMHeadModel`` +TFGPT2LMHeadModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFGPT2LMHeadModel :members: -``TFGPT2DoubleHeadsModel`` +TFGPT2DoubleHeadsModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFGPT2DoubleHeadsModel diff --git a/docs/source/model_doc/roberta.rst b/docs/source/model_doc/roberta.rst index 4112a44395..d3276d55e0 100644 --- a/docs/source/model_doc/roberta.rst +++ b/docs/source/model_doc/roberta.rst @@ -1,15 +1,29 @@ RoBERTa ---------------------------------------------------- -The RoBERTa model was proposed in `RoBERTa: A Robustly Optimized BERT Pretraining Approach`_ +The RoBERTa model was proposed in `RoBERTa: A Robustly Optimized BERT Pretraining Approach `_ by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. It is based on Google's BERT model released in 2018. It builds on BERT and modifies key hyperparameters, removing the next-sentence pretraining objective and training with much larger mini-batches and learning rates. -This implementation is the same as BertModel with a tiny embeddings tweak as well as a setup for Roberta pretrained -models. +The abstract from the paper is the following: + +*Language model pretraining has led to significant performance gains but careful comparison between different +approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, +and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication +study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and +training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of +every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These +results highlight the importance of previously overlooked design choices, and raise questions about the source +of recently reported improvements. We release our models and code.* + +Tips: + +- This implementation is the same as :class:`~transformers.BertModel` with a tiny embeddings tweak as well as a + setup for Roberta pretrained models. +- `Camembert <./camembert.html>`__ is a wrapper around RoBERTa. Refer to this page for usage examples. RobertaConfig ~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/source/model_doc/transformerxl.rst b/docs/source/model_doc/transformerxl.rst index 08f51bd350..5240df3df4 100644 --- a/docs/source/model_doc/transformerxl.rst +++ b/docs/source/model_doc/transformerxl.rst @@ -1,8 +1,11 @@ Transformer XL ---------------------------------------------------- +Overview +~~~~~~~~~~~~~~~~~~~~~ + The Transformer-XL model was proposed in -`Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context`_ +`Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context `__ by Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov. It's a causal (uni-directional) transformer with relative positioning (sinusoïdal) embeddings which can reuse previously computed hidden-states to attend to longer context (memory). @@ -23,46 +26,47 @@ coherent, novel text articles with thousands of tokens.* Tips: -- Transformer-XL uses relative sinusoidal positional embeddings so it's usually advised to pad the inputs on - the left rather than the right. +- Transformer-XL uses relative sinusoidal positional embeddings. Padding can be done on the left or on the right. + The original implementation trains on SQuAD with padding on the left, therefore the padding defaults are set to left. +- Transformer-XL is one of the few models that has no sequence length limit. -``TransfoXLConfig`` +TransfoXLConfig ~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TransfoXLConfig :members: -``TransfoXLTokenizer`` +TransfoXLTokenizer ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TransfoXLTokenizer :members: -``TransfoXLModel`` +TransfoXLModel ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TransfoXLModel :members: -``TransfoXLLMHeadModel`` +TransfoXLLMHeadModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TransfoXLLMHeadModel :members: -``TFTransfoXLModel`` +TFTransfoXLModel ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFTransfoXLModel :members: -``TFTransfoXLLMHeadModel`` +TFTransfoXLLMHeadModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFTransfoXLLMHeadModel diff --git a/docs/source/model_doc/xlm.rst b/docs/source/model_doc/xlm.rst index 86e6d199ce..7346693752 100644 --- a/docs/source/model_doc/xlm.rst +++ b/docs/source/model_doc/xlm.rst @@ -1,17 +1,34 @@ XLM ---------------------------------------------------- -The XLM model was proposed in `Cross-lingual Language Model Pretraining`_ +Overview +~~~~~~~~~~~~~~~~~~~~~ + +The XLM model was proposed in `Cross-lingual Language Model Pretraining `_ by Guillaume Lample*, Alexis Conneau*. It's a transformer pre-trained using one of the following objectives: - - a causal language modeling (CLM) objective (next token prediction), - - a masked language modeling (MLM) objective (Bert-like), or - - a Translation Language Modeling (TLM) object (extension of Bert's MLM to multiple language inputs) +- a causal language modeling (CLM) objective (next token prediction), +- a masked language modeling (MLM) objective (Bert-like), or +- a Translation Language Modeling (TLM) object (extension of Bert's MLM to multiple language inputs) -Original code can be found `here `_. +The abstract from the paper is the following: -This model is a PyTorch `torch.nn.Module`_ sub-class. Use it as a regular PyTorch Module and -refer to the PyTorch documentation for all matter related to general usage and behavior. +*Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. +In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. +We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual +data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain +state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, +our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, +we obtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On +supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming +the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.* + +Tips: + +- XLM has many different checkpoints, which were trained using different objectives: CLM, MLM or TLM. Make sure to + select the correct objective for your task (e.g. MLM checkpoints are not suitable for generation). +- XLM has multilingual checkpoints which leverage a specific `lang` parameter. Check out the + `multi-lingual <../multilingual.html>`__ page for more information. XLMConfig diff --git a/docs/source/model_doc/xlmroberta.rst b/docs/source/model_doc/xlmroberta.rst index 8944c53895..66ea48858a 100644 --- a/docs/source/model_doc/xlmroberta.rst +++ b/docs/source/model_doc/xlmroberta.rst @@ -1,22 +1,29 @@ XLM-RoBERTa ------------------------------------------ -The XLM-RoBERTa model was proposed in `Unsupervised Cross-lingual Representation Learning at Scale`_ -by Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov. It is based on Facebook's RoBERTa model released in 2019. - +The XLM-RoBERTa model was proposed in `Unsupervised Cross-lingual Representation Learning at Scale `__ +by Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, +Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov. It is based on Facebook's RoBERTa model released in 2019. It is a large multi-lingual language model, trained on 2.5TB of filtered CommonCrawl data. -This implementation is the same as RoBERTa. +The abstract from the paper is the following: -This model is a PyTorch `torch.nn.Module`_ sub-class. Use it as a regular PyTorch Module and -refer to the PyTorch documentation for all matter related to general usage and behavior. +*This paper shows that pretraining multilingual language models at scale leads to significant performance gains for +a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred +languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly +outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +13.8% average accuracy +on XNLI, +12.3% average F1 score on MLQA, and +2.1% average F1 score on NER. XLM-R performs particularly well on +low-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model. +We also present a detailed empirical evaluation of the key factors that are required to achieve these gains, +including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and +low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling +without sacrificing per-language performance; XLM-Ris very competitive with strong monolingual models on the GLUE +and XNLI benchmarks. We will make XLM-R code, data, and models publicly available.* -.. _`Unsupervised Cross-lingual Representation Learning at Scale`: - https://arxiv.org/abs/1911.02116 - -.. _`torch.nn.Module`: - https://pytorch.org/docs/stable/nn.html#module +Tips: +- This implementation is the same as RoBERTa. Refer to the `documentation of RoBERTa <./roberta.html>`__ for usage + examples as well as the information relative to the inputs and outputs. XLMRobertaConfig ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/source/model_doc/xlnet.rst b/docs/source/model_doc/xlnet.rst index 66b56afbc8..0f8c61098c 100644 --- a/docs/source/model_doc/xlnet.rst +++ b/docs/source/model_doc/xlnet.rst @@ -1,107 +1,123 @@ XLNet ---------------------------------------------------- -The XLNet model was proposed in `XLNet: Generalized Autoregressive Pretraining for Language Understanding`_ +Overview +~~~~~~~~~~~~~~~~~~~~~ + +The XLNet model was proposed in `XLNet: Generalized Autoregressive Pretraining for Language Understanding `_ by Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le. XLnet is an extension of the Transformer-XL model pre-trained using an autoregressive method to learn bidirectional contexts by maximizing the expected likelihood over all permutations of the input sequence factorization order. -The specific attention pattern can be controlled at training and test time using the `perm_mask` input. +The abstract from the paper is the following: -Due to the difficulty of training a fully auto-regressive model over various factorization order, -XLNet is pretrained using only a sub-set of the output tokens as target which are selected -with the `target_mapping` input. +*With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves +better performance than pretraining approaches based on autoregressive language modeling. However, relying on +corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a +pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive +pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over +all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive +formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, +into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by +a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.* -To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and -`target_mapping` inputs to control the attention span and outputs (see examples in `examples/run_generation.py`) +Tips: + +- The specific attention pattern can be controlled at training and test time using the `perm_mask` input. +- Due to the difficulty of training a fully auto-regressive model over various factorization order, + XLNet is pretrained using only a sub-set of the output tokens as target which are selected + with the `target_mapping` input. +- To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and + `target_mapping` inputs to control the attention span and outputs (see examples in `examples/run_generation.py`) +- XLNet is one of the few models that has no sequence length limit. -``XLNetConfig`` +XLNetConfig ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.XLNetConfig :members: -``XLNetTokenizer`` +XLNetTokenizer ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.XLNetTokenizer :members: -``XLNetModel`` +XLNetModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.XLNetModel :members: -``XLNetLMHeadModel`` +XLNetLMHeadModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.XLNetLMHeadModel :members: -``XLNetForSequenceClassification`` +XLNetForSequenceClassification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.XLNetForSequenceClassification :members: -``XLNetForTokenClassification`` +XLNetForTokenClassification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.XLNetForTokenClassification :members: -``XLNetForMultipleChoice`` +XLNetForMultipleChoice ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.XLNetForMultipleChoice :members: -``XLNetForQuestionAnsweringSimple`` +XLNetForQuestionAnsweringSimple ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.XLNetForQuestionAnsweringSimple :members: -``XLNetForQuestionAnswering`` +XLNetForQuestionAnswering ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.XLNetForQuestionAnswering :members: -``TFXLNetModel`` +TFXLNetModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFXLNetModel :members: -``TFXLNetLMHeadModel`` +TFXLNetLMHeadModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFXLNetLMHeadModel :members: -``TFXLNetForSequenceClassification`` +TFXLNetForSequenceClassification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFXLNetForSequenceClassification :members: -``TFXLNetForQuestionAnsweringSimple`` +TFXLNetForQuestionAnsweringSimple ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFXLNetForQuestionAnsweringSimple diff --git a/src/transformers/configuration_camembert.py b/src/transformers/configuration_camembert.py index ff55908584..1c284b7a5d 100644 --- a/src/transformers/configuration_camembert.py +++ b/src/transformers/configuration_camembert.py @@ -29,35 +29,10 @@ CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = { class CamembertConfig(RobertaConfig): - r""" - This is the configuration class to store the configuration of an :class:`~transformers.CamembertModel`. - It is used to instantiate an Camembert model according to the specified arguments, defining the model - architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of - the BERT `bert-base-uncased `__ architecture. - - Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used - to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig` - for more information. - - The :class:`~transformers.CamembertConfig` class directly inherits :class:`~transformers.BertConfig`. - It reuses the same defaults. Please check the parent class for more information. - - Example:: - - from transformers import CamembertModel, CamembertConfig - - # Initializing a CamemBERT configuration - configuration = CamembertConfig() - - # Initializing a model from the configuration - model = CamembertModel(configuration) - - # Accessing the model configuration - configuration = model.config - - Attributes: - pretrained_config_archive_map (Dict[str, str]): - A dictionary containing all the available pre-trained checkpoints. """ + This class overrides :class:`~transformers.RobertaConfig`. Please check the + superclass for the appropriate documentation alongside usage examples. + """ + pretrained_config_archive_map = CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP model_type = "camembert" diff --git a/src/transformers/file_utils.py b/src/transformers/file_utils.py index 97b95b5072..23fd3171d2 100644 --- a/src/transformers/file_utils.py +++ b/src/transformers/file_utils.py @@ -115,8 +115,8 @@ def add_start_docstrings_to_callable(*docstr): def docstring_decorator(fn): class_name = ":class:`~transformers.{}`".format(fn.__qualname__.split(".")[0]) intro = " The {} forward method, overrides the :func:`__call__` special method.".format(class_name) - note = r""" - + note = r""" + .. note:: Although the recipe for forward pass needs to be defined within this function, one should call the :class:`Module` instance afterwards diff --git a/src/transformers/modeling_albert.py b/src/transformers/modeling_albert.py index 1a79b83590..04c06e06db 100644 --- a/src/transformers/modeling_albert.py +++ b/src/transformers/modeling_albert.py @@ -376,15 +376,11 @@ class AlbertPreTrainedModel(PreTrainedModel): module.weight.data.fill_(1.0) -ALBERT_START_DOCSTRING = r""" - This model is a PyTorch `torch.nn.Module`_ sub-class. Use it as a regular PyTorch Module and - refer to the PyTorch documentation for all matter related to general usage and behavior. +ALBERT_START_DOCSTRING = r""" - .. _`ALBERT: A Lite BERT for Self-supervised Learning of Language Representations`: - https://arxiv.org/abs/1909.11942 - - .. _`torch.nn.Module`: - https://pytorch.org/docs/stable/nn.html#module + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general + usage and behavior. Args: config (:class:`~transformers.AlbertConfig`): Model configuration class with all the parameters of the model. @@ -395,29 +391,29 @@ ALBERT_START_DOCSTRING = r""" ALBERT_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.AlbertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. diff --git a/src/transformers/modeling_bert.py b/src/transformers/modeling_bert.py index 630ab3b62f..a66551fbee 100644 --- a/src/transformers/modeling_bert.py +++ b/src/transformers/modeling_bert.py @@ -545,9 +545,9 @@ class BertPreTrainedModel(PreTrainedModel): module.bias.data.zero_() -BERT_START_DOCSTRING = r""" - This model is a PyTorch `torch.nn.Module `_ sub-class. - Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general +BERT_START_DOCSTRING = r""" + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: @@ -559,29 +559,29 @@ BERT_START_DOCSTRING = r""" BERT_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. @@ -592,7 +592,7 @@ BERT_INPUTS_DOCSTRING = r""" This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. encoder_hidden_states (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): - Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention + Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on the padding token indices of the encoder input. This mask @@ -809,7 +809,7 @@ class BertModel(BertPreTrainedModel): @add_start_docstrings( - """Bert Model with two heads on top as done during the pre-training: a `masked language modeling` head and + """Bert Model with two heads on top as done during the pre-training: a `masked language modeling` head and a `next sentence prediction (classification)` head. """, BERT_START_DOCSTRING, ) @@ -1095,7 +1095,7 @@ class BertForNextSentencePrediction(BertPreTrainedModel): @add_start_docstrings( - """Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of + """Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. """, BERT_START_DOCSTRING, ) @@ -1373,7 +1373,7 @@ class BertForTokenClassification(BertPreTrainedModel): @add_start_docstrings( - """Bert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear + """Bert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). """, BERT_START_DOCSTRING, ) diff --git a/src/transformers/modeling_camembert.py b/src/transformers/modeling_camembert.py index 5aff198fae..a5eb1deae3 100644 --- a/src/transformers/modeling_camembert.py +++ b/src/transformers/modeling_camembert.py @@ -36,10 +36,10 @@ CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP = { } -CAMEMBERT_START_DOCSTRING = r""" +CAMEMBERT_START_DOCSTRING = r""" - This model is a PyTorch `torch.nn.Module `_ sub-class. - Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: diff --git a/src/transformers/modeling_ctrl.py b/src/transformers/modeling_ctrl.py index 09ec688f4d..2a5301e099 100644 --- a/src/transformers/modeling_ctrl.py +++ b/src/transformers/modeling_ctrl.py @@ -184,9 +184,10 @@ class CTRLPreTrainedModel(PreTrainedModel): module.weight.data.fill_(1.0) -CTRL_START_DOCSTRING = r""" - This model is a PyTorch `torch.nn.Module`_ sub-class. Use it as a regular PyTorch Module and - refer to the PyTorch documentation for all matter related to general usage and behavior. +CTRL_START_DOCSTRING = r""" + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general + usage and behavior. Parameters: config (:class:`~transformers.CTRLConfig`): Model configuration class with all the parameters of the model. @@ -194,15 +195,15 @@ CTRL_START_DOCSTRING = r""" Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. """ -CTRL_INPUTS_DOCSTRING = r""" +CTRL_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.CTRLTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ past (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model @@ -212,18 +213,18 @@ CTRL_INPUTS_DOCSTRING = r""" Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. diff --git a/src/transformers/modeling_distilbert.py b/src/transformers/modeling_distilbert.py index 5123b52ec2..7a66292d5e 100644 --- a/src/transformers/modeling_distilbert.py +++ b/src/transformers/modeling_distilbert.py @@ -351,10 +351,11 @@ class DistilBertPreTrainedModel(PreTrainedModel): DISTILBERT_START_DOCSTRING = r""" - This model is a PyTorch `torch.nn.Module `_ sub-class. - Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general + + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. - + Parameters: config (:class:`~transformers.DistilBertConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. @@ -364,18 +365,18 @@ DISTILBERT_START_DOCSTRING = r""" DISTILBERT_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.DistilBertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. diff --git a/src/transformers/modeling_gpt2.py b/src/transformers/modeling_gpt2.py index 8ce82aa477..a9c7910eb6 100644 --- a/src/transformers/modeling_gpt2.py +++ b/src/transformers/modeling_gpt2.py @@ -265,9 +265,10 @@ class GPT2PreTrainedModel(PreTrainedModel): module.weight.data.fill_(1.0) -GPT2_START_DOCSTRING = r""" - This model is a PyTorch `torch.nn.Module `_ sub-class. - Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general +GPT2_START_DOCSTRING = r""" + + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: @@ -276,15 +277,15 @@ GPT2_START_DOCSTRING = r""" Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. """ -GPT2_INPUTS_DOCSTRING = r""" +GPT2_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.GPT2Tokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ past (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model @@ -294,18 +295,18 @@ GPT2_INPUTS_DOCSTRING = r""" Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. @@ -503,7 +504,7 @@ class GPT2Model(GPT2PreTrainedModel): @add_start_docstrings( - """The GPT2 Model transformer with a language modeling head on top + """The GPT2 Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). """, GPT2_START_DOCSTRING, ) diff --git a/src/transformers/modeling_openai.py b/src/transformers/modeling_openai.py index d70db76f6c..1f098b3cc9 100644 --- a/src/transformers/modeling_openai.py +++ b/src/transformers/modeling_openai.py @@ -279,15 +279,11 @@ class OpenAIGPTPreTrainedModel(PreTrainedModel): module.weight.data.fill_(1.0) -OPENAI_GPT_START_DOCSTRING = r""" - This model is a PyTorch `torch.nn.Module`_ sub-class. Use it as a regular PyTorch Module and - refer to the PyTorch documentation for all matter related to general usage and behavior. +OPENAI_GPT_START_DOCSTRING = r""" - .. _`Improving Language Understanding by Generative Pre-Training`: - https://openai.com/blog/language-unsupervised/ - - .. _`torch.nn.Module`: - https://pytorch.org/docs/stable/nn.html#module + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general + usage and behavior. Parameters: config (:class:`~transformers.OpenAIGPTConfig`): Model configuration class with all the parameters of the model. @@ -295,32 +291,32 @@ OPENAI_GPT_START_DOCSTRING = r""" Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. """ -OPENAI_GPT_INPUTS_DOCSTRING = r""" +OPENAI_GPT_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.OpenAIGPTTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. diff --git a/src/transformers/modeling_roberta.py b/src/transformers/modeling_roberta.py index 8855ab3b43..bd8068be5b 100644 --- a/src/transformers/modeling_roberta.py +++ b/src/transformers/modeling_roberta.py @@ -92,10 +92,11 @@ class RobertaEmbeddings(BertEmbeddings): return position_ids.unsqueeze(0).expand(input_shape) -ROBERTA_START_DOCSTRING = r""" +ROBERTA_START_DOCSTRING = r""" - This model is a PyTorch `torch.nn.Module `_ sub-class. Use it as a regular PyTorch Module and - refer to the PyTorch documentation for all matter related to general usage and behavior. + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general + usage and behavior. Parameters: config (:class:`~transformers.RobertaConfig`): Model configuration class with all the parameters of the @@ -106,29 +107,29 @@ ROBERTA_START_DOCSTRING = r""" ROBERTA_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.RobertaTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. diff --git a/src/transformers/modeling_tf_albert.py b/src/transformers/modeling_tf_albert.py index 9211d6bbed..2a1d3f1c4d 100644 --- a/src/transformers/modeling_tf_albert.py +++ b/src/transformers/modeling_tf_albert.py @@ -478,8 +478,8 @@ class TFAlbertMLMHead(tf.keras.layers.Layer): return hidden_states -ALBERT_START_DOCSTRING = r""" - This model is a `tf.keras.Model `__ sub-class. +ALBERT_START_DOCSTRING = r""" + This model is a `tf.keras.Model `__ sub-class. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. @@ -490,16 +490,16 @@ ALBERT_START_DOCSTRING = r""" https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/Model .. note:: - + TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. - This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having + This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. - If you choose this second option, there are three possibilities you can use to gather all the input Tensors + If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` @@ -517,29 +517,29 @@ ALBERT_START_DOCSTRING = r""" ALBERT_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.AlbertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ token_type_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. diff --git a/src/transformers/modeling_tf_bert.py b/src/transformers/modeling_tf_bert.py index e973cd3277..e9f492468b 100644 --- a/src/transformers/modeling_tf_bert.py +++ b/src/transformers/modeling_tf_bert.py @@ -585,21 +585,21 @@ class TFBertPreTrainedModel(TFPreTrainedModel): BERT_START_DOCSTRING = r""" - This model is a `tf.keras.Model `__ sub-class. + This model is a `tf.keras.Model `__ sub-class. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. .. note:: - + TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. - This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having + This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. - If you choose this second option, there are three possibilities you can use to gather all the input Tensors + If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` @@ -617,20 +617,20 @@ BERT_START_DOCSTRING = r""" BERT_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - token_type_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token @@ -639,13 +639,13 @@ BERT_INPUTS_DOCSTRING = r""" position_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`__ head_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. - inputs_embeds (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, embedding_dim)`, `optional`, defaults to :obj:`None`): + inputs_embeds (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, embedding_dim)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. diff --git a/src/transformers/modeling_tf_ctrl.py b/src/transformers/modeling_tf_ctrl.py index 8180a3352e..1756da703e 100644 --- a/src/transformers/modeling_tf_ctrl.py +++ b/src/transformers/modeling_tf_ctrl.py @@ -360,10 +360,10 @@ CTRL_START_DOCSTRING = r""" - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. - This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having + This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. - If you choose this second option, there are three possibilities you can use to gather all the input Tensors + If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` @@ -371,7 +371,7 @@ CTRL_START_DOCSTRING = r""" :obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: :obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})` - + Parameters: config (:class:`~transformers.CTRLConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. @@ -381,12 +381,12 @@ CTRL_START_DOCSTRING = r""" CTRL_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.CTRLTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ past (:obj:`List[tf.Tensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model @@ -396,18 +396,18 @@ CTRL_INPUTS_DOCSTRING = r""" Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. diff --git a/src/transformers/modeling_tf_distilbert.py b/src/transformers/modeling_tf_distilbert.py index 1f7db7ced1..2f5f253cd6 100644 --- a/src/transformers/modeling_tf_distilbert.py +++ b/src/transformers/modeling_tf_distilbert.py @@ -471,21 +471,21 @@ class TFDistilBertPreTrainedModel(TFPreTrainedModel): DISTILBERT_START_DOCSTRING = r""" - This model is a `tf.keras.Model `__ sub-class. + This model is a `tf.keras.Model `__ sub-class. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. .. note:: - + TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. - This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having + This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. - If you choose this second option, there are three possibilities you can use to gather all the input Tensors + If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` @@ -503,31 +503,31 @@ DISTILBERT_START_DOCSTRING = r""" DISTILBERT_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. - inputs_embeds (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, embedding_dim)`, `optional`, defaults to :obj:`None`): + inputs_embeds (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, embedding_dim)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. training (:obj:`boolean`, `optional`, defaults to :obj:`False`): Whether to activate dropout modules (if set to :obj:`True`) during training or to de-activate them (if set to :obj:`False`) for evaluation. - + """ diff --git a/src/transformers/modeling_tf_gpt2.py b/src/transformers/modeling_tf_gpt2.py index 8ecbcfa339..5ff7491e8d 100644 --- a/src/transformers/modeling_tf_gpt2.py +++ b/src/transformers/modeling_tf_gpt2.py @@ -368,7 +368,7 @@ class TFGPT2PreTrainedModel(TFPreTrainedModel): base_model_prefix = "transformer" -GPT2_START_DOCSTRING = r""" +GPT2_START_DOCSTRING = r""" .. note:: TF 2.0 models accepts two formats as inputs: @@ -376,10 +376,10 @@ GPT2_START_DOCSTRING = r""" - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. - This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having + This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. - If you choose this second option, there are three possibilities you can use to gather all the input Tensors + If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` @@ -394,15 +394,15 @@ GPT2_START_DOCSTRING = r""" Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. """ -GPT2_INPUTS_DOCSTRING = r""" +GPT2_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.GPT2Tokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ past (:obj:`List[tf.Tensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model @@ -412,18 +412,18 @@ GPT2_INPUTS_DOCSTRING = r""" Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. diff --git a/src/transformers/modeling_tf_openai.py b/src/transformers/modeling_tf_openai.py index 69263639ec..8ed1654b6a 100644 --- a/src/transformers/modeling_tf_openai.py +++ b/src/transformers/modeling_tf_openai.py @@ -362,10 +362,10 @@ OPENAI_GPT_START_DOCSTRING = r""" - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. - This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having + This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. - If you choose this second option, there are three possibilities you can use to gather all the input Tensors + If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` @@ -373,7 +373,7 @@ OPENAI_GPT_START_DOCSTRING = r""" :obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: :obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})` - + Parameters: config (:class:`~transformers.OpenAIGPTConfig`): Model configuration class with all the parameters of the model. @@ -381,32 +381,32 @@ OPENAI_GPT_START_DOCSTRING = r""" Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. """ -OPENAI_GPT_INPUTS_DOCSTRING = r""" +OPENAI_GPT_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.GPT2Tokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. diff --git a/src/transformers/modeling_tf_roberta.py b/src/transformers/modeling_tf_roberta.py index 572d8e8b5f..d0cce05cee 100644 --- a/src/transformers/modeling_tf_roberta.py +++ b/src/transformers/modeling_tf_roberta.py @@ -21,7 +21,7 @@ import logging import tensorflow as tf from .configuration_roberta import RobertaConfig -from .file_utils import add_start_docstrings +from .file_utils import add_start_docstrings, add_start_docstrings_to_callable from .modeling_tf_bert import TFBertEmbeddings, TFBertMainLayer, gelu from .modeling_tf_utils import TFPreTrainedModel, get_initializer, shape_list @@ -105,21 +105,21 @@ class TFRobertaPreTrainedModel(TFPreTrainedModel): ROBERTA_START_DOCSTRING = r""" - This model is a `tf.keras.Model `__ sub-class. + This model is a `tf.keras.Model `__ sub-class. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. .. note:: - + TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. - This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having + This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. - If you choose this second option, there are three possibilities you can use to gather all the input Tensors + If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` @@ -137,20 +137,20 @@ ROBERTA_START_DOCSTRING = r""" ROBERTA_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.RobertaTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - token_type_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token @@ -159,13 +159,13 @@ ROBERTA_INPUTS_DOCSTRING = r""" position_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`__ head_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. - inputs_embeds (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, embedding_dim)`, `optional`, defaults to :obj:`None`): + inputs_embeds (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, embedding_dim)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. @@ -184,6 +184,7 @@ class TFRobertaModel(TFRobertaPreTrainedModel): super().__init__(config, *inputs, **kwargs) self.roberta = TFRobertaMainLayer(config, name="roberta") + @add_start_docstrings_to_callable(ROBERTA_INPUTS_DOCSTRING) def call(self, inputs, **kwargs): r""" Returns: @@ -266,6 +267,7 @@ class TFRobertaForMaskedLM(TFRobertaPreTrainedModel): def get_output_embeddings(self): return self.lm_head.decoder + @add_start_docstrings_to_callable(ROBERTA_INPUTS_DOCSTRING) def call(self, inputs, **kwargs): r""" Return: @@ -343,6 +345,7 @@ class TFRobertaForSequenceClassification(TFRobertaPreTrainedModel): self.roberta = TFRobertaMainLayer(config, name="roberta") self.classifier = TFRobertaClassificationHead(config, name="classifier") + @add_start_docstrings_to_callable(ROBERTA_INPUTS_DOCSTRING) def call(self, inputs, **kwargs): r""" Return: @@ -399,6 +402,7 @@ class TFRobertaForTokenClassification(TFRobertaPreTrainedModel): config.num_labels, kernel_initializer=get_initializer(config.initializer_range), name="classifier" ) + @add_start_docstrings_to_callable(ROBERTA_INPUTS_DOCSTRING) def call(self, inputs, **kwargs): r""" Return: diff --git a/src/transformers/modeling_tf_transfo_xl.py b/src/transformers/modeling_tf_transfo_xl.py index 92b251ca42..659685388e 100644 --- a/src/transformers/modeling_tf_transfo_xl.py +++ b/src/transformers/modeling_tf_transfo_xl.py @@ -630,19 +630,19 @@ class TFTransfoXLPreTrainedModel(TFPreTrainedModel): base_model_prefix = "transformer" -TRANSFO_XL_START_DOCSTRING = r""" +TRANSFO_XL_START_DOCSTRING = r""" .. note:: - + TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. - This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having + This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. - If you choose this second option, there are three possibilities you can use to gather all the input Tensors + If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` @@ -660,12 +660,12 @@ TRANSFO_XL_START_DOCSTRING = r""" TRANSFO_XL_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.TransfoXLTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model diff --git a/src/transformers/modeling_tf_xlm.py b/src/transformers/modeling_tf_xlm.py index 6dd2215ce8..44b991d08c 100644 --- a/src/transformers/modeling_tf_xlm.py +++ b/src/transformers/modeling_tf_xlm.py @@ -485,18 +485,18 @@ class TFXLMPreTrainedModel(TFPreTrainedModel): XLM_START_DOCSTRING = r""" - + .. note:: - + TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. - This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having + This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. - If you choose this second option, there are three possibilities you can use to gather all the input Tensors + If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` @@ -504,7 +504,7 @@ XLM_START_DOCSTRING = r""" :obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: :obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})` - + Parameters: config (:class:`~transformers.XLMConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. @@ -514,39 +514,39 @@ XLM_START_DOCSTRING = r""" XLM_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - langs (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + langs (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): A parallel sequence of tokens to be used to indicate the language of each token in the input. Indices are languages ids which can be obtained from the language names by using two conversion mappings provided in the configuration of the model (only provided for multilingual models). More precisely, the `language name -> language id` mapping is in `model.config.lang2id` (dict str -> int) and the `language id -> language name` mapping is `model.config.id2lang` (dict int -> str). - + See usage examples detailed in the `multilingual documentation `__. - token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`_ - lengths (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): + lengths (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Length of each sentence that can be used to avoid performing attention on padding token indices. You can also use `attention_mask` for the same result (see above), kept here for compatbility. Indices selected in ``[0, ..., input_ids.size(-1)]``: diff --git a/src/transformers/modeling_tf_xlnet.py b/src/transformers/modeling_tf_xlnet.py index a7cce9e350..6f4d56252c 100644 --- a/src/transformers/modeling_tf_xlnet.py +++ b/src/transformers/modeling_tf_xlnet.py @@ -694,19 +694,19 @@ class TFXLNetPreTrainedModel(TFPreTrainedModel): base_model_prefix = "transformer" -XLNET_START_DOCSTRING = r""" +XLNET_START_DOCSTRING = r""" .. note:: - + TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. - This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having + This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. - If you choose this second option, there are three possibilities you can use to gather all the input Tensors + If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` @@ -724,18 +724,18 @@ XLNET_START_DOCSTRING = r""" XLNET_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.XLNetTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ mems (:obj:`List[tf.Tensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model @@ -747,17 +747,17 @@ XLNET_INPUTS_DOCSTRING = r""" if ``perm_mask[k, i, j] = 1``, i does not attend to j in batch k. If None, each token attends to all the others (full bidirectional attention). Only used during pretraining (to define factorization order) or for sequential decoding (generation). - target_mapping (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, num_predict, sequence_length)`, `optional`, defaults to :obj:`None`): + target_mapping (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, num_predict, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to indicate the output tokens to use. If ``target_mapping[k, i, j] = 1``, the i-th predict in batch k is on the j-th token. Only used during pretraining for partial prediction or for sequential decoding (generation). - token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ - input_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + input_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Negative of `attention_mask`, i.e. with 0 for real tokens and 1 for padding. Kept for compatibility with the original code base. diff --git a/src/transformers/modeling_transfo_xl.py b/src/transformers/modeling_transfo_xl.py index 6fe846e448..85bb26a2a8 100644 --- a/src/transformers/modeling_transfo_xl.py +++ b/src/transformers/modeling_transfo_xl.py @@ -508,10 +508,10 @@ class TransfoXLPreTrainedModel(PreTrainedModel): self._init_bias(m.r_bias) -TRANSFO_XL_START_DOCSTRING = r""" +TRANSFO_XL_START_DOCSTRING = r""" - This model is a PyTorch `torch.nn.Module `_ sub-class. - Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: @@ -523,12 +523,12 @@ TRANSFO_XL_START_DOCSTRING = r""" TRANSFO_XL_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.TransfoXLTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model diff --git a/src/transformers/modeling_xlm.py b/src/transformers/modeling_xlm.py index 3fa046dfa5..de980eb7b6 100644 --- a/src/transformers/modeling_xlm.py +++ b/src/transformers/modeling_xlm.py @@ -251,6 +251,10 @@ class XLMPreTrainedModel(PreTrainedModel): XLM_START_DOCSTRING = r""" + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general + usage and behavior. + Parameters: config (:class:`~transformers.XLMConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. @@ -260,39 +264,39 @@ XLM_START_DOCSTRING = r""" XLM_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ - langs (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + langs (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): A parallel sequence of tokens to be used to indicate the language of each token in the input. Indices are languages ids which can be obtained from the language names by using two conversion mappings provided in the configuration of the model (only provided for multilingual models). More precisely, the `language name -> language id` mapping is in `model.config.lang2id` (dict str -> int) and the `language id -> language name` mapping is `model.config.id2lang` (dict int -> str). - + See usage examples detailed in the `multilingual documentation `__. - token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. - + `What are position IDs? <../glossary.html#position-ids>`_ - lengths (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): + lengths (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Length of each sentence that can be used to avoid performing attention on padding token indices. You can also use `attention_mask` for the same result (see above), kept here for compatbility. Indices selected in ``[0, ..., input_ids.size(-1)]``: diff --git a/src/transformers/modeling_xlm_roberta.py b/src/transformers/modeling_xlm_roberta.py index 1282bcea88..c00a2eb4f5 100644 --- a/src/transformers/modeling_xlm_roberta.py +++ b/src/transformers/modeling_xlm_roberta.py @@ -41,9 +41,11 @@ XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP = { } -XLM_ROBERTA_START_DOCSTRING = r""" - This model is a PyTorch `torch.nn.Module`_ sub-class. Use it as a regular PyTorch Module and - refer to the PyTorch documentation for all matter related to general usage and behavior. +XLM_ROBERTA_START_DOCSTRING = r""" + + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general + usage and behavior. Parameters: config (:class:`~transformers.XLMRobertaConfig`): Model configuration class with all the parameters of the diff --git a/src/transformers/modeling_xlnet.py b/src/transformers/modeling_xlnet.py index 077a1dfdc7..a0c5cebe59 100644 --- a/src/transformers/modeling_xlnet.py +++ b/src/transformers/modeling_xlnet.py @@ -505,9 +505,10 @@ class XLNetPreTrainedModel(PreTrainedModel): module.mask_emb.data.normal_(mean=0.0, std=self.config.initializer_range) -XLNET_START_DOCSTRING = r""" - This model is a PyTorch `torch.nn.Module `_ sub-class. - Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general +XLNET_START_DOCSTRING = r""" + + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: @@ -519,18 +520,18 @@ XLNET_START_DOCSTRING = r""" XLNET_INPUTS_DOCSTRING = r""" Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): - Indices of input sequence tokens in the vocabulary. - + Indices of input sequence tokens in the vocabulary. + Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. - + `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. - + `What are attention masks? <../glossary.html#attention-mask>`__ mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model @@ -542,17 +543,17 @@ XLNET_INPUTS_DOCSTRING = r""" if ``perm_mask[k, i, j] = 1``, i does not attend to j in batch k. If None, each token attends to all the others (full bidirectional attention). Only used during pretraining (to define factorization order) or for sequential decoding (generation). - target_mapping (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_predict, sequence_length)`, `optional`, defaults to :obj:`None`): + target_mapping (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_predict, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to indicate the output tokens to use. If ``target_mapping[k, i, j] = 1``, the i-th predict in batch k is on the j-th token. Only used during pretraining for partial prediction or for sequential decoding (generation). - token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token - + `What are token type IDs? <../glossary.html#token-type-ids>`_ - input_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + input_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Negative of `attention_mask`, i.e. with 0 for real tokens and 1 for padding. Kept for compatibility with the original code base.