diff --git a/README.md b/README.md index 9f161b8364..e0dd8d9f51 100644 --- a/README.md +++ b/README.md @@ -173,8 +173,9 @@ Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 23. **[Pegasus](https://github.com/google-research/pegasus)** (from Google) released with the paper [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://arxiv.org/abs/1912.08777)> by Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu. 24. **[MBart](https://github.com/pytorch/fairseq/tree/master/examples/mbart)** (from Facebook) released with the paper [Multilingual Denoising Pre-training for Neural Machine Translation](https://arxiv.org/abs/2001.08210) by Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer. 25. **[LXMERT](https://github.com/airsplay/lxmert)** (from UNC Chapel Hill) released with the paper [LXMERT: Learning Cross-Modality Encoder Representations from Transformers for Open-Domain Question Answering](https://arxiv.org/abs/1908.07490) by Hao Tan and Mohit Bansal. -26. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users). -27. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR. +26. **[Funnel Transformer](https://github.com/laiguokun/Funnel-Transformer)** (from CMU/Google Brain) released with the paper [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://arxiv.org/abs/2006.03236) by Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le. +27. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users). +28. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR. These implementations have been tested on several datasets (see the example scripts) and should match the performances of the original implementations (e.g. ~93 F1 on SQuAD for BERT Whole-Word-Masking, ~88 F1 on RocStories for OpenAI GPT, ~18.3 perplexity on WikiText 103 for Transformer-XL, ~0.916 Pearson R coefficient on STS-B for XLNet). You can find more details on the performances in the Examples section of the [documentation](https://huggingface.co/transformers/examples.html). diff --git a/docs/source/index.rst b/docs/source/index.rst index a2acc39466..dc034c5fbb 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -131,7 +131,10 @@ conversion utilities for the following models: 25. `LXMERT `_ (from UNC Chapel Hill) released with the paper `LXMERT: Learning Cross-Modality Encoder Representations from Transformers for Open-Domain Question Answering `_ by Hao Tan and Mohit Bansal. -26. `Other community models `_, contributed by the `community +26. `Funnel Transformer `_ (from CMU/Google Brain) released with the paper + `Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing + `_ by Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le. +27. `Other community models `_, contributed by the `community `_. .. toctree:: @@ -216,6 +219,7 @@ conversion utilities for the following models: model_doc/dpr model_doc/pegasus model_doc/mbart + model_doc/funnel model_doc/lxmert internal/modeling_utils internal/tokenization_utils diff --git a/docs/source/model_doc/funnel.rst b/docs/source/model_doc/funnel.rst new file mode 100644 index 0000000000..6ecaa201e3 --- /dev/null +++ b/docs/source/model_doc/funnel.rst @@ -0,0 +1,126 @@ +Funnel Transformer +------------------ + +Overview +~~~~~~~~~~~~~~~~~~~~~ + +The Funnel Transformer model was proposed in the paper +`Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing +`__. +It is a bidirectional transformer model, like BERT, but with a pooling operation after each block of layers, a bit +like in traditional convolutional neural networks (CNN) in computer vision. + +The abstract from the paper is the following: + +*With the success of language pretraining, it is highly desirable to develop more efficient architectures of good +scalability that can exploit the abundant unlabeled data at a lower cost. To improve the efficiency, we examine the +much-overlooked redundancy in maintaining a full-length token-level presentation, especially for tasks that only +require a single-vector presentation of the sequence. With this intuition, we propose Funnel-Transformer which +gradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. More +importantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, we further +improve the model capacity. In addition, to perform token-level predictions as required by common pretraining +objectives, Funnel-Transformer is able to recover a deep representation for each token from the reduced hidden sequence +via a decoder. Empirically, with comparable or fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on +a wide variety of sequence-level prediction tasks, including text classification, language understanding, and reading +comprehension.* + +Tips: + +- Since Funnel Transformer uses pooling, the sequence length of the hidden states changes after each block of layers. + The base model therefore has a final sequence length that is a quarter of the original one. This model can be used + directly for tasks that just require a sentence summary (like sequence classification or multiple choice). For other + tasks, the full model is used; this full model has a decoder that upsamples the final hidden states to the same + sequence length as the input. +- The Funnel Transformer checkpoints are all available with a full version and a base version. The first ones should + be used for :class:`~transformers.FunnelModel`, :class:`~transformers.FunnelForPreTraining`, + :class:`~transformers.FunnelForMaskedLM`, :class:`~transformers.FunnelForTokenClassification` and + class:`~transformers.FunnelForQuestionAnswering`. The second ones should be used for + :class:`~transformers.FunnelBaseModel`, :class:`~transformers.FunnelForSequenceClassification` and + :class:`~transformers.FunnelForMultipleChoice`. + +The original code can be found `here `_. + + +FunnelConfig +~~~~~~~~~~~~ + +.. autoclass:: transformers.FunnelConfig + :members: + + +FunnelTokenizer +~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.FunnelTokenizer + :members: build_inputs_with_special_tokens, get_special_tokens_mask, + create_token_type_ids_from_sequences, save_vocabulary + + +FunnelTokenizerFast +~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.FunnelTokenizerFast + :members: + + +Funnel specific outputs +~~~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.modeling_funnel.FunnelForPreTrainingOutput + :members: + + +FunnelBaseModel +~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.FunnelBaseModel + :members: + + +FunnelModel +~~~~~~~~~~~ + +.. autoclass:: transformers.FunnelModel + :members: + + +FunnelModelForPreTraining +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.FunnelForPreTraining + :members: + + +FunnelForMaskedLM +~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.FunnelForMaskedLM + :members: + + +FunnelForSequenceClassification +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.FunnelForSequenceClassification + :members: + + +FunnelForMultipleChoice +~~~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.FunnelForMultipleChoice + :members: + + +FunnelForTokenClassification +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.FunnelForTokenClassification + :members: + + +FunnelForQuestionAnswering +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.FunnelForQuestionAnswering + :members: diff --git a/docs/source/model_summary.rst b/docs/source/model_summary.rst index 9ddf38f7b6..fe253a266a 100644 --- a/docs/source/model_summary.rst +++ b/docs/source/model_summary.rst @@ -416,6 +416,38 @@ traditional GAN setting) then the ELECTRA model is trained for a few steps. The library provides a version of the model for masked language modeling, token classification and sentence classification. +Funnel Transformer +---------------------------------------------- + +.. raw:: html + + + Models + + + Doc + + +`Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing +`_, Zihang Dai et al. + +Funnel Transformer is a transformer model using pooling, a bit like a ResNet model: layers are grouped in blocks, and +at the beginning of each block (except the first one), the hidden states are pooled among the sequence dimension. This +way, their length is divided by 2, which speeds up the computation of the next hidden states. All pretrained models +have three blocks, which means the final hidden state has a sequence length that is one fourth of the original sequence +length. + +For tasks such as classification, this is not a problem, but for tasks like masked language modeling or token +classification, we need a hidden state with the same sequence length as the original input. In those cases, the final +hidden states are upsampled to the input sequence length and go through two additional layers. That's why there are two +versions of each checkpoint. The version suffixed with "-base" contains only the three blocks, while the version +without that suffix contains the three blocks and the upsampling head with its additional layers. + +The pretrained models available use the same pretraining objective as ELECTRA. + +The library provides a version of the model for masked language modeling, token classification, sentence +classification, multiple choice classification and question answering. + .. _longformer: Longformer diff --git a/docs/source/pretrained_models.rst b/docs/source/pretrained_models.rst index 5acdb9f761..9c277f4a83 100644 --- a/docs/source/pretrained_models.rst +++ b/docs/source/pretrained_models.rst @@ -5,366 +5,406 @@ Here is the full list of the currently provided pretrained models together with For a list that includes community-uploaded models, refer to `https://huggingface.co/models `__. -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| Architecture | Shortcut name | Details of the model | -+===================+============================================================+=======================================================================================================================================+ -| BERT | ``bert-base-uncased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on lower-cased English text. | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-large-uncased`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. | -| | | | Trained on lower-cased English text. | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-base-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on cased English text. | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-large-cased`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. | -| | | | Trained on cased English text. | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-base-multilingual-uncased`` | | (Original, not recommended) 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on lower-cased text in the top 102 languages with the largest Wikipedias | -| | | | -| | | (see `details `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-base-multilingual-cased`` | | (New, **recommended**) 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on cased text in the top 104 languages with the largest Wikipedias | -| | | | -| | | (see `details `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-base-chinese`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on cased Chinese Simplified and Traditional text. | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-base-german-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on cased German text by Deepset.ai | -| | | | -| | | (see `details on deepset.ai website `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-large-uncased-whole-word-masking`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. | -| | | | Trained on lower-cased English text using Whole-Word-Masking | -| | | | -| | | (see `details `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-large-cased-whole-word-masking`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. | -| | | | Trained on cased English text using Whole-Word-Masking | -| | | | -| | | (see `details `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-large-uncased-whole-word-masking-finetuned-squad`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. | -| | | | The ``bert-large-uncased-whole-word-masking`` model fine-tuned on SQuAD | -| | | | -| | | (see details of fine-tuning in the `example section `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-large-cased-whole-word-masking-finetuned-squad`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters | -| | | | The ``bert-large-cased-whole-word-masking`` model fine-tuned on SQuAD | -| | | | -| | | (see `details of fine-tuning in the example section `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-base-cased-finetuned-mrpc`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | The ``bert-base-cased`` model fine-tuned on MRPC | -| | | | -| | | (see `details of fine-tuning in the example section `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-base-german-dbmdz-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on cased German text by DBMDZ | -| | | | -| | | (see `details on dbmdz repository `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``bert-base-german-dbmdz-uncased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on uncased German text by DBMDZ | -| | | | -| | | (see `details on dbmdz repository `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``cl-tohoku/bert-base-japanese`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on Japanese text. Text is tokenized with MeCab and WordPiece and this requires some extra dependencies, | -| | | | `fugashi `__ which is a wrapper around `MeCab `__. | -| | | | Use ``pip install transformers["ja"]`` (or ``pip install -e .["ja"]`` if you install from source) to install them. | -| | | | -| | | (see `details on cl-tohoku repository `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``cl-tohoku/bert-base-japanese-whole-word-masking`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on Japanese text. Text is tokenized with MeCab and WordPiece and this requires some extra dependencies, | -| | | | `fugashi `__ which is a wrapper around `MeCab `__. | -| | | | Use ``pip install transformers["ja"]`` (or ``pip install -e .["ja"]`` if you install from source) to install them. | -| | | | -| | | (see `details on cl-tohoku repository `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``cl-tohoku/bert-base-japanese-char`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on Japanese text. Text is tokenized into characters. | -| | | | -| | | (see `details on cl-tohoku repository `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``cl-tohoku/bert-base-japanese-char-whole-word-masking`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on Japanese text using Whole-Word-Masking. Text is tokenized into characters. | -| | | | -| | | (see `details on cl-tohoku repository `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``TurkuNLP/bert-base-finnish-cased-v1`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on cased Finnish text. | -| | | | -| | | (see `details on turkunlp.org `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``TurkuNLP/bert-base-finnish-uncased-v1`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on uncased Finnish text. | -| | | | -| | | (see `details on turkunlp.org `__). | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``wietsedv/bert-base-dutch-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | Trained on cased Dutch text. | -| | | | -| | | (see `details on wietsedv repository `__). | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| GPT | ``openai-gpt`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | OpenAI GPT English model | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| GPT-2 | ``gpt2`` | | 12-layer, 768-hidden, 12-heads, 117M parameters. | -| | | | OpenAI GPT-2 English model | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``gpt2-medium`` | | 24-layer, 1024-hidden, 16-heads, 345M parameters. | -| | | | OpenAI's Medium-sized GPT-2 English model | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``gpt2-large`` | | 36-layer, 1280-hidden, 20-heads, 774M parameters. | -| | | | OpenAI's Large-sized GPT-2 English model | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``gpt2-xl`` | | 48-layer, 1600-hidden, 25-heads, 1558M parameters. | -| | | | OpenAI's XL-sized GPT-2 English model | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| Transformer-XL | ``transfo-xl-wt103`` | | 18-layer, 1024-hidden, 16-heads, 257M parameters. | -| | | | English model trained on wikitext-103 | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| XLNet | ``xlnet-base-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | -| | | | XLNet English model | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``xlnet-large-cased`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. | -| | | | XLNet Large English model | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| XLM | ``xlm-mlm-en-2048`` | | 12-layer, 2048-hidden, 16-heads | -| | | | XLM English model | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``xlm-mlm-ende-1024`` | | 6-layer, 1024-hidden, 8-heads | -| | | | XLM English-German model trained on the concatenation of English and German wikipedia | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``xlm-mlm-enfr-1024`` | | 6-layer, 1024-hidden, 8-heads | -| | | | XLM English-French model trained on the concatenation of English and French wikipedia | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``xlm-mlm-enro-1024`` | | 6-layer, 1024-hidden, 8-heads | -| | | | XLM English-Romanian Multi-language model | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``xlm-mlm-xnli15-1024`` | | 12-layer, 1024-hidden, 8-heads | -| | | | XLM Model pre-trained with MLM on the `15 XNLI languages `__. | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``xlm-mlm-tlm-xnli15-1024`` | | 12-layer, 1024-hidden, 8-heads | -| | | | XLM Model pre-trained with MLM + TLM on the `15 XNLI languages `__. | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``xlm-clm-enfr-1024`` | | 6-layer, 1024-hidden, 8-heads | -| | | | XLM English-French model trained with CLM (Causal Language Modeling) on the concatenation of English and French wikipedia | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``xlm-clm-ende-1024`` | | 6-layer, 1024-hidden, 8-heads | -| | | | XLM English-German model trained with CLM (Causal Language Modeling) on the concatenation of English and German wikipedia | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``xlm-mlm-17-1280`` | | 16-layer, 1280-hidden, 16-heads | -| | | | XLM model trained with MLM (Masked Language Modeling) on 17 languages. | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``xlm-mlm-100-1280`` | | 16-layer, 1280-hidden, 16-heads | -| | | | XLM model trained with MLM (Masked Language Modeling) on 100 languages. | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| RoBERTa | ``roberta-base`` | | 12-layer, 768-hidden, 12-heads, 125M parameters | -| | | | RoBERTa using the BERT-base architecture | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``roberta-large`` | | 24-layer, 1024-hidden, 16-heads, 355M parameters | -| | | | RoBERTa using the BERT-large architecture | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``roberta-large-mnli`` | | 24-layer, 1024-hidden, 16-heads, 355M parameters | -| | | | ``roberta-large`` fine-tuned on `MNLI `__. | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``distilroberta-base`` | | 6-layer, 768-hidden, 12-heads, 82M parameters | -| | | | The DistilRoBERTa model distilled from the RoBERTa model `roberta-base` checkpoint. | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``roberta-base-openai-detector`` | | 12-layer, 768-hidden, 12-heads, 125M parameters | -| | | | ``roberta-base`` fine-tuned by OpenAI on the outputs of the 1.5B-parameter GPT-2 model. | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``roberta-large-openai-detector`` | | 24-layer, 1024-hidden, 16-heads, 355M parameters | -| | | | ``roberta-large`` fine-tuned by OpenAI on the outputs of the 1.5B-parameter GPT-2 model. | -| | | | -| | | (see `details `__) | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| DistilBERT | ``distilbert-base-uncased`` | | 6-layer, 768-hidden, 12-heads, 66M parameters | -| | | | The DistilBERT model distilled from the BERT model `bert-base-uncased` checkpoint | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``distilbert-base-uncased-distilled-squad`` | | 6-layer, 768-hidden, 12-heads, 66M parameters | -| | | | The DistilBERT model distilled from the BERT model `bert-base-uncased` checkpoint, with an additional linear layer. | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``distilbert-base-cased`` | | 6-layer, 768-hidden, 12-heads, 65M parameters | -| | | | The DistilBERT model distilled from the BERT model `bert-base-cased` checkpoint | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``distilbert-base-cased-distilled-squad`` | | 6-layer, 768-hidden, 12-heads, 65M parameters | -| | | | The DistilBERT model distilled from the BERT model `bert-base-cased` checkpoint, with an additional question answering layer. | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``distilgpt2`` | | 6-layer, 768-hidden, 12-heads, 82M parameters | -| | | | The DistilGPT2 model distilled from the GPT2 model `gpt2` checkpoint. | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``distilbert-base-german-cased`` | | 6-layer, 768-hidden, 12-heads, 66M parameters | -| | | | The German DistilBERT model distilled from the German DBMDZ BERT model `bert-base-german-dbmdz-cased` checkpoint. | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``distilbert-base-multilingual-cased`` | | 6-layer, 768-hidden, 12-heads, 134M parameters | -| | | | The multilingual DistilBERT model distilled from the Multilingual BERT model `bert-base-multilingual-cased` checkpoint. | -| | | | -| | | (see `details `__) | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| CTRL | ``ctrl`` | | 48-layer, 1280-hidden, 16-heads, 1.6B parameters | -| | | | Salesforce's Large-sized CTRL English model | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| CamemBERT | ``camembert-base`` | | 12-layer, 768-hidden, 12-heads, 110M parameters | -| | | | CamemBERT using the BERT-base architecture | -| | | | -| | | (see `details `__) | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| ALBERT | ``albert-base-v1`` | | 12 repeating layers, 128 embedding, 768-hidden, 12-heads, 11M parameters | -| | | | ALBERT base model | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``albert-large-v1`` | | 24 repeating layers, 128 embedding, 1024-hidden, 16-heads, 17M parameters | -| | | | ALBERT large model | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``albert-xlarge-v1`` | | 24 repeating layers, 128 embedding, 2048-hidden, 16-heads, 58M parameters | -| | | | ALBERT xlarge model | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``albert-xxlarge-v1`` | | 12 repeating layer, 128 embedding, 4096-hidden, 64-heads, 223M parameters | -| | | | ALBERT xxlarge model | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``albert-base-v2`` | | 12 repeating layers, 128 embedding, 768-hidden, 12-heads, 11M parameters | -| | | | ALBERT base model with no dropout, additional training data and longer training | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``albert-large-v2`` | | 24 repeating layers, 128 embedding, 1024-hidden, 16-heads, 17M parameters | -| | | | ALBERT large model with no dropout, additional training data and longer training | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``albert-xlarge-v2`` | | 24 repeating layers, 128 embedding, 2048-hidden, 16-heads, 58M parameters | -| | | | ALBERT xlarge model with no dropout, additional training data and longer training | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``albert-xxlarge-v2`` | | 12 repeating layer, 128 embedding, 4096-hidden, 64-heads, 223M parameters | -| | | | ALBERT xxlarge model with no dropout, additional training data and longer training | -| | | | -| | | (see `details `__) | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| T5 | ``t5-small`` | | ~60M parameters with 6-layers, 512-hidden-state, 2048 feed-forward hidden-state, 8-heads, | -| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``t5-base`` | | ~220M parameters with 12-layers, 768-hidden-state, 3072 feed-forward hidden-state, 12-heads, | -| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``t5-large`` | | ~770M parameters with 24-layers, 1024-hidden-state, 4096 feed-forward hidden-state, 16-heads, | -| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``t5-3B`` | | ~2.8B parameters with 24-layers, 1024-hidden-state, 16384 feed-forward hidden-state, 32-heads, | -| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``t5-11B`` | | ~11B parameters with 24-layers, 1024-hidden-state, 65536 feed-forward hidden-state, 128-heads, | -| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| XLM-RoBERTa | ``xlm-roberta-base`` | | ~125M parameters with 12-layers, 768-hidden-state, 3072 feed-forward hidden-state, 8-heads, | -| | | | Trained on on 2.5 TB of newly created clean CommonCrawl data in 100 languages | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``xlm-roberta-large`` | | ~355M parameters with 24-layers, 1027-hidden-state, 4096 feed-forward hidden-state, 16-heads, | -| | | | Trained on 2.5 TB of newly created clean CommonCrawl data in 100 languages | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| FlauBERT | ``flaubert/flaubert_small_cased`` | | 6-layer, 512-hidden, 8-heads, 54M parameters | -| | | | FlauBERT small architecture | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``flaubert/flaubert_base_uncased`` | | 12-layer, 768-hidden, 12-heads, 137M parameters | -| | | | FlauBERT base architecture with uncased vocabulary | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``flaubert/flaubert_base_cased`` | | 12-layer, 768-hidden, 12-heads, 138M parameters | -| | | | FlauBERT base architecture with cased vocabulary | -| | | | -| | | (see `details `__) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``flaubert/flaubert_large_cased`` | | 24-layer, 1024-hidden, 16-heads, 373M parameters | -| | | | FlauBERT large architecture | -| | | | -| | | (see `details `__) | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| Bart | ``facebook/bart-large`` | | 24-layer, 1024-hidden, 16-heads, 406M parameters | -| | | | -| | | (see `details `_) | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``facebook/bart-base`` | | 12-layer, 768-hidden, 16-heads, 139M parameters | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``facebook/bart-large-mnli`` | | Adds a 2 layer classification head with 1 million parameters | -| | | | bart-large base architecture with a classification head, finetuned on MNLI | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``facebook/bart-large-cnn`` | | 12-layer, 1024-hidden, 16-heads, 406M parameters (same as base) | -| | | | bart-large base architecture finetuned on cnn summarization task | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| DialoGPT | ``DialoGPT-small`` | | 12-layer, 768-hidden, 12-heads, 124M parameters | -| | | | Trained on English text: 147M conversation-like exchanges extracted from Reddit. | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``DialoGPT-medium`` | | 24-layer, 1024-hidden, 16-heads, 355M parameters | -| | | | Trained on English text: 147M conversation-like exchanges extracted from Reddit. | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``DialoGPT-large`` | | 36-layer, 1280-hidden, 20-heads, 774M parameters | -| | | | Trained on English text: 147M conversation-like exchanges extracted from Reddit. | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| Reformer | ``reformer-enwik8`` | | 12-layer, 1024-hidden, 8-heads, 149M parameters | -| | | | Trained on English Wikipedia data - enwik8. | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``reformer-crime-and-punishment`` | | 6-layer, 256-hidden, 2-heads, 3M parameters | -| | | | Trained on English text: Crime and Punishment novel by Fyodor Dostoyevsky. | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| MarianMT | ``Helsinki-NLP/opus-mt-{src}-{tgt}`` | | 12-layer, 512-hidden, 8-heads, ~74M parameter Machine translation models. Parameter counts vary depending on vocab size. | -| | | | (see `model list `_) | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| Pegasus | ``google/pegasus-{dataset}`` | | 16-layer, 1024-hidden, 16-heads, ~568M parameter, 2.2 GB for summary. `model list `__ | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| Longformer | ``allenai/longformer-base-4096`` | | 12-layer, 768-hidden, 12-heads, ~149M parameters | -| | | | Starting from RoBERTa-base checkpoint, trained on documents of max length 4,096 | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``allenai/longformer-large-4096`` | | 24-layer, 1024-hidden, 16-heads, ~435M parameters | -| | | | Starting from RoBERTa-large checkpoint, trained on documents of max length 4,096 | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| MBart | ``facebook/mbart-large-cc25`` | | 24-layer, 1024-hidden, 16-heads, 610M parameters | -| | | | mBART (bart-large architecture) model trained on 25 languages' monolingual corpus | -| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| | ``facebook/mbart-large-en-ro`` | | 24-layer, 1024-hidden, 16-heads, 610M parameters | -| | | | mbart-large-cc25 model finetuned on WMT english romanian translation. | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ -| Lxmert | ``lxmert-base-uncased`` | | 9-language layers, 9-relationship layers, and 12-cross-modality layers | -| | | | 768-hidden, 12-heads (for each layer) ~ 228M parameters | -| | | | Starting from lxmert-base checkpoint, trained on over 9 million image-text couplets from COCO, VisualGenome, GQA, VQA | -+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| Architecture | Shortcut name | Details of the model | ++====================+============================================================+=======================================================================================================================================+ +| BERT | ``bert-base-uncased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on lower-cased English text. | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-large-uncased`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. | +| | | | Trained on lower-cased English text. | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-base-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on cased English text. | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-large-cased`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. | +| | | | Trained on cased English text. | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-base-multilingual-uncased`` | | (Original, not recommended) 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on lower-cased text in the top 102 languages with the largest Wikipedias | +| | | | +| | | (see `details `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-base-multilingual-cased`` | | (New, **recommended**) 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on cased text in the top 104 languages with the largest Wikipedias | +| | | | +| | | (see `details `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-base-chinese`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on cased Chinese Simplified and Traditional text. | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-base-german-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on cased German text by Deepset.ai | +| | | | +| | | (see `details on deepset.ai website `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-large-uncased-whole-word-masking`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. | +| | | | Trained on lower-cased English text using Whole-Word-Masking | +| | | | +| | | (see `details `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-large-cased-whole-word-masking`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. | +| | | | Trained on cased English text using Whole-Word-Masking | +| | | | +| | | (see `details `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-large-uncased-whole-word-masking-finetuned-squad`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. | +| | | | The ``bert-large-uncased-whole-word-masking`` model fine-tuned on SQuAD | +| | | | +| | | (see details of fine-tuning in the `example section `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-large-cased-whole-word-masking-finetuned-squad`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters | +| | | | The ``bert-large-cased-whole-word-masking`` model fine-tuned on SQuAD | +| | | | +| | | (see `details of fine-tuning in the example section `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-base-cased-finetuned-mrpc`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | The ``bert-base-cased`` model fine-tuned on MRPC | +| | | | +| | | (see `details of fine-tuning in the example section `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-base-german-dbmdz-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on cased German text by DBMDZ | +| | | | +| | | (see `details on dbmdz repository `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``bert-base-german-dbmdz-uncased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on uncased German text by DBMDZ | +| | | | +| | | (see `details on dbmdz repository `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``cl-tohoku/bert-base-japanese`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on Japanese text. Text is tokenized with MeCab and WordPiece and this requires some extra dependencies, | +| | | | `fugashi `__ which is a wrapper around `MeCab `__. | +| | | | Use ``pip install transformers["ja"]`` (or ``pip install -e .["ja"]`` if you install from source) to install them. | +| | | | +| | | (see `details on cl-tohoku repository `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``cl-tohoku/bert-base-japanese-whole-word-masking`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on Japanese text. Text is tokenized with MeCab and WordPiece and this requires some extra dependencies, | +| | | | `fugashi `__ which is a wrapper around `MeCab `__. | +| | | | Use ``pip install transformers["ja"]`` (or ``pip install -e .["ja"]`` if you install from source) to install them. | +| | | | +| | | (see `details on cl-tohoku repository `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``cl-tohoku/bert-base-japanese-char`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on Japanese text. Text is tokenized into characters. | +| | | | +| | | (see `details on cl-tohoku repository `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``cl-tohoku/bert-base-japanese-char-whole-word-masking`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on Japanese text using Whole-Word-Masking. Text is tokenized into characters. | +| | | | +| | | (see `details on cl-tohoku repository `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``TurkuNLP/bert-base-finnish-cased-v1`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on cased Finnish text. | +| | | | +| | | (see `details on turkunlp.org `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``TurkuNLP/bert-base-finnish-uncased-v1`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on uncased Finnish text. | +| | | | +| | | (see `details on turkunlp.org `__). | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``wietsedv/bert-base-dutch-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | Trained on cased Dutch text. | +| | | | +| | | (see `details on wietsedv repository `__). | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| GPT | ``openai-gpt`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | OpenAI GPT English model | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| GPT-2 | ``gpt2`` | | 12-layer, 768-hidden, 12-heads, 117M parameters. | +| | | | OpenAI GPT-2 English model | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``gpt2-medium`` | | 24-layer, 1024-hidden, 16-heads, 345M parameters. | +| | | | OpenAI's Medium-sized GPT-2 English model | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``gpt2-large`` | | 36-layer, 1280-hidden, 20-heads, 774M parameters. | +| | | | OpenAI's Large-sized GPT-2 English model | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``gpt2-xl`` | | 48-layer, 1600-hidden, 25-heads, 1558M parameters. | +| | | | OpenAI's XL-sized GPT-2 English model | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| Transformer-XL | ``transfo-xl-wt103`` | | 18-layer, 1024-hidden, 16-heads, 257M parameters. | +| | | | English model trained on wikitext-103 | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| XLNet | ``xlnet-base-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. | +| | | | XLNet English model | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``xlnet-large-cased`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. | +| | | | XLNet Large English model | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| XLM | ``xlm-mlm-en-2048`` | | 12-layer, 2048-hidden, 16-heads | +| | | | XLM English model | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``xlm-mlm-ende-1024`` | | 6-layer, 1024-hidden, 8-heads | +| | | | XLM English-German model trained on the concatenation of English and German wikipedia | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``xlm-mlm-enfr-1024`` | | 6-layer, 1024-hidden, 8-heads | +| | | | XLM English-French model trained on the concatenation of English and French wikipedia | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``xlm-mlm-enro-1024`` | | 6-layer, 1024-hidden, 8-heads | +| | | | XLM English-Romanian Multi-language model | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``xlm-mlm-xnli15-1024`` | | 12-layer, 1024-hidden, 8-heads | +| | | | XLM Model pre-trained with MLM on the `15 XNLI languages `__. | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``xlm-mlm-tlm-xnli15-1024`` | | 12-layer, 1024-hidden, 8-heads | +| | | | XLM Model pre-trained with MLM + TLM on the `15 XNLI languages `__. | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``xlm-clm-enfr-1024`` | | 6-layer, 1024-hidden, 8-heads | +| | | | XLM English-French model trained with CLM (Causal Language Modeling) on the concatenation of English and French wikipedia | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``xlm-clm-ende-1024`` | | 6-layer, 1024-hidden, 8-heads | +| | | | XLM English-German model trained with CLM (Causal Language Modeling) on the concatenation of English and German wikipedia | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``xlm-mlm-17-1280`` | | 16-layer, 1280-hidden, 16-heads | +| | | | XLM model trained with MLM (Masked Language Modeling) on 17 languages. | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``xlm-mlm-100-1280`` | | 16-layer, 1280-hidden, 16-heads | +| | | | XLM model trained with MLM (Masked Language Modeling) on 100 languages. | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| RoBERTa | ``roberta-base`` | | 12-layer, 768-hidden, 12-heads, 125M parameters | +| | | | RoBERTa using the BERT-base architecture | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``roberta-large`` | | 24-layer, 1024-hidden, 16-heads, 355M parameters | +| | | | RoBERTa using the BERT-large architecture | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``roberta-large-mnli`` | | 24-layer, 1024-hidden, 16-heads, 355M parameters | +| | | | ``roberta-large`` fine-tuned on `MNLI `__. | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``distilroberta-base`` | | 6-layer, 768-hidden, 12-heads, 82M parameters | +| | | | The DistilRoBERTa model distilled from the RoBERTa model `roberta-base` checkpoint. | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``roberta-base-openai-detector`` | | 12-layer, 768-hidden, 12-heads, 125M parameters | +| | | | ``roberta-base`` fine-tuned by OpenAI on the outputs of the 1.5B-parameter GPT-2 model. | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``roberta-large-openai-detector`` | | 24-layer, 1024-hidden, 16-heads, 355M parameters | +| | | | ``roberta-large`` fine-tuned by OpenAI on the outputs of the 1.5B-parameter GPT-2 model. | +| | | | +| | | (see `details `__) | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| DistilBERT | ``distilbert-base-uncased`` | | 6-layer, 768-hidden, 12-heads, 66M parameters | +| | | | The DistilBERT model distilled from the BERT model `bert-base-uncased` checkpoint | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``distilbert-base-uncased-distilled-squad`` | | 6-layer, 768-hidden, 12-heads, 66M parameters | +| | | | The DistilBERT model distilled from the BERT model `bert-base-uncased` checkpoint, with an additional linear layer. | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``distilbert-base-cased`` | | 6-layer, 768-hidden, 12-heads, 65M parameters | +| | | | The DistilBERT model distilled from the BERT model `bert-base-cased` checkpoint | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``distilbert-base-cased-distilled-squad`` | | 6-layer, 768-hidden, 12-heads, 65M parameters | +| | | | The DistilBERT model distilled from the BERT model `bert-base-cased` checkpoint, with an additional question answering layer. | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``distilgpt2`` | | 6-layer, 768-hidden, 12-heads, 82M parameters | +| | | | The DistilGPT2 model distilled from the GPT2 model `gpt2` checkpoint. | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``distilbert-base-german-cased`` | | 6-layer, 768-hidden, 12-heads, 66M parameters | +| | | | The German DistilBERT model distilled from the German DBMDZ BERT model `bert-base-german-dbmdz-cased` checkpoint. | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``distilbert-base-multilingual-cased`` | | 6-layer, 768-hidden, 12-heads, 134M parameters | +| | | | The multilingual DistilBERT model distilled from the Multilingual BERT model `bert-base-multilingual-cased` checkpoint. | +| | | | +| | | (see `details `__) | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| CTRL | ``ctrl`` | | 48-layer, 1280-hidden, 16-heads, 1.6B parameters | +| | | | Salesforce's Large-sized CTRL English model | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| CamemBERT | ``camembert-base`` | | 12-layer, 768-hidden, 12-heads, 110M parameters | +| | | | CamemBERT using the BERT-base architecture | +| | | | +| | | (see `details `__) | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| ALBERT | ``albert-base-v1`` | | 12 repeating layers, 128 embedding, 768-hidden, 12-heads, 11M parameters | +| | | | ALBERT base model | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``albert-large-v1`` | | 24 repeating layers, 128 embedding, 1024-hidden, 16-heads, 17M parameters | +| | | | ALBERT large model | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``albert-xlarge-v1`` | | 24 repeating layers, 128 embedding, 2048-hidden, 16-heads, 58M parameters | +| | | | ALBERT xlarge model | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``albert-xxlarge-v1`` | | 12 repeating layer, 128 embedding, 4096-hidden, 64-heads, 223M parameters | +| | | | ALBERT xxlarge model | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``albert-base-v2`` | | 12 repeating layers, 128 embedding, 768-hidden, 12-heads, 11M parameters | +| | | | ALBERT base model with no dropout, additional training data and longer training | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``albert-large-v2`` | | 24 repeating layers, 128 embedding, 1024-hidden, 16-heads, 17M parameters | +| | | | ALBERT large model with no dropout, additional training data and longer training | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``albert-xlarge-v2`` | | 24 repeating layers, 128 embedding, 2048-hidden, 16-heads, 58M parameters | +| | | | ALBERT xlarge model with no dropout, additional training data and longer training | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``albert-xxlarge-v2`` | | 12 repeating layer, 128 embedding, 4096-hidden, 64-heads, 223M parameters | +| | | | ALBERT xxlarge model with no dropout, additional training data and longer training | +| | | | +| | | (see `details `__) | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| T5 | ``t5-small`` | | ~60M parameters with 6-layers, 512-hidden-state, 2048 feed-forward hidden-state, 8-heads, | +| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``t5-base`` | | ~220M parameters with 12-layers, 768-hidden-state, 3072 feed-forward hidden-state, 12-heads, | +| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``t5-large`` | | ~770M parameters with 24-layers, 1024-hidden-state, 4096 feed-forward hidden-state, 16-heads, | +| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``t5-3B`` | | ~2.8B parameters with 24-layers, 1024-hidden-state, 16384 feed-forward hidden-state, 32-heads, | +| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``t5-11B`` | | ~11B parameters with 24-layers, 1024-hidden-state, 65536 feed-forward hidden-state, 128-heads, | +| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| XLM-RoBERTa | ``xlm-roberta-base`` | | ~125M parameters with 12-layers, 768-hidden-state, 3072 feed-forward hidden-state, 8-heads, | +| | | | Trained on on 2.5 TB of newly created clean CommonCrawl data in 100 languages | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``xlm-roberta-large`` | | ~355M parameters with 24-layers, 1027-hidden-state, 4096 feed-forward hidden-state, 16-heads, | +| | | | Trained on 2.5 TB of newly created clean CommonCrawl data in 100 languages | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| FlauBERT | ``flaubert/flaubert_small_cased`` | | 6-layer, 512-hidden, 8-heads, 54M parameters | +| | | | FlauBERT small architecture | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``flaubert/flaubert_base_uncased`` | | 12-layer, 768-hidden, 12-heads, 137M parameters | +| | | | FlauBERT base architecture with uncased vocabulary | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``flaubert/flaubert_base_cased`` | | 12-layer, 768-hidden, 12-heads, 138M parameters | +| | | | FlauBERT base architecture with cased vocabulary | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``flaubert/flaubert_large_cased`` | | 24-layer, 1024-hidden, 16-heads, 373M parameters | +| | | | FlauBERT large architecture | +| | | | +| | | (see `details `__) | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| Bart | ``facebook/bart-large`` | | 24-layer, 1024-hidden, 16-heads, 406M parameters | +| | | | +| | | (see `details `_) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``facebook/bart-base`` | | 12-layer, 768-hidden, 16-heads, 139M parameters | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``facebook/bart-large-mnli`` | | Adds a 2 layer classification head with 1 million parameters | +| | | | bart-large base architecture with a classification head, finetuned on MNLI | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``facebook/bart-large-cnn`` | | 12-layer, 1024-hidden, 16-heads, 406M parameters (same as base) | +| | | | bart-large base architecture finetuned on cnn summarization task | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| DialoGPT | ``DialoGPT-small`` | | 12-layer, 768-hidden, 12-heads, 124M parameters | +| | | | Trained on English text: 147M conversation-like exchanges extracted from Reddit. | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``DialoGPT-medium`` | | 24-layer, 1024-hidden, 16-heads, 355M parameters | +| | | | Trained on English text: 147M conversation-like exchanges extracted from Reddit. | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``DialoGPT-large`` | | 36-layer, 1280-hidden, 20-heads, 774M parameters | +| | | | Trained on English text: 147M conversation-like exchanges extracted from Reddit. | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| Reformer | ``reformer-enwik8`` | | 12-layer, 1024-hidden, 8-heads, 149M parameters | +| | | | Trained on English Wikipedia data - enwik8. | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``reformer-crime-and-punishment`` | | 6-layer, 256-hidden, 2-heads, 3M parameters | +| | | | Trained on English text: Crime and Punishment novel by Fyodor Dostoyevsky. | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| MarianMT | ``Helsinki-NLP/opus-mt-{src}-{tgt}`` | | 12-layer, 512-hidden, 8-heads, ~74M parameter Machine translation models. Parameter counts vary depending on vocab size. | +| | | | (see `model list `_) | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| Pegasus | ``google/pegasus-{dataset}`` | | 16-layer, 1024-hidden, 16-heads, ~568M parameter, 2.2 GB for summary. `model list `__ | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| Longformer | ``allenai/longformer-base-4096`` | | 12-layer, 768-hidden, 12-heads, ~149M parameters | +| | | | Starting from RoBERTa-base checkpoint, trained on documents of max length 4,096 | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``allenai/longformer-large-4096`` | | 24-layer, 1024-hidden, 16-heads, ~435M parameters | +| | | | Starting from RoBERTa-large checkpoint, trained on documents of max length 4,096 | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| MBart | ``facebook/mbart-large-cc25`` | | 24-layer, 1024-hidden, 16-heads, 610M parameters | +| | | | mBART (bart-large architecture) model trained on 25 languages' monolingual corpus | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``facebook/mbart-large-en-ro`` | | 24-layer, 1024-hidden, 16-heads, 610M parameters | +| | | | mbart-large-cc25 model finetuned on WMT english romanian translation. | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| Lxmert | ``lxmert-base-uncased`` | | 9-language layers, 9-relationship layers, and 12-cross-modality layers | +| | | | 768-hidden, 12-heads (for each layer) ~ 228M parameters | +| | | | Starting from lxmert-base checkpoint, trained on over 9 million image-text couplets from COCO, VisualGenome, GQA, VQA | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| Funnel Transformer | ``funnel-transformer/small`` | | 14 layers: 3 blocks of 4 layers then 2 layers decoder, 768-hidden, 12-heads, 130M parameters | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``funnel-transformer/small-base`` | | 12 layers: 3 blocks of 4 layers (no decoder), 768-hidden, 12-heads, 115M parameters | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``funnel-transformer/medium`` | | 14 layers: 3 blocks 6, 3x2, 3x2 layers then 2 layers decoder, 768-hidden, 12-heads, 130M parameters | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``funnel-transformer/medium-base`` | | 12 layers: 3 blocks 6, 3x2, 3x2 layers(no decoder), 768-hidden, 12-heads, 115M parameters | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``funnel-transformer/intermediate`` | | 20 layers: 3 blocks of 6 layers then 2 layers decoder, 768-hidden, 12-heads, 177M parameters | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``funnel-transformer/intermediate-base`` | | 18 layers: 3 blocks of 6 layers (no decoder), 768-hidden, 12-heads, 161M parameters | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``funnel-transformer/large`` | | 26 layers: 3 blocks of 8 layers then 2 layers decoder, 1024-hidden, 12-heads, 386M parameters | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``funnel-transformer/large-base`` | | 24 layers: 3 blocks of 8 layers (no decoder), 1024-hidden, 12-heads, 358M parameters | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``funnel-transformer/xlarge`` | | 32 layers: 3 blocks of 10 layers then 2 layers decoder, 1024-hidden, 12-heads, 468M parameters | +| | | | +| | | (see `details `__) | +| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| | ``funnel-transformer/xlarge-base`` | | 30 layers: 3 blocks of 10 layers (no decoder), 1024-hidden, 12-heads, 440M parameters | +| | | | +| | | (see `details `__) | ++--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+ diff --git a/src/transformers/__init__.py b/src/transformers/__init__.py index 624246b338..9d59714d83 100755 --- a/src/transformers/__init__.py +++ b/src/transformers/__init__.py @@ -29,6 +29,7 @@ from .configuration_dpr import DPR_PRETRAINED_CONFIG_ARCHIVE_MAP, DPRConfig from .configuration_electra import ELECTRA_PRETRAINED_CONFIG_ARCHIVE_MAP, ElectraConfig from .configuration_encoder_decoder import EncoderDecoderConfig from .configuration_flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig +from .configuration_funnel import FUNNEL_PRETRAINED_CONFIG_ARCHIVE_MAP, FunnelConfig from .configuration_gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config from .configuration_longformer import LONGFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP, LongformerConfig from .configuration_lxmert import LXMERT_PRETRAINED_CONFIG_ARCHIVE_MAP, LxmertConfig @@ -155,6 +156,7 @@ from .tokenization_dpr import ( ) from .tokenization_electra import ElectraTokenizer, ElectraTokenizerFast from .tokenization_flaubert import FlaubertTokenizer +from .tokenization_funnel import FunnelTokenizer, FunnelTokenizerFast from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast from .tokenization_longformer import LongformerTokenizer, LongformerTokenizerFast from .tokenization_lxmert import LxmertTokenizer, LxmertTokenizerFast @@ -327,6 +329,18 @@ if is_torch_available(): FlaubertModel, FlaubertWithLMHeadModel, ) + from .modeling_funnel import ( + FUNNEL_PRETRAINED_MODEL_ARCHIVE_LIST, + FunnelBaseModel, + FunnelForMaskedLM, + FunnelForMultipleChoice, + FunnelForPreTraining, + FunnelForQuestionAnswering, + FunnelForSequenceClassification, + FunnelForTokenClassification, + FunnelModel, + load_tf_weights_in_funnel, + ) from .modeling_gpt2 import ( GPT2_PRETRAINED_MODEL_ARCHIVE_LIST, GPT2DoubleHeadsModel, diff --git a/src/transformers/commands/convert.py b/src/transformers/commands/convert.py index f9e7822409..4238f078d8 100644 --- a/src/transformers/commands/convert.py +++ b/src/transformers/commands/convert.py @@ -15,6 +15,12 @@ def convert_command_factory(args: Namespace): ) +IMPORT_ERROR_MESSAGE = """transformers can only be used from the commandline to convert TensorFlow models in PyTorch, +In that case, it requires TensorFlow to be installed. Please see +https://www.tensorflow.org/install/ for installation instructions. +""" + + class ConvertCommand(BaseTransformersCLICommand): @staticmethod def register_subcommand(parser: ArgumentParser): @@ -69,12 +75,7 @@ class ConvertCommand(BaseTransformersCLICommand): convert_tf_checkpoint_to_pytorch, ) except ImportError: - msg = ( - "transformers can only be used from the commandline to convert TensorFlow models in PyTorch, " - "In that case, it requires TensorFlow to be installed. Please see " - "https://www.tensorflow.org/install/ for installation instructions." - ) - raise ImportError(msg) + raise ImportError(IMPORT_ERROR_MESSAGE) convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output) elif self._model_type == "bert": @@ -83,12 +84,16 @@ class ConvertCommand(BaseTransformersCLICommand): convert_tf_checkpoint_to_pytorch, ) except ImportError: - msg = ( - "transformers can only be used from the commandline to convert TensorFlow models in PyTorch, " - "In that case, it requires TensorFlow to be installed. Please see " - "https://www.tensorflow.org/install/ for installation instructions." + raise ImportError(IMPORT_ERROR_MESSAGE) + + convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output) + elif self._model_type == "funnel": + try: + from transformers.convert_funnel_original_tf_checkpoint_to_pytorch import ( + convert_tf_checkpoint_to_pytorch, ) - raise ImportError(msg) + except ImportError: + raise ImportError(IMPORT_ERROR_MESSAGE) convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output) elif self._model_type == "gpt": @@ -103,12 +108,7 @@ class ConvertCommand(BaseTransformersCLICommand): convert_transfo_xl_checkpoint_to_pytorch, ) except ImportError: - msg = ( - "transformers can only be used from the commandline to convert TensorFlow models in PyTorch, " - "In that case, it requires TensorFlow to be installed. Please see " - "https://www.tensorflow.org/install/ for installation instructions." - ) - raise ImportError(msg) + raise ImportError(IMPORT_ERROR_MESSAGE) if "ckpt" in self._tf_checkpoint.lower(): TF_CHECKPOINT = self._tf_checkpoint @@ -125,12 +125,7 @@ class ConvertCommand(BaseTransformersCLICommand): convert_gpt2_checkpoint_to_pytorch, ) except ImportError: - msg = ( - "transformers can only be used from the commandline to convert TensorFlow models in PyTorch, " - "In that case, it requires TensorFlow to be installed. Please see " - "https://www.tensorflow.org/install/ for installation instructions." - ) - raise ImportError(msg) + raise ImportError(IMPORT_ERROR_MESSAGE) convert_gpt2_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output) elif self._model_type == "xlnet": @@ -139,12 +134,7 @@ class ConvertCommand(BaseTransformersCLICommand): convert_xlnet_checkpoint_to_pytorch, ) except ImportError: - msg = ( - "transformers can only be used from the commandline to convert TensorFlow models in PyTorch, " - "In that case, it requires TensorFlow to be installed. Please see " - "https://www.tensorflow.org/install/ for installation instructions." - ) - raise ImportError(msg) + raise ImportError(IMPORT_ERROR_MESSAGE) convert_xlnet_checkpoint_to_pytorch( self._tf_checkpoint, self._config, self._pytorch_dump_output, self._finetuning_task_name diff --git a/src/transformers/configuration_auto.py b/src/transformers/configuration_auto.py index 6dc1e5dd0d..c1191fe24d 100644 --- a/src/transformers/configuration_auto.py +++ b/src/transformers/configuration_auto.py @@ -26,6 +26,7 @@ from .configuration_distilbert import DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, from .configuration_electra import ELECTRA_PRETRAINED_CONFIG_ARCHIVE_MAP, ElectraConfig from .configuration_encoder_decoder import EncoderDecoderConfig from .configuration_flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig +from .configuration_funnel import FUNNEL_PRETRAINED_CONFIG_ARCHIVE_MAP, FunnelConfig from .configuration_gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config from .configuration_longformer import LONGFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP, LongformerConfig from .configuration_lxmert import LXMERT_PRETRAINED_CONFIG_ARCHIVE_MAP, LxmertConfig @@ -67,6 +68,7 @@ ALL_PRETRAINED_CONFIG_ARCHIVE_MAP = dict( ELECTRA_PRETRAINED_CONFIG_ARCHIVE_MAP, LONGFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP, RETRIBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, + FUNNEL_PRETRAINED_CONFIG_ARCHIVE_MAP, LXMERT_PRETRAINED_CONFIG_ARCHIVE_MAP, ] for key, value, in pretrained_map.items() @@ -168,6 +170,10 @@ CONFIG_MAPPING = OrderedDict( "encoder-decoder", EncoderDecoderConfig, ), + ( + "funnel", + FunnelConfig, + ), ( "lxmert", LxmertConfig, @@ -230,6 +236,7 @@ class AutoConfig: - `ctrl` : :class:`~transformers.CTRLConfig` (CTRL model) - `flaubert` : :class:`~transformers.FlaubertConfig` (Flaubert model) - `electra` : :class:`~transformers.ElectraConfig` (ELECTRA model) + - `funnel`: :class:`~transformers.FunnelConfig` (Funnel Transformer model) Args: pretrained_model_name_or_path (:obj:`string`): diff --git a/src/transformers/configuration_funnel.py b/src/transformers/configuration_funnel.py new file mode 100644 index 0000000000..857371e549 --- /dev/null +++ b/src/transformers/configuration_funnel.py @@ -0,0 +1,183 @@ +# coding=utf-8 +# Copyright 2020, Hugging Face +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" Funnel Transformer model configuration """ + +from .configuration_utils import PretrainedConfig +from .utils import logging + + +logger = logging.get_logger(__name__) + +FUNNEL_PRETRAINED_CONFIG_ARCHIVE_MAP = { + "funnel-transformer/small": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/small/config.json", + "funnel-transformer/small-base": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/small-base/config.json", + "funnel-transformer/medium": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/medium/config.json", + "funnel-transformer/medium-base": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/medium-base/config.json", + "funnel-transformer/intermediate": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/intermediate/config.json", + "funnel-transformer/intermediate-base": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/intermediate-base/config.json", + "funnel-transformer/large": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/large/config.json", + "funnel-transformer/large-base": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/large-base/config.json", + "funnel-transformer/xlarge": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/xlarge/config.json", + "funnel-transformer/xlarge-base": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/xlarge-base/config.json", +} + + +class FunnelConfig(PretrainedConfig): + r""" + This is the configuration class to store the configuration of a :class:`~transformers.FunnelModel`. + It is used to instantiate an Funnel Transformer model according to the specified arguments, defining the model + architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of + the Funnel Transformer `funnel-transformer/small `__ architecture. + + Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used + to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig` + for more information. + + + Args: + vocab_size (:obj:`int`, `optional`, defaults to 30522): + Vocabulary size of the Funnel transformer. Defines the different tokens that + can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.FunnelModel`. + block_sizes (:obj:`List[int]`, `optional`, defaults to :obj:`[4, 4, 4]`): + The sizes of the blocks used in the model. + block_repeats (:obj:`List[int]`, `optional`): + If passed along, each layer of each block is repeated the number of times indicated. + num_decoder_layers (:obj:`int`, `optional`, defaults to 2): + The number of layers in the decoder (when not using the base model). + d_model (:obj:`int`, `optional`, defaults to 768): + Dimensionality of the model's hidden states. + n_head (:obj:`int`, `optional`, defaults to 12): + Number of attention heads for each attention layer in the Transformer encoder. + d_head (:obj:`int`, `optional`, defaults to 64): + Dimensionality of the model's heads. + d_inner (:obj:`int`, `optional`, defaults to 3072): + Inner dimension in the feed-forward blocks. + hidden_act (:obj:`str` or :obj:`callable`, `optional`, defaults to :obj:`"gelu_new"`): + The non-linear activation function (function or string) in the encoder and pooler. + If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported. + hidden_dropout (:obj:`float`, `optional`, defaults to 0.1): + The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler. + attention_dropout (:obj:`float`, `optional`, defaults to 0.1): + The dropout probability for the attention probabilities. + activation_dropout (:obj:`float`, `optional`, defaults to 0.0): + The dropout probability used between the two layers of the feed-forward blocks. + max_position_embeddings (:obj:`int`, `optional`, defaults to 512): + The maximum sequence length that this model might ever be used with. + Typically set this to something large just in case (e.g., 512 or 1024 or 2048). + type_vocab_size (:obj:`int`, `optional`, defaults to 3): + The vocabulary size of the `token_type_ids` passed into :class:`~transformers.FunnelModel`. + initializer_range (:obj:`float`, `optional`, defaults to 0.1): + The standard deviation of the `uniform initializer` for initializing all weight matrices in attention + layers. + initializer_std (:obj:`float`, `optional`): + The standard deviation of the `normal initializer` for initializing the embedding matrix and the weight of + linear layers. Will default to 1 for the embedding matrix and the value given by Xavier initialization for + linear layers. + layer_norm_eps (:obj:`float`, `optional`, defaults to 1e-9): + The epsilon used by the layer normalization layers. + pooling_type (:obj:`str`, `optional`, defaults to :obj:`"mean"`): + Possible values are ``"mean"`` or ``"max"``. The way pooling is performed at the beginning of each + block. + attention_type (:obj:`str`, `optional`, defaults to :obj:`"relative_shift"`): + Possible values are ``"relative_shift"`` or ``"factorized"``. The former is faster on CPU/GPU while + the latter is faster on TPU. + separate_cls (:obj:`bool`, `optional`, defaults to :obj:`True`): + Whether or not to separate the cls token when applying pooling. + truncate_seq (:obj:`bool`, `optional`, defaults to :obj:`False`): + When using ``separate_cls``, whether or not to truncate the last token when pooling, to avoid getting + a sequence length that is not a multiple of 2. + pool_q_only (:obj:`bool`, `optional`, defaults to :obj:`False`): + Whether or not to apply the pooling only to the query or to query, key and values for the attention + layers. + """ + model_type = "funnel" + + def __init__( + self, + vocab_size=30522, + block_sizes=[4, 4, 4], + block_repeats=None, + num_decoder_layers=2, + d_model=768, + n_head=12, + d_head=64, + d_inner=3072, + hidden_act="gelu_new", + hidden_dropout=0.1, + attention_dropout=0.1, + activation_dropout=0.0, + max_position_embeddings=512, + type_vocab_size=3, + initializer_range=0.1, + initializer_std=None, + layer_norm_eps=1e-9, + pooling_type="mean", + attention_type="relative_shift", + separate_cls=True, + truncate_seq=True, + pool_q_only=True, + **kwargs + ): + super().__init__(**kwargs) + + self.vocab_size = vocab_size + self.block_sizes = block_sizes + self.block_repeats = [1] * len(block_sizes) if block_repeats is None else block_repeats + assert len(block_sizes) == len( + self.block_repeats + ), "`block_sizes` and `block_repeats` should have the same length." + self.num_decoder_layers = num_decoder_layers + self.d_model = d_model + self.n_head = n_head + self.d_head = d_head + self.d_inner = d_inner + self.hidden_act = hidden_act + self.hidden_dropout = hidden_dropout + self.attention_dropout = attention_dropout + self.activation_dropout = activation_dropout + self.max_position_embeddings = max_position_embeddings + self.type_vocab_size = type_vocab_size + self.initializer_range = initializer_range + self.initializer_std = initializer_std + self.layer_norm_eps = layer_norm_eps + assert pooling_type in [ + "mean", + "max", + ], f"Got {pooling_type} for `pooling_type` but only 'mean' and 'max' are supported." + self.pooling_type = pooling_type + assert attention_type in [ + "relative_shift", + "factorized", + ], f"Got {attention_type} for `attention_type` but only 'relative_shift' and 'factorized' are supported." + self.attention_type = attention_type + self.separate_cls = separate_cls + self.truncate_seq = truncate_seq + self.pool_q_only = pool_q_only + + @property + def hidden_size(self): + return self.d_model + + @property + def num_attention_heads(self): + return self.n_head + + @property + def num_hidden_layers(self): + return sum(self.block_sizes) + + @property + def num_blocks(self): + return len(self.block_sizes) diff --git a/src/transformers/convert_funnel_original_tf_checkpoint_to_pytorch.py b/src/transformers/convert_funnel_original_tf_checkpoint_to_pytorch.py new file mode 100755 index 0000000000..5d93fc24db --- /dev/null +++ b/src/transformers/convert_funnel_original_tf_checkpoint_to_pytorch.py @@ -0,0 +1,61 @@ +# coding=utf-8 +# Copyright 2020 The HuggingFace Inc. team. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Convert Funnel checkpoint.""" + + +import argparse +import logging + +import torch + +from transformers import FunnelConfig, FunnelForPreTraining, load_tf_weights_in_funnel + + +logging.basicConfig(level=logging.INFO) + + +def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, config_file, pytorch_dump_path): + # Initialise PyTorch model + config = FunnelConfig.from_json_file(config_file) + print("Building PyTorch model from configuration: {}".format(str(config))) + model = FunnelForPreTraining(config) + + # Load weights from tf checkpoint + load_tf_weights_in_funnel(model, config, tf_checkpoint_path) + + # Save pytorch-model + print("Save PyTorch model to {}".format(pytorch_dump_path)) + torch.save(model.state_dict(), pytorch_dump_path) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + # Required parameters + parser.add_argument( + "--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path." + ) + parser.add_argument( + "--config_file", + default=None, + type=str, + required=True, + help="The config json file corresponding to the pre-trained model. \n" + "This specifies the model architecture.", + ) + parser.add_argument( + "--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model." + ) + args = parser.parse_args() + convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.config_file, args.pytorch_dump_path) diff --git a/src/transformers/modeling_auto.py b/src/transformers/modeling_auto.py index 7f85ce16dd..56ed629e9a 100644 --- a/src/transformers/modeling_auto.py +++ b/src/transformers/modeling_auto.py @@ -29,6 +29,7 @@ from .configuration_auto import ( ElectraConfig, EncoderDecoderConfig, FlaubertConfig, + FunnelConfig, GPT2Config, LongformerConfig, LxmertConfig, @@ -108,6 +109,14 @@ from .modeling_flaubert import ( FlaubertModel, FlaubertWithLMHeadModel, ) +from .modeling_funnel import ( + FunnelForMaskedLM, + FunnelForMultipleChoice, + FunnelForQuestionAnswering, + FunnelForSequenceClassification, + FunnelForTokenClassification, + FunnelModel, +) from .modeling_gpt2 import GPT2LMHeadModel, GPT2Model from .modeling_longformer import ( LongformerForMaskedLM, @@ -202,6 +211,7 @@ MODEL_MAPPING = OrderedDict( (CTRLConfig, CTRLModel), (ElectraConfig, ElectraModel), (ReformerConfig, ReformerModel), + (FunnelConfig, FunnelModel), (LxmertConfig, LxmertModel), ] ) @@ -254,6 +264,7 @@ MODEL_WITH_LM_HEAD_MAPPING = OrderedDict( (ElectraConfig, ElectraForMaskedLM), (EncoderDecoderConfig, EncoderDecoderModel), (ReformerConfig, ReformerModelWithLMHead), + (FunnelConfig, FunnelForMaskedLM), ] ) @@ -291,6 +302,7 @@ MODEL_FOR_MASKED_LM_MAPPING = OrderedDict( (XLMConfig, XLMWithLMHeadModel), (ElectraConfig, ElectraForMaskedLM), (ReformerConfig, ReformerForMaskedLM), + (FunnelConfig, FunnelForMaskedLM), ] ) @@ -320,6 +332,7 @@ MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING = OrderedDict( (FlaubertConfig, FlaubertForSequenceClassification), (XLMConfig, XLMForSequenceClassification), (ElectraConfig, ElectraForSequenceClassification), + (FunnelConfig, FunnelForSequenceClassification), ] ) @@ -339,6 +352,7 @@ MODEL_FOR_QUESTION_ANSWERING_MAPPING = OrderedDict( (XLMConfig, XLMForQuestionAnsweringSimple), (ElectraConfig, ElectraForQuestionAnswering), (ReformerConfig, ReformerForQuestionAnswering), + (FunnelConfig, FunnelForQuestionAnswering), ] ) @@ -357,6 +371,7 @@ MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING = OrderedDict( (AlbertConfig, AlbertForTokenClassification), (ElectraConfig, ElectraForTokenClassification), (FlaubertConfig, FlaubertForTokenClassification), + (FunnelConfig, FunnelForTokenClassification), ] ) @@ -374,6 +389,7 @@ MODEL_FOR_MULTIPLE_CHOICE_MAPPING = OrderedDict( (AlbertConfig, AlbertForMultipleChoice), (XLMConfig, XLMForMultipleChoice), (FlaubertConfig, FlaubertForMultipleChoice), + (FunnelConfig, FunnelForMultipleChoice), ] ) @@ -421,6 +437,7 @@ class AutoModel: - isInstance of `xlm` configuration class: :class:`~transformers.XLMModel` (XLM model) - isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertModel` (Flaubert model) - isInstance of `electra` configuration class: :class:`~transformers.ElectraModel` (Electra model) + - isInstance of `funnel` configuration class: :class:`~transformers.FunnelModel` (Funnel Transformer model) Examples:: @@ -462,6 +479,7 @@ class AutoModel: - `ctrl`: :class:`~transformers.CTRLModel` (Salesforce CTRL model) - `flaubert`: :class:`~transformers.FlaubertModel` (Flaubert model) - `electra`: :class:`~transformers.ElectraModel` (Electra model) + - `funnel`: :class:`~transformers.FunnelModel` (Funnel Transformer model) The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated) To train the model, you should first set it back in training mode with `model.train()` @@ -729,6 +747,7 @@ class AutoModelWithLMHead: - isInstance of `xlm` configuration class: :class:`~transformers.XLMWithLMHeadModel` (XLM model) - isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertWithLMHeadModel` (Flaubert model) - isInstance of `electra` configuration class: :class:`~transformers.ElectraForMaskedLM` (Electra model) + - isInstance of `funnel` configuration class: :class:`~transformers.FunnelForMaskedLM` (Funnel Transformer model) Examples:: @@ -774,6 +793,7 @@ class AutoModelWithLMHead: - `ctrl`: :class:`~transformers.CTRLLMHeadModel` (Salesforce CTRL model) - `flaubert`: :class:`~transformers.FlaubertWithLMHeadModel` (Flaubert model) - `electra`: :class:`~transformers.ElectraForMaskedLM` (Electra model) + - `funnel`: :class:`~transformers.FunnelForMaskedLM` (Funnel Transformer model) The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated) To train the model, you should first set it back in training mode with `model.train()` @@ -1024,6 +1044,7 @@ class AutoModelForMaskedLM: - isInstance of `electra` configuration class: :class:`~transformers.ElectraForMaskedLM` (Electra model) - isInstance of `camembert` configuration class: :class:`~transformers.CamembertForMaskedLM` (Camembert model) - isInstance of `albert` configuration class: :class:`~transformers.AlbertForMaskedLM` (Albert model) + - isInstance of `funnel` configuration class: :class:`~transformers.FunnelForMaskedLM` (Funnel Transformer model) Examples:: @@ -1060,6 +1081,7 @@ class AutoModelForMaskedLM: - `flaubert`: :class:`~transformers.FlaubertWithLMHeadModel` (Flaubert model) - `electra`: :class:`~transformers.ElectraForMaskedLM` (Electra model) - `bert`: :class:`~transformers.BertLMHeadModel` (Bert model) + - `funnel`: :class:`~transformers.FunnelForMaskedLM` (Funnel Transformer model) The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated) To train the model, you should first set it back in training mode with `model.train()` @@ -1304,7 +1326,7 @@ class AutoModelForSequenceClassification: - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetForSequenceClassification` (XLNet model) - isInstance of `xlm` configuration class: :class:`~transformers.XLMForSequenceClassification` (XLM model) - isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertForSequenceClassification` (Flaubert model) - + - isInstance of `funnel` configuration class: :class:`~transformers.FunnelModelForSequenceClassification` (Funnel Transformer model) Examples:: @@ -1340,6 +1362,7 @@ class AutoModelForSequenceClassification: - `bert`: :class:`~transformers.BertForSequenceClassification` (Bert model) - `xlnet`: :class:`~transformers.XLNetForSequenceClassification` (XLNet model) - `flaubert`: :class:`~transformers.FlaubertForSequenceClassification` (Flaubert model) + - `funnel`: :class:`~transformers.FunnelForSequenceClassification` (Funnel Transformer model) The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated) To train the model, you should first set it back in training mode with `model.train()` @@ -1454,6 +1477,7 @@ class AutoModelForQuestionAnswering: - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetForQuestionAnswering` (XLNet model) - isInstance of `xlm` configuration class: :class:`~transformers.XLMForQuestionAnswering` (XLM model) - isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertForQuestionAnswering` (XLM model) + - isInstance of `funnel` configuration class: :class:`~transformers.FunnelForQuestionAnswering` (Funnel Transformer model) Examples:: @@ -1488,6 +1512,7 @@ class AutoModelForQuestionAnswering: - `xlnet`: :class:`~transformers.XLNetForQuestionAnswering` (XLNet model) - `xlm`: :class:`~transformers.XLMForQuestionAnswering` (XLM model) - `flaubert`: :class:`~transformers.FlaubertForQuestionAnswering` (XLM model) + - `funnel`: :class:`~transformers.FunnelForQuestionAnswering` (Funnel Transformer model) The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated) To train the model, you should first set it back in training mode with `model.train()` @@ -1604,6 +1629,7 @@ class AutoModelForTokenClassification: - isInstance of `camembert` configuration class: :class:`~transformers.CamembertModelForTokenClassification` (Camembert model) - isInstance of `roberta` configuration class: :class:`~transformers.RobertaModelForTokenClassification` (Roberta model) - isInstance of `electra` configuration class: :class:`~transformers.ElectraForTokenClassification` (Electra model) + - isInstance of `funnel` configuration class: :class:`~transformers.FunnelForTokenClassification` (Funnel Transformer model) Examples:: @@ -1641,6 +1667,7 @@ class AutoModelForTokenClassification: - `flaubert`: :class:`~transformers.FlaubertForTokenClassification` (Flaubert model) - `roberta`: :class:`~transformers.RobertaForTokenClassification` (Roberta model) - `electra`: :class:`~transformers.ElectraForTokenClassification` (Electra model) + - `funnel`: :class:`~transformers.FunnelForTokenClassification` (Funnel Transformer model) The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated) To train the model, you should first set it back in training mode with `model.train()` diff --git a/src/transformers/modeling_funnel.py b/src/transformers/modeling_funnel.py new file mode 100644 index 0000000000..bb0d34e398 --- /dev/null +++ b/src/transformers/modeling_funnel.py @@ -0,0 +1,1544 @@ +# coding=utf-8 +# Copyright 2020-present Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" PyTorch Funnel Transformer model. """ + +import os +from dataclasses import dataclass +from typing import Optional, Tuple + +import numpy as np +import torch +from torch import nn +from torch.nn import CrossEntropyLoss, MSELoss +from torch.nn import functional as F + +from .activations import ACT2FN +from .configuration_funnel import FunnelConfig +from .file_utils import ( + ModelOutput, + add_code_sample_docstrings, + add_start_docstrings, + add_start_docstrings_to_callable, + replace_return_docstrings, +) +from .modeling_outputs import ( + BaseModelOutput, + MaskedLMOutput, + MultipleChoiceModelOutput, + QuestionAnsweringModelOutput, + SequenceClassifierOutput, + TokenClassifierOutput, +) +from .modeling_utils import PreTrainedModel +from .utils import logging + + +logger = logging.get_logger(__name__) + +_CONFIG_FOR_DOC = "FunnelConfig" +_TOKENIZER_FOR_DOC = "FunnelTokenizer" + +FUNNEL_PRETRAINED_MODEL_ARCHIVE_LIST = [ + "funnel-transformer/small", # B4-4-4H768 + "funnel-transformer/small-base", # B4-4-4H768, no decoder + "funnel-transformer/medium", # B6-3x2-3x2H768 + "funnel-transformer/medium-base", # B6-3x2-3x2H768, no decoder + "funnel-transformer/intermediate", # B6-6-6H768 + "funnel-transformer/intermediate-base", # B6-6-6H768, no decoder + "funnel-transformer/large", # B8-8-8H1024 + "funnel-transformer/large-base", # B8-8-8H1024, no decoder + "funnel-transformer/xlarge-base", # B10-10-10H1024 + "funnel-transformer/xlarge", # B10-10-10H1024, no decoder +] + + +FunnelLayerNorm = nn.LayerNorm + +INF = 1e6 + + +def load_tf_weights_in_funnel(model, config, tf_checkpoint_path): + """Load tf checkpoints in a pytorch model.""" + try: + import re + + import numpy as np + import tensorflow as tf + except ImportError: + logger.error( + "Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see " + "https://www.tensorflow.org/install/ for installation instructions." + ) + raise + tf_path = os.path.abspath(tf_checkpoint_path) + logger.info("Converting TensorFlow checkpoint from {}".format(tf_path)) + # Load weights from TF model + init_vars = tf.train.list_variables(tf_path) + names = [] + arrays = [] + for name, shape in init_vars: + logger.info("Loading TF weight {} with shape {}".format(name, shape)) + array = tf.train.load_variable(tf_path, name) + names.append(name) + arrays.append(array) + + _layer_map = { + "k": "k_head", + "q": "q_head", + "v": "v_head", + "o": "post_proj", + "layer_1": "linear_1", + "layer_2": "linear_2", + "rel_attn": "attention", + "ff": "ffn", + "kernel": "weight", + "gamma": "weight", + "beta": "bias", + "lookup_table": "weight", + "word_embedding": "word_embeddings", + "input": "embeddings", + } + + for name, array in zip(names, arrays): + name = name.split("/") + # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v + # which are not required for using pretrained model + if any( + n in ["adam_v", "adam_m", "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"] + for n in name + ): + logger.info("Skipping {}".format("/".join(name))) + continue + if name[0] == "generator": + continue + pointer = model + skipped = False + for m_name in name[1:]: + if not isinstance(pointer, FunnelPositionwiseFFN) and re.fullmatch(r"layer_\d+", m_name): + layer_index = int(re.search("layer_(\d+)", m_name).groups()[0]) + if layer_index < config.num_hidden_layers: + block_idx = 0 + while layer_index >= config.block_sizes[block_idx]: + layer_index -= config.block_sizes[block_idx] + block_idx += 1 + pointer = pointer.blocks[block_idx][layer_index] + else: + layer_index -= config.num_hidden_layers + pointer = pointer.layers[layer_index] + elif m_name == "r" and isinstance(pointer, FunnelRelMultiheadAttention): + pointer = pointer.r_kernel + break + elif m_name in _layer_map: + pointer = getattr(pointer, _layer_map[m_name]) + else: + try: + pointer = getattr(pointer, m_name) + except AttributeError: + print("Skipping {}".format("/".join(name)), array.shape) + skipped = True + break + if not skipped: + if len(pointer.shape) != len(array.shape): + array = array.reshape(pointer.shape) + if m_name == "kernel": + array = np.transpose(array) + pointer.data = torch.from_numpy(array) + + return model + + +class FunnelEmbeddings(nn.Module): + def __init__(self, config): + super().__init__() + self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id) + self.layer_norm = FunnelLayerNorm(config.d_model, eps=config.layer_norm_eps) + self.dropout = nn.Dropout(config.hidden_dropout) + + def forward(self, input_ids=None, inputs_embeds=None): + if inputs_embeds is None: + inputs_embeds = self.word_embeddings(input_ids) + embeddings = self.layer_norm(inputs_embeds) + embeddings = self.dropout(embeddings) + return embeddings + + +class FunnelAttentionStructure(nn.Module): + """ + Contains helpers for `FunnelRelMultiheadAttention `. + """ + + cls_token_type_id: int = 2 + + def __init__(self, config): + super().__init__() + self.config = config + self.sin_dropout = nn.Dropout(config.hidden_dropout) + self.cos_dropout = nn.Dropout(config.hidden_dropout) + # Track where we are at in terms of pooling from the original input, e.g., by how much the sequence length was + # dividide. + self.pooling_mult = None + + def init_attention_inputs(self, input_embeds, attention_mask=None, token_type_ids=None): + """ Returns the attention inputs associated to the inputs of the model. """ + # input_embeds has shape batch_size x seq_len x d_model + # attention_mask and token_type_ids have shape batch_size x seq_len + self.pooling_mult = 1 + self.seq_len = seq_len = input_embeds.size(1) + position_embeds = self.get_position_embeds(seq_len, input_embeds.dtype, input_embeds.device) + token_type_mat = self.token_type_ids_to_mat(token_type_ids) if token_type_ids is not None else None + cls_mask = ( + F.pad(input_embeds.new_ones([seq_len - 1, seq_len - 1]), (1, 0, 1, 0)) + if self.config.separate_cls + else None + ) + return (position_embeds, token_type_mat, attention_mask, cls_mask) + + def token_type_ids_to_mat(self, token_type_ids): + """Convert `token_type_ids` to `token_type_mat`.""" + token_type_mat = token_type_ids[:, :, None] == token_type_ids[:, None] + # Treat as in the same segment as both A & B + cls_ids = token_type_ids == self.cls_token_type_id + cls_mat = cls_ids[:, :, None] | cls_ids[:, None] + return cls_mat | token_type_mat + + def get_position_embeds(self, seq_len, dtype, device): + """ + Create and cache inputs related to relative position encoding. Those are very different depending on whether we + are using the factorized or the relative shift attention: + + For the factorized attention, it returns the matrices (phi, pi, psi, omega) used in the paper, appendix A.2.2, + final formula. + + For the relative shif attention, it returns all possible vectors R used in the paper, appendix A.2.1, final + formula. + + Paper link: https://arxiv.org/abs/2006.03236 + """ + d_model = self.config.d_model + if self.config.attention_type == "factorized": + # Notations from the paper, appending A.2.2, final formula. + # We need to create and return the matrics phi, psi, pi and omega. + pos_seq = torch.arange(0, seq_len, 1.0, dtype=dtype, device=device) + freq_seq = torch.arange(0, d_model // 2, 1.0, dtype=dtype, device=device) + inv_freq = 1 / (10000 ** (freq_seq / (d_model // 2))) + sinusoid = pos_seq[:, None] * inv_freq[None] + sin_embed = torch.sin(sinusoid) + sin_embed_d = self.sin_dropout(sin_embed) + cos_embed = torch.cos(sinusoid) + cos_embed_d = self.cos_dropout(cos_embed) + # This is different from the formula on the paper... + phi = torch.cat([sin_embed_d, sin_embed_d], dim=-1) + psi = torch.cat([cos_embed, sin_embed], dim=-1) + pi = torch.cat([cos_embed_d, cos_embed_d], dim=-1) + omega = torch.cat([-sin_embed, cos_embed], dim=-1) + return (phi, pi, psi, omega) + else: + # Notations from the paper, appending A.2.1, final formula. + # We need to create and return all the possible vectors R for all blocks and shifts. + freq_seq = torch.arange(0, d_model // 2, 1.0, dtype=dtype, device=device) + inv_freq = 1 / (10000 ** (freq_seq / (d_model // 2))) + # Maximum relative positions for the first input + rel_pos_id = torch.arange(-seq_len * 2, seq_len * 2, 1.0, dtype=dtype, device=device) + zero_offset = seq_len * 2 + sinusoid = rel_pos_id[:, None] * inv_freq[None] + sin_embed = self.sin_dropout(torch.sin(sinusoid)) + cos_embed = self.cos_dropout(torch.cos(sinusoid)) + pos_embed = torch.cat([sin_embed, cos_embed], dim=-1) + + pos = torch.arange(0, seq_len, dtype=dtype, device=device) + pooled_pos = pos + position_embeds_list = [] + for block_index in range(0, self.config.num_blocks): + # For each block with block_index > 0, we need two types position embeddings: + # - Attention(pooled-q, unpooled-kv) + # - Attention(pooled-q, pooled-kv) + # For block_index = 0 we only need the second one and leave the first one as None. + + # First type + if block_index == 0: + position_embeds_pooling = None + else: + pooled_pos = self.stride_pool_pos(pos, block_index) + + # construct rel_pos_id + stride = 2 ** (block_index - 1) + rel_pos = self.relative_pos(pos, stride, pooled_pos, shift=2) + rel_pos = rel_pos[:, None] + zero_offset + rel_pos = rel_pos.expand(rel_pos.size(0), d_model) + position_embeds_pooling = torch.gather(pos_embed, 0, rel_pos) + + # Second type + pos = pooled_pos + stride = 2 ** block_index + rel_pos = self.relative_pos(pos, stride) + + rel_pos = rel_pos[:, None] + zero_offset + rel_pos = rel_pos.expand(rel_pos.size(0), d_model) + position_embeds_no_pooling = torch.gather(pos_embed, 0, rel_pos) + + position_embeds_list.append([position_embeds_no_pooling, position_embeds_pooling]) + return position_embeds_list + + def stride_pool_pos(self, pos_id, block_index): + """ + Pool `pos_id` while keeping the cls token separate (if `config.separate_cls=True`). + """ + if self.config.separate_cls: + # Under separate , we treat the as the first token in + # the previous block of the 1st real block. Since the 1st real + # block always has position 1, the position of the previous block + # will be at `1 - 2 ** block_index`. + cls_pos = pos_id.new_tensor([-(2 ** block_index) + 1]) + pooled_pos_id = pos_id[1:-1] if self.config.truncate_seq else pos_id[1:] + return torch.cat([cls_pos, pooled_pos_id[::2]], 0) + else: + return pos_id[::2] + + def relative_pos(self, pos, stride, pooled_pos=None, shift=1): + """ + Build the relative positional vector between `pos` and `pooled_pos`. + """ + if pooled_pos is None: + pooled_pos = pos + + ref_point = pooled_pos[0] - pos[0] + num_remove = shift * len(pooled_pos) + max_dist = ref_point + num_remove * stride + min_dist = pooled_pos[0] - pos[-1] + + return torch.arange(max_dist, min_dist - 1, -stride, dtype=torch.long, device=pos.device) + + def stride_pool(self, tensor, axis): + """ + Perform pooling by stride slicing the tensor along the given axis. + """ + if tensor is None: + return None + + # Do the stride pool recursively if axis is a list or a tuple of ints. + if isinstance(axis, (list, tuple)): + for ax in axis: + tensor = self.stride_pool(tensor, ax) + return tensor + + # Do the stride pool recursively if tensor is a list or tuple of tensors. + if isinstance(tensor, (tuple, list)): + return type(tensor)(self.stride_pool(x, axis) for x in tensor) + + # Deal with negative axis + axis %= tensor.ndim + + axis_slice = ( + slice(None, -1, 2) if self.config.separate_cls and self.config.truncate_seq else slice(None, None, 2) + ) + enc_slice = [slice(None)] * axis + [axis_slice] + if self.config.separate_cls: + cls_slice = [slice(None)] * axis + [slice(None, 1)] + tensor = torch.cat([tensor[cls_slice], tensor], axis=axis) + return tensor[enc_slice] + + def pool_tensor(self, tensor, mode="mean", stride=2): + """Apply 1D pooling to a tensor of size [B x T (x H)].""" + if tensor is None: + return None + + # Do the pool recursively if tensor is a list or tuple of tensors. + if isinstance(tensor, (tuple, list)): + return type(tensor)(self.pool_tensor(tensor, mode=mode, stride=stride) for x in tensor) + + if self.config.separate_cls: + suffix = tensor[:, :-1] if self.config.truncate_seq else tensor + tensor = torch.cat([tensor[:, :1], suffix], dim=1) + + ndim = tensor.ndim + if ndim == 2: + tensor = tensor[:, None, :, None] + elif ndim == 3: + tensor = tensor[:, None, :, :] + # Stride is applied on the second-to-last dimension. + stride = (stride, 1) + + tensor = tensor.float() + if mode == "mean": + tensor = F.avg_pool2d(tensor, stride, stride=stride, ceil_mode=True) + elif mode == "max": + tensor = F.max_pool2d(tensor, stride, stride=stride, ceil_mode=True) + elif mode == "min": + tensor = -F.max_pool2d(-tensor, stride, stride=stride, ceil_mode=True) + else: + raise NotImplementedError("The supported modes are 'mean', 'max' and 'min'.") + + if ndim == 2: + return tensor[:, 0, :, 0] + elif ndim == 3: + return tensor[:, 0] + return tensor + + def pre_attention_pooling(self, output, attention_inputs): + """ Pool `output` and the proper parts of `attention_inputs` before the attention layer. """ + position_embeds, token_type_mat, attention_mask, cls_mask = attention_inputs + if self.config.pool_q_only: + if self.config.attention_type == "factorized": + position_embeds = self.stride_pool(position_embeds[:2], 0) + position_embeds[2:] + token_type_mat = self.stride_pool(token_type_mat, 1) + cls_mask = self.stride_pool(cls_mask, 0) + output = self.pool_tensor(output, mode=self.config.pooling_type) + else: + self.pooling_mult *= 2 + if self.config.attention_type == "factorized": + position_embeds = self.stride_pool(position_embeds, 0) + token_type_mat = self.stride_pool(token_type_mat, [1, 2]) + cls_mask = self.stride_pool(cls_mask, [1, 2]) + attention_mask = self.pool_tensor(attention_mask, mode="min") + output = self.pool_tensor(output, mode=self.config.pooling_type) + attention_inputs = (position_embeds, token_type_mat, attention_mask, cls_mask) + return output, attention_inputs + + def post_attention_pooling(self, attention_inputs): + """ Pool the proper parts of `attention_inputs` after the attention layer. """ + position_embeds, token_type_mat, attention_mask, cls_mask = attention_inputs + if self.config.pool_q_only: + self.pooling_mult *= 2 + if self.config.attention_type == "factorized": + position_embeds = position_embeds[:2] + self.stride_pool(position_embeds[2:], 0) + token_type_mat = self.stride_pool(token_type_mat, 2) + cls_mask = self.stride_pool(cls_mask, 1) + attention_mask = self.pool_tensor(attention_mask, mode="min") + attention_inputs = (position_embeds, token_type_mat, attention_mask, cls_mask) + return attention_inputs + + +def _relative_shift_gather(positional_attn, context_len, shift): + batch_size, n_head, seq_len, max_rel_len = positional_attn.shape + # max_rel_len = 2 * context_len + shift -1 is the numbers of possible relative positions i-j + + # What's next is the same as doing the following gather, which might be clearer code but less efficient. + # idxs = context_len + torch.arange(0, context_len).unsqueeze(0) - torch.arange(0, context_len).unsqueeze(1) + # # matrix of context_len + i-j + # return positional_attn.gather(3, idxs.expand([bs, n_head, context_len, context_len])) + + positional_attn = torch.reshape(positional_attn, [batch_size, n_head, max_rel_len, seq_len]) + positional_attn = positional_attn[:, :, shift:, :] + positional_attn = torch.reshape(positional_attn, [batch_size, n_head, seq_len, max_rel_len - shift]) + positional_attn = positional_attn[..., :context_len] + return positional_attn + + +class FunnelRelMultiheadAttention(nn.Module): + def __init__(self, config, block_index): + super().__init__() + self.config = config + self.block_index = block_index + d_model, n_head, d_head = config.d_model, config.n_head, config.d_head + + self.hidden_dropout = nn.Dropout(config.hidden_dropout) + self.attention_dropout = nn.Dropout(config.attention_dropout) + + self.q_head = nn.Linear(d_model, n_head * d_head, bias=False) + self.k_head = nn.Linear(d_model, n_head * d_head) + self.v_head = nn.Linear(d_model, n_head * d_head) + + self.r_w_bias = nn.Parameter(torch.zeros([n_head, d_head])) + self.r_r_bias = nn.Parameter(torch.zeros([n_head, d_head])) + self.r_kernel = nn.Parameter(torch.zeros([d_model, n_head, d_head])) + self.r_s_bias = nn.Parameter(torch.zeros([n_head, d_head])) + self.seg_embed = nn.Parameter(torch.zeros([2, n_head, d_head])) + + self.post_proj = nn.Linear(n_head * d_head, d_model) + self.layer_norm = FunnelLayerNorm(d_model, eps=config.layer_norm_eps) + self.scale = 1.0 / (d_head ** 0.5) + + def relative_positional_attention(self, position_embeds, q_head, context_len, cls_mask=None): + """ Relative attention score for the positional encodings """ + # q_head has shape batch_size x sea_len x n_head x d_head + if self.config.attention_type == "factorized": + # Notations from the paper, appending A.2.2, final formula (https://arxiv.org/abs/2006.03236) + # phi and pi have shape seq_len x d_model, psi and omega have shape context_len x d_model + phi, pi, psi, omega = position_embeds + # Shape n_head x d_head + u = self.r_r_bias * self.scale + # Shape d_model x n_head x d_head + w_r = self.r_kernel + + # Shape batch_size x sea_len x n_head x d_model + q_r_attention = torch.einsum("binh,dnh->bind", q_head + u, w_r) + q_r_attention_1 = q_r_attention * phi[:, None] + q_r_attention_2 = q_r_attention * pi[:, None] + + # Shape batch_size x n_head x seq_len x context_len + positional_attn = torch.einsum("bind,jd->bnij", q_r_attention_1, psi) + torch.einsum( + "bind,jd->bnij", q_r_attention_2, omega + ) + else: + shift = 2 if q_head.shape[1] != context_len else 1 + # Notations from the paper, appending A.2.1, final formula (https://arxiv.org/abs/2006.03236) + # Grab the proper positional encoding, shape max_rel_len x d_model + r = position_embeds[self.block_index][shift - 1] + # Shape n_head x d_head + v = self.r_r_bias * self.scale + # Shape d_model x n_head x d_head + w_r = self.r_kernel + + # Shape max_rel_len x n_head x d_model + r_head = torch.einsum("td,dnh->tnh", r, w_r) + # Shape batch_size x n_head x seq_len x max_rel_len + positional_attn = torch.einsum("binh,tnh->bnit", q_head + v, r_head) + # Shape batch_size x n_head x seq_len x context_len + positional_attn = _relative_shift_gather(positional_attn, context_len, shift) + + if cls_mask is not None: + positional_attn *= cls_mask + return positional_attn + + def relative_token_type_attention(self, token_type_mat, q_head, cls_mask=None): + """ Relative attention score for the token_type_ids """ + if token_type_mat is None: + return 0 + batch_size, seq_len, context_len = token_type_mat.shape + # q_head has shape batch_size x seq_len x n_head x d_head + # Shape n_head x d_head + r_s_bias = self.r_s_bias * self.scale + + # Shape batch_size x n_head x seq_len x 2 + token_type_bias = torch.einsum("bind,snd->bnis", q_head + r_s_bias, self.seg_embed) + # Shape batch_size x n_head x seq_len x context_len + token_type_mat = token_type_mat[:, None].expand([batch_size, q_head.shape[2], seq_len, context_len]) + # Shapes batch_size x n_head x seq_len + diff_token_type, same_token_type = torch.split(token_type_bias, 1, dim=-1) + # Shape batch_size x n_head x seq_len x context_len + token_type_attn = torch.where( + token_type_mat, same_token_type.expand(token_type_mat.shape), diff_token_type.expand(token_type_mat.shape) + ) + + if cls_mask is not None: + token_type_attn *= cls_mask + return token_type_attn + + def forward(self, query, key, value, attention_inputs, head_mask=None, output_attentions=False): + # q has shape batch_size x seq_len x d_model + # k and v have shapes batch_size x context_len x d_model + position_embeds, token_type_mat, attention_mask, cls_mask = attention_inputs + + batch_size, seq_len, _ = query.shape + context_len = key.shape[1] + n_head, d_head = self.config.n_head, self.config.d_head + + # Shape batch_size x seq_len x n_head x d_head + q_head = self.q_head(query).view(batch_size, seq_len, n_head, d_head) + # Shapes batch_size x context_len x n_head x d_head + k_head = self.k_head(key).view(batch_size, context_len, n_head, d_head) + v_head = self.v_head(value).view(batch_size, context_len, n_head, d_head) + + q_head = q_head * self.scale + # Shape n_head x d_head + r_w_bias = self.r_w_bias * self.scale + # Shapes batch_size x n_head x seq_len x context_len + content_score = torch.einsum("bind,bjnd->bnij", q_head + r_w_bias, k_head) + positional_attn = self.relative_positional_attention(position_embeds, q_head, context_len, cls_mask) + token_type_attn = self.relative_token_type_attention(token_type_mat, q_head, cls_mask) + + # merge attention scores + attn_score = content_score + positional_attn + token_type_attn + + # precision safe in case of mixed precision training + dtype = attn_score.dtype + attn_score = attn_score.float() + # perform masking + if attention_mask is not None: + attn_score = attn_score - INF * attention_mask[:, None, None].float() + # attention probability + attn_prob = torch.softmax(attn_score, dim=-1, dtype=dtype) + attn_prob = self.attention_dropout(attn_prob) + + # attention output, shape batch_size x seq_len x n_head x d_head + attn_vec = torch.einsum("bnij,bjnd->bind", attn_prob, v_head) + + # Shape shape batch_size x seq_len x d_model + attn_out = self.post_proj(attn_vec.reshape(batch_size, seq_len, n_head * d_head)) + attn_out = self.hidden_dropout(attn_out) + + output = self.layer_norm(query + attn_out) + return (output, attn_prob) if output_attentions else (output,) + + +class FunnelPositionwiseFFN(nn.Module): + def __init__(self, config): + super().__init__() + self.linear_1 = nn.Linear(config.d_model, config.d_inner) + self.activation_function = ACT2FN[config.hidden_act] + self.activation_dropout = nn.Dropout(config.activation_dropout) + self.linear_2 = nn.Linear(config.d_inner, config.d_model) + self.dropout = nn.Dropout(config.hidden_dropout) + self.layer_norm = FunnelLayerNorm(config.d_model, config.layer_norm_eps) + + def forward(self, hidden): + h = self.linear_1(hidden) + h = self.activation_function(h) + h = self.activation_dropout(h) + h = self.linear_2(h) + h = self.dropout(h) + return self.layer_norm(hidden + h) + + +class FunnelLayer(nn.Module): + def __init__(self, config, block_index): + super().__init__() + self.attention = FunnelRelMultiheadAttention(config, block_index) + self.ffn = FunnelPositionwiseFFN(config) + + def forward(self, q, k, v, attention_inputs, output_attentions=False): + attn = self.attention(q, k, v, attention_inputs, output_attentions=output_attentions) + output = self.ffn(attn[0]) + return (output, attn[1]) if output_attentions else (output,) + + +class FunnelEncoder(nn.Module): + def __init__(self, config): + super().__init__() + self.config = config + self.attention_structure = FunnelAttentionStructure(config) + self.blocks = nn.ModuleList( + [ + nn.ModuleList([FunnelLayer(config, block_index) for _ in range(block_size)]) + for block_index, block_size in enumerate(config.block_sizes) + ] + ) + + def forward( + self, + inputs_embeds, + attention_mask=None, + token_type_ids=None, + output_attentions=False, + output_hidden_states=False, + return_dict=False, + ): + # The pooling is not implemented on long tensors, so we convert this mask. + attention_mask = attention_mask.type_as(inputs_embeds) + attention_inputs = self.attention_structure.init_attention_inputs( + inputs_embeds, + attention_mask=attention_mask, + token_type_ids=token_type_ids, + ) + hidden = inputs_embeds + + all_hidden_states = (inputs_embeds,) if output_hidden_states else None + all_attentions = () if output_attentions else None + + for block_index, block in enumerate(self.blocks): + pooling_flag = hidden.size(1) > (2 if self.config.separate_cls else 1) + pooling_flag = pooling_flag and block_index > 0 + if pooling_flag: + pooled_hidden, attention_inputs = self.attention_structure.pre_attention_pooling( + hidden, attention_inputs + ) + for (layer_index, layer) in enumerate(block): + for repeat_index in range(self.config.block_repeats[block_index]): + do_pooling = (repeat_index == 0) and (layer_index == 0) and pooling_flag + if do_pooling: + query = pooled_hidden + key = value = hidden if self.config.pool_q_only else pooled_hidden + else: + query = key = value = hidden + layer_output = layer(query, key, value, attention_inputs, output_attentions=output_attentions) + hidden = layer_output[0] + if do_pooling: + attention_inputs = self.attention_structure.post_attention_pooling(attention_inputs) + + if output_attentions: + all_attentions = all_attentions + layer_output[1:] + if output_hidden_states: + all_hidden_states = all_hidden_states + (hidden,) + + if not return_dict: + return tuple(v for v in [hidden, all_hidden_states, all_attentions] if v is not None) + return BaseModelOutput(last_hidden_state=hidden, hidden_states=all_hidden_states, attentions=all_attentions) + + +def upsample(x, stride, target_len, separate_cls=True, truncate_seq=False): + """Upsample tensor `x` to match `target_len` by repeating the tokens `stride` time on the sequence length + dimension.""" + if stride == 1: + return x + if separate_cls: + cls = x[:, :1] + x = x[:, 1:] + output = torch.repeat_interleave(x, repeats=stride, dim=1) + if separate_cls: + if truncate_seq: + output = nn.functional.pad(output, (0, 0, 0, stride - 1, 0, 0)) + output = output[:, : target_len - 1] + output = torch.cat([cls, output], dim=1) + else: + output = output[:, :target_len] + return output + + +class FunnelDecoder(nn.Module): + def __init__(self, config): + super().__init__() + self.config = config + self.attention_structure = FunnelAttentionStructure(config) + self.layers = nn.ModuleList([FunnelLayer(config, 0) for _ in range(config.num_decoder_layers)]) + + def forward( + self, + final_hidden, + first_block_hidden, + attention_mask=None, + token_type_ids=None, + output_attentions=False, + output_hidden_states=False, + return_dict=False, + ): + upsampled_hidden = upsample( + final_hidden, + stride=2 ** (len(self.config.block_sizes) - 1), + target_len=first_block_hidden.shape[1], + separate_cls=self.config.separate_cls, + truncate_seq=self.config.truncate_seq, + ) + + hidden = upsampled_hidden + first_block_hidden + all_hidden_states = (hidden,) if output_hidden_states else None + all_attentions = () if output_attentions else None + + attention_inputs = self.attention_structure.init_attention_inputs( + hidden, + attention_mask=attention_mask, + token_type_ids=token_type_ids, + ) + + for layer in self.layers: + layer_output = layer(hidden, hidden, hidden, attention_inputs, output_attentions=output_attentions) + hidden = layer_output[0] + + if output_attentions: + all_attentions = all_attentions + layer_output[1:] + if output_hidden_states: + all_hidden_states = all_hidden_states + (hidden,) + + if not return_dict: + return tuple(v for v in [hidden, all_hidden_states, all_attentions] if v is not None) + return BaseModelOutput(last_hidden_state=hidden, hidden_states=all_hidden_states, attentions=all_attentions) + + +class FunnelDiscriminatorPredictions(nn.Module): + """Prediction module for the discriminator, made up of two dense layers.""" + + def __init__(self, config): + super().__init__() + self.config = config + self.dense = nn.Linear(config.d_model, config.d_model) + self.dense_prediction = nn.Linear(config.d_model, 1) + + def forward(self, discriminator_hidden_states): + hidden_states = self.dense(discriminator_hidden_states) + hidden_states = ACT2FN[self.config.hidden_act](hidden_states) + logits = self.dense_prediction(hidden_states).squeeze() + return logits + + +class FunnelPreTrainedModel(PreTrainedModel): + """An abstract class to handle weights initialization and + a simple interface for downloading and loading pretrained models. + """ + + config_class = FunnelConfig + load_tf_weights = load_tf_weights_in_funnel + base_model_prefix = "funnel" + + def _init_weights(self, module): + classname = module.__class__.__name__ + if classname.find("Linear") != -1: + if getattr(module, "weight", None) is not None: + if self.config.initializer_std is None: + fan_out, fan_in = module.weight.shape + std = np.sqrt(1.0 / float(fan_in + fan_out)) + else: + std = self.config.initializer_std + nn.init.normal_(module.weight, std=std) + if getattr(module, "bias", None) is not None: + nn.init.constant_(module.bias, 0.0) + elif classname == "FunnelRelMultiheadAttention": + nn.init.uniform_(module.r_w_bias, b=self.config.initializer_range) + nn.init.uniform_(module.r_r_bias, b=self.config.initializer_range) + nn.init.uniform_(module.r_kernel, b=self.config.initializer_range) + nn.init.uniform_(module.r_s_bias, b=self.config.initializer_range) + nn.init.uniform_(module.seg_embed, b=self.config.initializer_range) + elif classname == "FunnelEmbeddings": + std = 1.0 if self.config.initializer_std is None else self.config.initializer_std + nn.init.normal_(module.word_embeddings.weight, std=std) + + +class FunnelClassificationHead(nn.Module): + def __init__(self, config, n_labels): + super().__init__() + self.linear_hidden = nn.Linear(config.d_model, config.d_model) + self.dropout = nn.Dropout(config.hidden_dropout) + self.linear_out = nn.Linear(config.d_model, n_labels) + + def forward(self, hidden): + hidden = self.linear_hidden(hidden) + hidden = F.tanh(hidden) + hidden = self.dropout(hidden) + return self.linear_out(hidden) + + +@dataclass +class FunnelForPreTrainingOutput(ModelOutput): + """ + Output type of :class:`~transformers.FunnelForPreTrainingModel`. + + Args: + loss (`optional`, returned when ``labels`` is provided, ``torch.FloatTensor`` of shape :obj:`(1,)`): + Total loss of the ELECTRA-style objective. + logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`): + Prediction scores of the head (scores for each token before SoftMax). + hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``): + Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) + of shape :obj:`(batch_size, sequence_length, hidden_size)`. + + Hidden-states of the model at the output of each layer plus the initial embedding outputs. + attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``): + Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape + :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. + + Attentions weights after the attention softmax, used to compute the weighted average in the self-attention + heads. + """ + + loss: Optional[torch.FloatTensor] = None + logits: torch.FloatTensor = None + hidden_states: Optional[Tuple[torch.FloatTensor]] = None + attentions: Optional[Tuple[torch.FloatTensor]] = None + + +FUNNEL_START_DOCSTRING = r""" The Funnel Transformer model was proposed in + `Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing + `__ by Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le. + + This model is a PyTorch `torch.nn.Module `_ sub-class. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general + usage and behavior. + + Parameters: + config (:class:`~transformers.FunnelConfig`): Model configuration class with all the parameters of the model. + Initializing with a config file does not load the weights associated with the model, only the configuration. + Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. +""" + +FUNNEL_INPUTS_DOCSTRING = r""" + Inputs: + input_ids (:obj:`torch.LongTensor` of shape :obj:`{0}`): + Indices of input sequence tokens in the vocabulary. + + Indices can be obtained using :class:`transformers.FunnelTokenizer`. + See :func:`transformers.PreTrainedTokenizer.encode` and + :func:`transformers.PreTrainedTokenizer.__call__` for details. + + `What are input IDs? <../glossary.html#input-ids>`__ + attention_mask (:obj:`torch.FloatTensor` of shape :obj:`{0}`, `optional`, defaults to :obj:`None`): + Mask to avoid performing attention on padding token indices. + Mask values selected in ``[0, 1]``: + ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. + + `What are attention masks? <../glossary.html#attention-mask>`__ + token_type_ids (:obj:`torch.LongTensor` of shape :obj:`{0}`, `optional`, defaults to :obj:`None`): + Segment token indices to indicate first and second portions of the inputs. + Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` + corresponds to a `sentence B` token + + `What are token type IDs? <../glossary.html#token-type-ids>`_ + inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): + Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. + This is useful if you want more control over how to convert `input_ids` indices into associated vectors + than the model's internal embedding lookup matrix. + output_attentions (:obj:`bool`, `optional`, defaults to :obj:`None`): + If set to ``True``, the attentions tensors of all attention layers are returned. See ``attentions`` under returned tensors for more detail. + output_hidden_states (:obj:`bool`, `optional`, defaults to :obj:`None`): + If set to ``True``, the hidden states of all layers are returned. See ``hidden_states`` under returned tensors for more detail. + return_dict (:obj:`bool`, `optional`, defaults to :obj:`None`): + If set to ``True``, the model will return a :class:`~transformers.file_utils.ModelOutput` instead of a + plain tuple. +""" + + +@add_start_docstrings( + """ The base Funnel Transformer Model transformer outputting raw hidden-states without upsampling head (also called + decoder) or any task-specific head on top.""", + FUNNEL_START_DOCSTRING, +) +class FunnelBaseModel(FunnelPreTrainedModel): + def __init__(self, config): + super().__init__(config) + + self.embeddings = FunnelEmbeddings(config) + self.encoder = FunnelEncoder(config) + + self.init_weights() + + def get_input_embeddings(self): + return self.embeddings.word_embeddings + + def set_input_embeddings(self, new_embeddings): + self.embeddings.word_embeddings = new_embeddings + + @add_start_docstrings_to_callable(FUNNEL_INPUTS_DOCSTRING.format("(batch_size, sequence_length)")) + @add_code_sample_docstrings( + tokenizer_class=_TOKENIZER_FOR_DOC, + checkpoint="funnel-transformer/small-base", + output_type=BaseModelOutput, + config_class=_CONFIG_FOR_DOC, + ) + def forward( + self, + input_ids=None, + attention_mask=None, + token_type_ids=None, + position_ids=None, + head_mask=None, + inputs_embeds=None, + output_attentions=None, + output_hidden_states=None, + return_dict=None, + ): + output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions + output_hidden_states = ( + output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states + ) + return_dict = return_dict if return_dict is not None else self.config.use_return_dict + + if input_ids is not None and inputs_embeds is not None: + raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time") + elif input_ids is not None: + input_shape = input_ids.size() + elif inputs_embeds is not None: + input_shape = inputs_embeds.size()[:-1] + else: + raise ValueError("You have to specify either input_ids or inputs_embeds") + + device = input_ids.device if input_ids is not None else inputs_embeds.device + + if attention_mask is None: + attention_mask = torch.ones(input_shape, device=device) + if token_type_ids is None: + token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=device) + + # TODO: deal with head_mask + if inputs_embeds is None: + inputs_embeds = self.embeddings(input_ids) + + encoder_outputs = self.encoder( + inputs_embeds, + attention_mask=attention_mask, + token_type_ids=token_type_ids, + output_attentions=output_attentions, + output_hidden_states=output_hidden_states, + return_dict=return_dict, + ) + + return encoder_outputs + + +@add_start_docstrings( + "The bare base Funnel Transformer Model transformer outputting raw hidden-states without any specific head on top.", + FUNNEL_START_DOCSTRING, +) +class FunnelModel(FunnelPreTrainedModel): + def __init__(self, config): + super().__init__(config) + self.config = config + self.embeddings = FunnelEmbeddings(config) + self.encoder = FunnelEncoder(config) + self.decoder = FunnelDecoder(config) + + self.init_weights() + + def get_input_embeddings(self): + return self.embeddings.word_embeddings + + def set_input_embeddings(self, new_embeddings): + self.embeddings.word_embeddings = new_embeddings + + @add_start_docstrings_to_callable(FUNNEL_INPUTS_DOCSTRING.format("(batch_size, sequence_length)")) + @add_code_sample_docstrings( + tokenizer_class=_TOKENIZER_FOR_DOC, + checkpoint="funnel-transformer/small", + output_type=BaseModelOutput, + config_class=_CONFIG_FOR_DOC, + ) + def forward( + self, + input_ids=None, + attention_mask=None, + token_type_ids=None, + inputs_embeds=None, + output_attentions=None, + output_hidden_states=None, + return_dict=None, + ): + output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions + output_hidden_states = ( + output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states + ) + return_dict = return_dict if return_dict is not None else self.config.use_return_dict + + if input_ids is not None and inputs_embeds is not None: + raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time") + elif input_ids is not None: + input_shape = input_ids.size() + elif inputs_embeds is not None: + input_shape = inputs_embeds.size()[:-1] + else: + raise ValueError("You have to specify either input_ids or inputs_embeds") + + device = input_ids.device if input_ids is not None else inputs_embeds.device + + if attention_mask is None: + attention_mask = torch.ones(input_shape, device=device) + if token_type_ids is None: + token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=device) + + # TODO: deal with head_mask + if inputs_embeds is None: + inputs_embeds = self.embeddings(input_ids) + + encoder_outputs = self.encoder( + inputs_embeds, + attention_mask=attention_mask, + token_type_ids=token_type_ids, + output_attentions=output_attentions, + output_hidden_states=True, + return_dict=return_dict, + ) + + decoder_outputs = self.decoder( + final_hidden=encoder_outputs[0], + first_block_hidden=encoder_outputs[1][self.config.block_sizes[0]], + attention_mask=attention_mask, + token_type_ids=token_type_ids, + output_attentions=output_attentions, + output_hidden_states=output_hidden_states, + return_dict=return_dict, + ) + + if not return_dict: + idx = 0 + outputs = (decoder_outputs[0],) + if output_hidden_states: + idx += 1 + outputs = outputs + (encoder_outputs[1] + decoder_outputs[idx],) + if output_attentions: + idx += 1 + outputs = outputs + (encoder_outputs[2] + decoder_outputs[idx],) + return outputs + + return BaseModelOutput( + last_hidden_state=decoder_outputs[0], + hidden_states=(encoder_outputs.hidden_states + decoder_outputs.hidden_states) + if output_hidden_states + else None, + attentions=(encoder_outputs.attentions + decoder_outputs.attentions) if output_attentions else None, + ) + + +add_start_docstrings( + """ + Funnel Transformer model with a binary classification head on top as used during pretraining for identifying + generated tokens.""", + FUNNEL_START_DOCSTRING, +) + + +class FunnelForPreTraining(FunnelPreTrainedModel): + def __init__(self, config): + super().__init__(config) + + self.funnel = FunnelModel(config) + self.discriminator_predictions = FunnelDiscriminatorPredictions(config) + self.init_weights() + + @add_start_docstrings_to_callable(FUNNEL_INPUTS_DOCSTRING) + @replace_return_docstrings(output_type=FunnelForPreTrainingOutput, config_class=_CONFIG_FOR_DOC) + def forward( + self, + input_ids=None, + attention_mask=None, + token_type_ids=None, + inputs_embeds=None, + labels=None, + output_attentions=None, + output_hidden_states=None, + return_dict=None, + ): + r""" + labels (``torch.LongTensor`` of shape ``(batch_size, sequence_length)``, `optional`, defaults to :obj:`None`): + Labels for computing the ELECTRA-style loss. Input should be a sequence of tokens (see :obj:`input_ids` docstring) + Indices should be in ``[0, 1]``. + ``0`` indicates the token is an original token, + ``1`` indicates the token was replaced. + + Returns: + + Examples:: + + >>> from transformers import FunnelTokenizer, FunnelForPreTraining + >>> import torch + + >>> tokenizer = FunnelTokenizer.from_pretrained('funnel-transformer/small') + >>> model = FunnelForPreTraining.from_pretrained('funnel-transformer/small') + + >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 + >>> logits = model(input_ids).logits + """ + return_dict = return_dict if return_dict is not None else self.config.use_return_dict + + discriminator_hidden_states = self.funnel( + input_ids, + attention_mask=attention_mask, + token_type_ids=token_type_ids, + inputs_embeds=inputs_embeds, + output_attentions=output_attentions, + output_hidden_states=output_hidden_states, + return_dict=return_dict, + ) + discriminator_sequence_output = discriminator_hidden_states[0] + + logits = self.discriminator_predictions(discriminator_sequence_output) + + loss = None + if labels is not None: + loss_fct = nn.BCEWithLogitsLoss() + if attention_mask is not None: + active_loss = attention_mask.view(-1, discriminator_sequence_output.shape[1]) == 1 + active_logits = logits.view(-1, discriminator_sequence_output.shape[1])[active_loss] + active_labels = labels[active_loss] + loss = loss_fct(active_logits, active_labels.float()) + else: + loss = loss_fct(logits.view(-1, discriminator_sequence_output.shape[1]), labels.float()) + + if not return_dict: + output = (logits,) + discriminator_hidden_states[1:] + return ((loss,) + output) if loss is not None else output + + return FunnelForPreTrainingOutput( + loss=loss, + logits=logits, + hidden_states=discriminator_hidden_states.hidden_states, + attentions=discriminator_hidden_states.attentions, + ) + + +@add_start_docstrings("""Funnel Transformer Model with a `language modeling` head on top. """, FUNNEL_START_DOCSTRING) +class FunnelForMaskedLM(FunnelPreTrainedModel): + def __init__(self, config): + super().__init__(config) + + self.funnel = FunnelModel(config) + self.lm_head = nn.Linear(config.d_model, config.vocab_size) + + self.init_weights() + + def get_output_embeddings(self): + return self.lm_head + + @add_start_docstrings_to_callable(FUNNEL_INPUTS_DOCSTRING.format("(batch_size, sequence_length)")) + @add_code_sample_docstrings( + tokenizer_class=_TOKENIZER_FOR_DOC, + checkpoint="funnel-transformer/small", + output_type=MaskedLMOutput, + config_class=_CONFIG_FOR_DOC, + ) + def forward( + self, + input_ids=None, + attention_mask=None, + token_type_ids=None, + inputs_embeds=None, + labels=None, + output_attentions=None, + output_hidden_states=None, + return_dict=None, + ): + r""" + labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + Labels for computing the masked language modeling loss. + Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) + Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels + in ``[0, ..., config.vocab_size]`` + """ + return_dict = return_dict if return_dict is not None else self.config.use_return_dict + + outputs = self.funnel( + input_ids, + attention_mask=attention_mask, + token_type_ids=token_type_ids, + inputs_embeds=inputs_embeds, + output_attentions=output_attentions, + output_hidden_states=output_hidden_states, + return_dict=return_dict, + ) + + last_hidden_state = outputs[0] + prediction_logits = self.lm_head(last_hidden_state) + + masked_lm_loss = None + if labels is not None: + loss_fct = CrossEntropyLoss() # -100 index = padding token + masked_lm_loss = loss_fct(prediction_logits.view(-1, self.config.vocab_size), labels.view(-1)) + + if not return_dict: + output = (prediction_logits,) + outputs[1:] + return ((masked_lm_loss,) + output) if masked_lm_loss is not None else output + + return MaskedLMOutput( + loss=masked_lm_loss, + logits=prediction_logits, + hidden_states=outputs.hidden_states, + attentions=outputs.attentions, + ) + + +@add_start_docstrings( + """Funnel Transfprmer Model with a sequence classification/regression head on top (two linear layer on top of + the first timestep of the last hidden state) e.g. for GLUE tasks. """, + FUNNEL_START_DOCSTRING, +) +class FunnelForSequenceClassification(FunnelPreTrainedModel): + def __init__(self, config): + super().__init__(config) + self.num_labels = config.num_labels + + self.funnel = FunnelBaseModel(config) + self.classifier = FunnelClassificationHead(config, config.num_labels) + self.init_weights() + + @add_start_docstrings_to_callable(FUNNEL_INPUTS_DOCSTRING.format("(batch_size, sequence_length)")) + @add_code_sample_docstrings( + tokenizer_class=_TOKENIZER_FOR_DOC, + checkpoint="funnel-transformer/small-base", + output_type=SequenceClassifierOutput, + config_class=_CONFIG_FOR_DOC, + ) + def forward( + self, + input_ids=None, + attention_mask=None, + token_type_ids=None, + inputs_embeds=None, + labels=None, + output_attentions=None, + output_hidden_states=None, + return_dict=None, + ): + r""" + labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): + Labels for computing the sequence classification/regression loss. + Indices should be in :obj:`[0, ..., config.num_labels - 1]`. + If :obj:`config.num_labels == 1` a regression loss is computed (Mean-Square loss), + If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy). + """ + return_dict = return_dict if return_dict is not None else self.config.use_return_dict + + outputs = self.funnel( + input_ids, + attention_mask=attention_mask, + token_type_ids=token_type_ids, + inputs_embeds=inputs_embeds, + output_attentions=output_attentions, + output_hidden_states=output_hidden_states, + return_dict=return_dict, + ) + + last_hidden_state = outputs[0] + pooled_output = last_hidden_state[:, 0] + logits = self.classifier(pooled_output) + + loss = None + if labels is not None: + if self.num_labels == 1: + # We are doing regression + loss_fct = MSELoss() + loss = loss_fct(logits.view(-1), labels.view(-1)) + else: + loss_fct = CrossEntropyLoss() + loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) + + if not return_dict: + output = (logits,) + outputs[1:] + return ((loss,) + output) if loss is not None else output + + return SequenceClassifierOutput( + loss=loss, + logits=logits, + hidden_states=outputs.hidden_states, + attentions=outputs.attentions, + ) + + +@add_start_docstrings( + """Funnel Transformer Model with a multiple choice classification head on top (two linear layer on top of + the first timestep of the last hidden state, and a softmax) e.g. for RocStories/SWAG tasks. """, + FUNNEL_START_DOCSTRING, +) +class FunnelForMultipleChoice(FunnelPreTrainedModel): + def __init__(self, config): + super().__init__(config) + + self.funnel = FunnelBaseModel(config) + self.classifier = FunnelClassificationHead(config, 1) + self.init_weights() + + @add_start_docstrings_to_callable(FUNNEL_INPUTS_DOCSTRING.format("(batch_size, num_choices, sequence_length)")) + @add_code_sample_docstrings( + tokenizer_class=_TOKENIZER_FOR_DOC, + checkpoint="funnel-transformer/small-base", + output_type=MultipleChoiceModelOutput, + config_class=_CONFIG_FOR_DOC, + ) + def forward( + self, + input_ids=None, + attention_mask=None, + token_type_ids=None, + inputs_embeds=None, + labels=None, + output_attentions=None, + output_hidden_states=None, + return_dict=None, + ): + r""" + labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): + Labels for computing the multiple choice classification loss. + Indices should be in ``[0, ..., num_choices-1]`` where `num_choices` is the size of the second dimension + of the input tensors. (see `input_ids` above) + """ + return_dict = return_dict if return_dict is not None else self.config.use_return_dict + num_choices = input_ids.shape[1] if input_ids is not None else inputs_embeds.shape[1] + + input_ids = input_ids.view(-1, input_ids.size(-1)) if input_ids is not None else None + attention_mask = attention_mask.view(-1, attention_mask.size(-1)) if attention_mask is not None else None + token_type_ids = token_type_ids.view(-1, token_type_ids.size(-1)) if token_type_ids is not None else None + inputs_embeds = ( + inputs_embeds.view(-1, inputs_embeds.size(-2), inputs_embeds.size(-1)) + if inputs_embeds is not None + else None + ) + + outputs = self.funnel( + input_ids, + attention_mask=attention_mask, + token_type_ids=token_type_ids, + inputs_embeds=inputs_embeds, + output_attentions=output_attentions, + output_hidden_states=output_hidden_states, + return_dict=return_dict, + ) + + last_hidden_state = outputs[0] + pooled_output = last_hidden_state[:, 0] + logits = self.classifier(pooled_output) + reshaped_logits = logits.view(-1, num_choices) + + loss = None + if labels is not None: + loss_fct = CrossEntropyLoss() + loss = loss_fct(reshaped_logits, labels) + + if not return_dict: + output = (reshaped_logits,) + outputs[1:] + return ((loss,) + output) if loss is not None else output + + return MultipleChoiceModelOutput( + loss=loss, + logits=reshaped_logits, + hidden_states=outputs.hidden_states, + attentions=outputs.attentions, + ) + + +@add_start_docstrings( + """Funnel Transformer Model with a token classification head on top (a linear layer on top of + the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. """, + FUNNEL_START_DOCSTRING, +) +class FunnelForTokenClassification(FunnelPreTrainedModel): + def __init__(self, config): + super().__init__(config) + self.num_labels = config.num_labels + + self.funnel = FunnelModel(config) + self.dropout = nn.Dropout(config.hidden_dropout) + self.classifier = nn.Linear(config.hidden_size, config.num_labels) + + self.init_weights() + + @add_start_docstrings_to_callable(FUNNEL_INPUTS_DOCSTRING.format("(batch_size, sequence_length)")) + @add_code_sample_docstrings( + tokenizer_class=_TOKENIZER_FOR_DOC, + checkpoint="funnel-transformer/small", + output_type=TokenClassifierOutput, + config_class=_CONFIG_FOR_DOC, + ) + def forward( + self, + input_ids=None, + attention_mask=None, + token_type_ids=None, + inputs_embeds=None, + labels=None, + output_attentions=None, + output_hidden_states=None, + return_dict=None, + ): + r""" + labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): + Labels for computing the token classification loss. + Indices should be in ``[0, ..., config.num_labels - 1]``. + """ + return_dict = return_dict if return_dict is not None else self.config.use_return_dict + + outputs = self.funnel( + input_ids, + attention_mask=attention_mask, + token_type_ids=token_type_ids, + inputs_embeds=inputs_embeds, + output_attentions=output_attentions, + output_hidden_states=output_hidden_states, + return_dict=return_dict, + ) + + last_hidden_state = outputs[0] + last_hidden_state = self.dropout(last_hidden_state) + logits = self.classifier(last_hidden_state) + + loss = None + if labels is not None: + loss_fct = CrossEntropyLoss() + # Only keep active parts of the loss + if attention_mask is not None: + active_loss = attention_mask.view(-1) == 1 + active_logits = logits.view(-1, self.num_labels) + active_labels = torch.where( + active_loss, labels.view(-1), torch.tensor(loss_fct.ignore_index).type_as(labels) + ) + loss = loss_fct(active_logits, active_labels) + else: + loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) + + if not return_dict: + output = (logits,) + outputs[1:] + return ((loss,) + output) if loss is not None else output + + return TokenClassifierOutput( + loss=loss, + logits=logits, + hidden_states=outputs.hidden_states, + attentions=outputs.attentions, + ) + + +@add_start_docstrings( + """Funnel Transformer Model with a span classification head on top for extractive question-answering tasks like + SQuAD (a linear layer on top of the hidden-states output to compute `span start logits` and `span end logits`).""", + FUNNEL_START_DOCSTRING, +) +class FunnelForQuestionAnswering(FunnelPreTrainedModel): + def __init__(self, config): + super().__init__(config) + self.num_labels = config.num_labels + + self.funnel = FunnelModel(config) + self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels) + + self.init_weights() + + @add_start_docstrings_to_callable(FUNNEL_INPUTS_DOCSTRING.format("(batch_size, sequence_length)")) + @add_code_sample_docstrings( + tokenizer_class=_TOKENIZER_FOR_DOC, + checkpoint="funnel-transformer/small", + output_type=QuestionAnsweringModelOutput, + config_class=_CONFIG_FOR_DOC, + ) + def forward( + self, + input_ids=None, + attention_mask=None, + token_type_ids=None, + inputs_embeds=None, + start_positions=None, + end_positions=None, + output_attentions=None, + output_hidden_states=None, + return_dict=None, + ): + r""" + start_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): + Labels for position (index) of the start of the labelled span for computing the token classification loss. + Positions are clamped to the length of the sequence (`sequence_length`). + Position outside of the sequence are not taken into account for computing the loss. + end_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): + Labels for position (index) of the end of the labelled span for computing the token classification loss. + Positions are clamped to the length of the sequence (`sequence_length`). + Position outside of the sequence are not taken into account for computing the loss. + """ + return_dict = return_dict if return_dict is not None else self.config.use_return_dict + + outputs = self.funnel( + input_ids, + attention_mask=attention_mask, + token_type_ids=token_type_ids, + inputs_embeds=inputs_embeds, + output_attentions=output_attentions, + output_hidden_states=output_hidden_states, + return_dict=return_dict, + ) + + last_hidden_state = outputs[0] + + logits = self.qa_outputs(last_hidden_state) + start_logits, end_logits = logits.split(1, dim=-1) + start_logits = start_logits.squeeze(-1) + end_logits = end_logits.squeeze(-1) + + total_loss = None + if start_positions is not None and end_positions is not None: + # If we are on multi-GPU, split add a dimension + if len(start_positions.size()) > 1: + start_positions = start_positions.squeze(-1) + if len(end_positions.size()) > 1: + end_positions = end_positions.squeeze(-1) + # sometimes the start/end positions are outside our model inputs, we ignore these terms + ignored_index = start_logits.size(1) + start_positions.clamp_(0, ignored_index) + end_positions.clamp_(0, ignored_index) + + loss_fct = CrossEntropyLoss(ignore_index=ignored_index) + start_loss = loss_fct(start_logits, start_positions) + end_loss = loss_fct(end_logits, end_positions) + total_loss = (start_loss + end_loss) / 2 + + if not return_dict: + output = (start_logits, end_logits) + outputs[1:] + return ((total_loss,) + output) if total_loss is not None else output + + return QuestionAnsweringModelOutput( + loss=total_loss, + start_logits=start_logits, + end_logits=end_logits, + hidden_states=outputs.hidden_states, + attentions=outputs.attentions, + ) diff --git a/src/transformers/tokenization_auto.py b/src/transformers/tokenization_auto.py index 626e576759..7dc44e6bf5 100644 --- a/src/transformers/tokenization_auto.py +++ b/src/transformers/tokenization_auto.py @@ -27,6 +27,7 @@ from .configuration_auto import ( DistilBertConfig, ElectraConfig, FlaubertConfig, + FunnelConfig, GPT2Config, LongformerConfig, LxmertConfig, @@ -54,6 +55,7 @@ from .tokenization_ctrl import CTRLTokenizer from .tokenization_distilbert import DistilBertTokenizer, DistilBertTokenizerFast from .tokenization_electra import ElectraTokenizer, ElectraTokenizerFast from .tokenization_flaubert import FlaubertTokenizer +from .tokenization_funnel import FunnelTokenizer, FunnelTokenizerFast from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast from .tokenization_longformer import LongformerTokenizer, LongformerTokenizerFast from .tokenization_lxmert import LxmertTokenizer, LxmertTokenizerFast @@ -93,6 +95,7 @@ TOKENIZER_MAPPING = OrderedDict( (RobertaConfig, (RobertaTokenizer, RobertaTokenizerFast)), (ReformerConfig, (ReformerTokenizer, None)), (ElectraConfig, (ElectraTokenizer, ElectraTokenizerFast)), + (FunnelConfig, (FunnelTokenizer, FunnelTokenizerFast)), (LxmertConfig, (LxmertTokenizer, LxmertTokenizerFast)), (BertConfig, (BertTokenizer, BertTokenizerFast)), (OpenAIGPTConfig, (OpenAIGPTTokenizer, OpenAIGPTTokenizerFast)), @@ -131,6 +134,7 @@ class AutoTokenizer: - `xlm`: XLMTokenizer (XLM model) - `ctrl`: CTRLTokenizer (Salesforce CTRL model) - `electra`: ElectraTokenizer (Google ELECTRA model) + - `funnel`: FunnelTokenizer (Funnel Transformer model) - `lxmert`: LxmertTokenizer (Lxmert model) This class cannot be instantiated using `__init__()` (throw an error). @@ -167,6 +171,7 @@ class AutoTokenizer: - `xlm`: XLMTokenizer (XLM model) - `ctrl`: CTRLTokenizer (Salesforce CTRL model) - `electra`: ElectraTokenizer (Google ELECTRA model) + - `funnel`: FunnelTokenizer (Funnel Transformer model) - `lxmert`: LxmertTokenizer (Lxmert model) Params: diff --git a/src/transformers/tokenization_funnel.py b/src/transformers/tokenization_funnel.py new file mode 100644 index 0000000000..a0ca4e4011 --- /dev/null +++ b/src/transformers/tokenization_funnel.py @@ -0,0 +1,232 @@ +# coding=utf-8 +# Copyright 2020 The HuggingFace Inc. team. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" Tokenization class for Funnel Transformer.""" + +from typing import List, Optional + +from .tokenization_bert import BertTokenizer, BertTokenizerFast +from .utils import logging + + +logger = logging.get_logger(__name__) + +VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt"} + +_model_names = [ + "small", + "small-base", + "medium", + "medium-base", + "intermediate", + "intermediate-base", + "large", + "large-base", + "xlarge", + "xlarge-base", +] + +PRETRAINED_VOCAB_FILES_MAP = { + "vocab_file": { + "funnel-transformer/small": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/small/vocab.txt", + "funnel-transformer/small-base": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/small-base/vocab.txt", + "funnel-transformer/medium": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/medium/vocab.txt", + "funnel-transformer/medium-base": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/medium-base/vocab.txt", + "funnel-transformer/intermediate": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/intermediate/vocab.txt", + "funnel-transformer/intermediate-base": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/intermediate-base/vocab.txt", + "funnel-transformer/large": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/large/vocab.txt", + "funnel-transformer/large-base": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/large-base/vocab.txt", + "funnel-transformer/xlarge": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/xlarge/vocab.txt", + "funnel-transformer/xlarge-base": "https://s3.amazonaws.com/models.huggingface.co/bert/funnel-transformer/xlarge-base/vocab.txt", + } +} +PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {f"funnel-transformer/{name}": 512 for name in _model_names} +PRETRAINED_INIT_CONFIGURATION = {f"funnel-transformer/{name}": {"do_lower_case": True} for name in _model_names} + + +class FunnelTokenizer(BertTokenizer): + r""" + Tokenizer for the Funnel Transformer models. + + :class:`~transformers.FunnelTokenizer` is identical to :class:`~transformers.BertTokenizer` and runs end-to-end + tokenization: punctuation splitting + wordpiece. + + Refer to superclass :class:`~transformers.BertTokenizer` for usage examples and documentation concerning + parameters. + """ + + vocab_files_names = VOCAB_FILES_NAMES + pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP + max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES + pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION + cls_token_type_id: int = 2 + + def __init__( + self, + vocab_file, + do_lower_case=True, + do_basic_tokenize=True, + never_split=None, + unk_token="", + sep_token="", + pad_token="", + cls_token="", + mask_token="", + bos_token="", + eos_token="", + tokenize_chinese_chars=True, + strip_accents=None, + **kwargs + ): + super().__init__( + vocab_file, + do_lower_case=do_lower_case, + do_basic_tokenize=do_basic_tokenize, + never_split=never_split, + unk_token=unk_token, + sep_token=sep_token, + pad_token=pad_token, + cls_token=cls_token, + mask_token=mask_token, + bos_token=bos_token, + eos_token=eos_token, + tokenize_chinese_chars=tokenize_chinese_chars, + strip_accents=strip_accents, + **kwargs, + ) + + def create_token_type_ids_from_sequences( + self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None + ) -> List[int]: + """ + Creates a mask from the two sequences passed to be used in a sequence-pair classification task. + Funnel Transformer expects a sequence pair mask that has the following format: + + :: + + 2 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 + | first sequence | second sequence | + + if token_ids_1 is None, only returns the first portion of the mask (0's). + + Args: + token_ids_0 (:obj:`List[int]`): + List of ids. + token_ids_1 (:obj:`List[int]`, `optional`): + Optional second list of IDs for sequence pairs. + + Returns: + :obj:`List[int]`: List of `token type IDs <../glossary.html#token-type-ids>`_ according to the given + sequence(s). + """ + sep = [self.sep_token_id] + cls = [self.cls_token_id] + if token_ids_1 is None: + return len(cls) * [self.cls_token_type_id] + len(token_ids_0 + sep) * [0] + return len(cls) * [self.cls_token_type_id] + len(token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1] + + +class FunnelTokenizerFast(BertTokenizerFast): + r""" + "Fast" tokenizer for the Funnel Transformer models (backed by HuggingFace's :obj:`tokenizers` library). + + :class:`~transformers.FunnelTokenizerFast` is identical to :class:`~transformers.BertTokenizerFast` and runs + end-to-end tokenization: punctuation splitting + wordpiece. + + Refer to superclass :class:`~transformers.BertTokenizerFast` for usage examples and documentation concerning + parameters. + """ + + vocab_files_names = VOCAB_FILES_NAMES + pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP + max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES + pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION + cls_token_type_id: int = 2 + + def __init__( + self, + vocab_file, + do_lower_case=True, + unk_token="", + sep_token="", + pad_token="", + cls_token="", + mask_token="", + bos_token="", + eos_token="", + clean_text=True, + tokenize_chinese_chars=True, + strip_accents=None, + wordpieces_prefix="##", + **kwargs + ): + super().__init__( + vocab_file, + do_lower_case=do_lower_case, + unk_token=unk_token, + sep_token=sep_token, + pad_token=pad_token, + cls_token=cls_token, + mask_token=mask_token, + bos_token=bos_token, + eos_token=eos_token, + clean_text=clean_text, + tokenize_chinese_chars=tokenize_chinese_chars, + strip_accents=strip_accents, + wordpieces_prefix=wordpieces_prefix, + **kwargs, + ) + + def create_token_type_ids_from_sequences( + self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None + ) -> List[int]: + """ + Creates a mask from the two sequences passed to be used in a sequence-pair classification task. + Funnel Transformer expects a sequence pair mask that has the following format: + + :: + + 2 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 + | first sequence | second sequence | + + if token_ids_1 is None, only returns the first portion of the mask (0's). + + Args: + token_ids_0 (:obj:`List[int]`): + List of ids. + token_ids_1 (:obj:`List[int]`, `optional`): + Optional second list of IDs for sequence pairs. + + Returns: + :obj:`List[int]`: List of `token type IDs <../glossary.html#token-type-ids>`_ according to the given + sequence(s). + """ + sep = [self.sep_token_id] + cls = [self.cls_token_id] + if token_ids_1 is None: + return len(cls) * [self.cls_token_type_id] + len(token_ids_0 + sep) * [0] + return len(cls) * [self.cls_token_type_id] + len(token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1] + + def _convert_encoding(self, encoding, **kwargs): + # The fast tokenizer doesn't use the function above so we fix the cls token type id when decoding the fast + # tokenzier output. + encoding_dict = super()._convert_encoding(encoding, **kwargs) + if "token_type_ids" in encoding_dict: + # Note: we can't assume the token is in first position because left padding is a thing, hence the + # double list comprehension. + encoding_dict["token_type_ids"] = [ + [self.cls_token_type_id if i == self.cls_token_id else t for i, t in zip(input_ids, type_ids)] + for input_ids, type_ids in zip(encoding_dict["input_ids"], encoding_dict["token_type_ids"]) + ] + return encoding_dict diff --git a/tests/test_modeling_common.py b/tests/test_modeling_common.py index 18d96d3d04..28b7cb87e8 100755 --- a/tests/test_modeling_common.py +++ b/tests/test_modeling_common.py @@ -539,7 +539,10 @@ class ModelTesterMixin: outputs = model(**self._prepare_for_class(inputs_dict, model_class)) hidden_states = outputs[-1] - self.assertEqual(len(hidden_states), self.model_tester.num_hidden_layers + 1) + expected_num_layers = getattr( + self.model_tester, "expected_num_hidden_layers", self.model_tester.num_hidden_layers + 1 + ) + self.assertEqual(len(hidden_states), expected_num_layers) if hasattr(self.model_tester, "encoder_seq_length"): seq_length = self.model_tester.encoder_seq_length if hasattr(self.model_tester, "chunk_length") and self.model_tester.chunk_length > 1: diff --git a/tests/test_modeling_funnel.py b/tests/test_modeling_funnel.py new file mode 100644 index 0000000000..0197a8fec0 --- /dev/null +++ b/tests/test_modeling_funnel.py @@ -0,0 +1,454 @@ +# coding=utf-8 +# Copyright 2020 HuggingFace Inc. team. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import unittest + +from transformers import FunnelTokenizer, is_torch_available +from transformers.testing_utils import require_torch, slow, torch_device + +from .test_configuration_common import ConfigTester +from .test_modeling_common import ModelTesterMixin, ids_tensor + + +if is_torch_available(): + import torch + + from transformers import ( + FunnelBaseModel, + FunnelConfig, + FunnelForMaskedLM, + FunnelForMultipleChoice, + FunnelForPreTraining, + FunnelForQuestionAnswering, + FunnelForSequenceClassification, + FunnelForTokenClassification, + FunnelModel, + ) + + +class FunnelModelTester: + """You can also import this e.g, from .test_modeling_funnel import FunnelModelTester """ + + def __init__( + self, + parent, + batch_size=13, + seq_length=7, + is_training=True, + use_input_mask=True, + use_token_type_ids=True, + use_labels=True, + vocab_size=99, + block_sizes=[1, 1, 2], + num_decoder_layers=1, + d_model=32, + n_head=4, + d_head=8, + d_inner=37, + hidden_act="gelu_new", + hidden_dropout=0.1, + attention_dropout=0.1, + activation_dropout=0.0, + max_position_embeddings=512, + type_vocab_size=3, + num_labels=3, + num_choices=4, + scope=None, + base=False, + ): + self.parent = parent + self.batch_size = batch_size + self.seq_length = seq_length + self.is_training = is_training + self.use_input_mask = use_input_mask + self.use_token_type_ids = use_token_type_ids + self.use_labels = use_labels + self.vocab_size = vocab_size + self.block_sizes = block_sizes + self.num_decoder_layers = num_decoder_layers + self.d_model = d_model + self.n_head = n_head + self.d_head = d_head + self.d_inner = d_inner + self.hidden_act = hidden_act + self.hidden_dropout = hidden_dropout + self.attention_dropout = attention_dropout + self.activation_dropout = activation_dropout + self.max_position_embeddings = max_position_embeddings + self.type_vocab_size = type_vocab_size + self.type_sequence_label_size = 2 + self.num_labels = num_labels + self.num_choices = num_choices + self.scope = scope + + # Used in the tests to check the size of the first attention layer + self.num_attention_heads = n_head + # Used in the tests to check the size of the first hidden state + self.hidden_size = self.d_model + # Used in the tests to check the number of output hidden states/attentions + self.num_hidden_layers = sum(self.block_sizes) + (0 if base else self.num_decoder_layers) + # FunnelModel adds two hidden layers: input embeddings and the sum of the upsampled encoder hidden state with + # the last hidden state of the first block (which is the first hidden state of the decoder). + if not base: + self.expected_num_hidden_layers = self.num_hidden_layers + 2 + + def prepare_config_and_inputs(self): + input_ids = ids_tensor([self.batch_size, self.seq_length], self.vocab_size) + + input_mask = None + if self.use_input_mask: + input_mask = ids_tensor([self.batch_size, self.seq_length], vocab_size=2) + + token_type_ids = None + if self.use_token_type_ids: + token_type_ids = ids_tensor([self.batch_size, self.seq_length], self.type_vocab_size) + + sequence_labels = None + token_labels = None + choice_labels = None + if self.use_labels: + sequence_labels = ids_tensor([self.batch_size], self.type_sequence_label_size) + token_labels = ids_tensor([self.batch_size, self.seq_length], self.num_labels) + choice_labels = ids_tensor([self.batch_size], self.num_choices) + fake_token_labels = ids_tensor([self.batch_size, self.seq_length], 1) + + config = FunnelConfig( + vocab_size=self.vocab_size, + block_sizes=self.block_sizes, + num_decoder_layers=self.num_decoder_layers, + d_model=self.d_model, + n_head=self.n_head, + d_head=self.d_head, + d_inner=self.d_inner, + hidden_act=self.hidden_act, + hidden_dropout=self.hidden_dropout, + attention_dropout=self.attention_dropout, + activation_dropout=self.activation_dropout, + max_position_embeddings=self.max_position_embeddings, + type_vocab_size=self.type_vocab_size, + return_dict=True, + ) + + return ( + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + fake_token_labels, + ) + + def create_and_check_model( + self, + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + fake_token_labels, + ): + model = FunnelModel(config=config) + model.to(torch_device) + model.eval() + result = model(input_ids, attention_mask=input_mask, token_type_ids=token_type_ids) + result = model(input_ids, token_type_ids=token_type_ids) + result = model(input_ids) + self.parent.assertEqual(result.last_hidden_state.shape, (self.batch_size, self.seq_length, self.d_model)) + + model.config.truncate_seq = False + result = model(input_ids) + self.parent.assertEqual(result.last_hidden_state.shape, (self.batch_size, self.seq_length, self.d_model)) + + model.config.separate_cls = False + result = model(input_ids) + self.parent.assertEqual(result.last_hidden_state.shape, (self.batch_size, self.seq_length, self.d_model)) + + def create_and_check_base_model( + self, + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + fake_token_labels, + ): + model = FunnelBaseModel(config=config) + model.to(torch_device) + model.eval() + result = model(input_ids, attention_mask=input_mask, token_type_ids=token_type_ids) + result = model(input_ids, token_type_ids=token_type_ids) + result = model(input_ids) + self.parent.assertEqual(result.last_hidden_state.shape, (self.batch_size, 2, self.d_model)) + + model.config.truncate_seq = False + result = model(input_ids) + self.parent.assertEqual(result.last_hidden_state.shape, (self.batch_size, 3, self.d_model)) + + model.config.separate_cls = False + result = model(input_ids) + self.parent.assertEqual(result.last_hidden_state.shape, (self.batch_size, 2, self.d_model)) + + def create_and_check_for_pretraining( + self, + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + fake_token_labels, + ): + config.num_labels = self.num_labels + model = FunnelForPreTraining(config=config) + model.to(torch_device) + model.eval() + result = model(input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, labels=fake_token_labels) + self.parent.assertEqual(result.logits.shape, (self.batch_size, self.seq_length)) + + def create_and_check_for_masked_lm( + self, + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + fake_token_labels, + ): + model = FunnelForMaskedLM(config=config) + model.to(torch_device) + model.eval() + result = model(input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, labels=token_labels) + self.parent.assertEqual(result.logits.shape, (self.batch_size, self.seq_length, self.vocab_size)) + + def create_and_check_for_sequence_classification( + self, + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + fake_token_labels, + ): + config.num_labels = self.num_labels + model = FunnelForSequenceClassification(config) + model.to(torch_device) + model.eval() + result = model(input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, labels=sequence_labels) + self.parent.assertEqual(result.logits.shape, (self.batch_size, self.num_labels)) + + def create_and_check_for_multiple_choice( + self, + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + fake_token_labels, + ): + config.num_choices = self.num_choices + model = FunnelForMultipleChoice(config=config) + model.to(torch_device) + model.eval() + multiple_choice_inputs_ids = input_ids.unsqueeze(1).expand(-1, self.num_choices, -1).contiguous() + multiple_choice_token_type_ids = token_type_ids.unsqueeze(1).expand(-1, self.num_choices, -1).contiguous() + multiple_choice_input_mask = input_mask.unsqueeze(1).expand(-1, self.num_choices, -1).contiguous() + result = model( + multiple_choice_inputs_ids, + attention_mask=multiple_choice_input_mask, + token_type_ids=multiple_choice_token_type_ids, + labels=choice_labels, + ) + self.parent.assertEqual(result.logits.shape, (self.batch_size, self.num_choices)) + + def create_and_check_for_token_classification( + self, + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + fake_token_labels, + ): + config.num_labels = self.num_labels + model = FunnelForTokenClassification(config=config) + model.to(torch_device) + model.eval() + result = model(input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, labels=token_labels) + self.parent.assertEqual(result.logits.shape, (self.batch_size, self.seq_length, self.num_labels)) + + def create_and_check_for_question_answering( + self, + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + fake_token_labels, + ): + model = FunnelForQuestionAnswering(config=config) + model.to(torch_device) + model.eval() + result = model( + input_ids, + attention_mask=input_mask, + token_type_ids=token_type_ids, + start_positions=sequence_labels, + end_positions=sequence_labels, + ) + self.parent.assertEqual(result.start_logits.shape, (self.batch_size, self.seq_length)) + self.parent.assertEqual(result.end_logits.shape, (self.batch_size, self.seq_length)) + + def prepare_config_and_inputs_for_common(self): + config_and_inputs = self.prepare_config_and_inputs() + ( + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + fake_token_labels, + ) = config_and_inputs + inputs_dict = {"input_ids": input_ids, "token_type_ids": token_type_ids, "attention_mask": input_mask} + return config, inputs_dict + + +@require_torch +class FunnelModelTest(ModelTesterMixin, unittest.TestCase): + test_head_masking = False + test_pruning = False + all_model_classes = ( + ( + FunnelModel, + FunnelForMaskedLM, + FunnelForPreTraining, + FunnelForQuestionAnswering, + FunnelForTokenClassification, + ) + if is_torch_available() + else () + ) + + def setUp(self): + self.model_tester = FunnelModelTester(self) + self.config_tester = ConfigTester(self, config_class=FunnelConfig) + + def test_config(self): + self.config_tester.run_common_tests() + + def test_model(self): + config_and_inputs = self.model_tester.prepare_config_and_inputs() + self.model_tester.create_and_check_model(*config_and_inputs) + + def test_for_pretraining(self): + config_and_inputs = self.model_tester.prepare_config_and_inputs() + self.model_tester.create_and_check_for_pretraining(*config_and_inputs) + + def test_for_masked_lm(self): + config_and_inputs = self.model_tester.prepare_config_and_inputs() + self.model_tester.create_and_check_for_masked_lm(*config_and_inputs) + + def test_for_token_classification(self): + config_and_inputs = self.model_tester.prepare_config_and_inputs() + self.model_tester.create_and_check_for_token_classification(*config_and_inputs) + + def test_for_question_answering(self): + config_and_inputs = self.model_tester.prepare_config_and_inputs() + self.model_tester.create_and_check_for_question_answering(*config_and_inputs) + + +@require_torch +class FunnelBaseModelTest(ModelTesterMixin, unittest.TestCase): + test_head_masking = False + test_pruning = False + all_model_classes = ( + (FunnelBaseModel, FunnelForMultipleChoice, FunnelForSequenceClassification) if is_torch_available() else () + ) + + def setUp(self): + self.model_tester = FunnelModelTester(self, base=True) + self.config_tester = ConfigTester(self, config_class=FunnelConfig) + + def test_config(self): + self.config_tester.run_common_tests() + + def test_base_model(self): + config_and_inputs = self.model_tester.prepare_config_and_inputs() + self.model_tester.create_and_check_base_model(*config_and_inputs) + + def test_for_sequence_classification(self): + config_and_inputs = self.model_tester.prepare_config_and_inputs() + self.model_tester.create_and_check_for_sequence_classification(*config_and_inputs) + + def test_for_multiple_choice(self): + config_and_inputs = self.model_tester.prepare_config_and_inputs() + self.model_tester.create_and_check_for_multiple_choice(*config_and_inputs) + + +@require_torch +class FunnelModelIntegrationTest(unittest.TestCase): + def test_inference_tiny_model(self): + batch_size = 13 + sequence_length = 7 + input_ids = torch.arange(0, batch_size * sequence_length).long().reshape(batch_size, sequence_length) + lengths = [0, 1, 2, 3, 4, 5, 6, 4, 1, 3, 5, 0, 1] + token_type_ids = torch.tensor([[2] + [0] * a + [1] * (sequence_length - a - 1) for a in lengths]) + + model = FunnelModel.from_pretrained("sgugger/funnel-random-tiny") + output = model(input_ids, token_type_ids=token_type_ids)[0].abs() + + expected_output_sum = torch.tensor(2344.9023) + expected_output_mean = torch.tensor(0.8053) + self.assertTrue(torch.allclose(output.sum(), expected_output_sum, atol=1e-4)) + self.assertTrue(torch.allclose(output.mean(), expected_output_mean, atol=1e-4)) + + attention_mask = torch.tensor([[1] * 7, [1] * 4 + [0] * 3] * 6 + [[0, 1, 1, 0, 0, 1, 1]]) + output = model(input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)[0].abs() + + expected_output_sum = torch.tensor(2363.2178) + expected_output_mean = torch.tensor(0.8115) + self.assertTrue(torch.allclose(output.sum(), expected_output_sum, atol=1e-4)) + self.assertTrue(torch.allclose(output.mean(), expected_output_mean, atol=1e-4)) + + @slow + def test_inference_model(self): + tokenizer = FunnelTokenizer.from_pretrained("huggingface/funnel-small") + model = FunnelModel.from_pretrained("huggingface/funnel-small") + inputs = tokenizer("Hello! I am the Funnel Transformer model.", return_tensors="pt") + output = model(**inputs)[0] + + expected_output_sum = torch.tensor(235.7827) + expected_output_mean = torch.tensor(0.0256) + self.assertTrue(torch.allclose(output.sum(), expected_output_sum, atol=1e-4)) + self.assertTrue(torch.allclose(output.mean(), expected_output_mean, atol=1e-4)) diff --git a/tests/test_tokenization_funnel.py b/tests/test_tokenization_funnel.py new file mode 100644 index 0000000000..8c9ce7a3d4 --- /dev/null +++ b/tests/test_tokenization_funnel.py @@ -0,0 +1,78 @@ +# coding=utf-8 +# Copyright 2020 HuggingFace Inc. team. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import os +import unittest + +from transformers.tokenization_funnel import VOCAB_FILES_NAMES, FunnelTokenizer, FunnelTokenizerFast + +from .test_tokenization_common import TokenizerTesterMixin + + +class FunnelTokenizationTest(TokenizerTesterMixin, unittest.TestCase): + + tokenizer_class = FunnelTokenizer + test_rust_tokenizer = True + + def setUp(self): + super().setUp() + + vocab_tokens = [ + "", + "", + "", + "want", + "##want", + "##ed", + "wa", + "un", + "runn", + "##ing", + ",", + "low", + "lowest", + ] + self.vocab_file = os.path.join(self.tmpdirname, VOCAB_FILES_NAMES["vocab_file"]) + with open(self.vocab_file, "w", encoding="utf-8") as vocab_writer: + vocab_writer.write("".join([x + "\n" for x in vocab_tokens])) + + def get_tokenizer(self, **kwargs): + return FunnelTokenizer.from_pretrained(self.tmpdirname, **kwargs) + + def get_rust_tokenizer(self, **kwargs): + return FunnelTokenizerFast.from_pretrained(self.tmpdirname, **kwargs) + + def get_input_output_texts(self, tokenizer): + input_text = "UNwant\u00E9d,running" + output_text = "unwanted, running" + return input_text, output_text + + def test_full_tokenizer(self): + tokenizer = self.tokenizer_class(self.vocab_file) + + tokens = tokenizer.tokenize("UNwant\u00E9d,running") + self.assertListEqual(tokens, ["un", "##want", "##ed", ",", "runn", "##ing"]) + self.assertListEqual(tokenizer.convert_tokens_to_ids(tokens), [7, 4, 5, 10, 8, 9]) + + def test_token_type_ids(self): + tokenizers = self.get_tokenizers(do_lower_case=False) + for tokenizer in tokenizers: + inputs = tokenizer("UNwant\u00E9d,running") + sentence_len = len(inputs["input_ids"]) - 1 + self.assertListEqual(inputs["token_type_ids"], [2] + [0] * sentence_len) + + inputs = tokenizer("UNwant\u00E9d,running", "UNwant\u00E9d,running") + self.assertListEqual(inputs["token_type_ids"], [2] + [0] * sentence_len + [1] * sentence_len) diff --git a/utils/check_repo.py b/utils/check_repo.py index 9a3154ec31..201db81966 100644 --- a/utils/check_repo.py +++ b/utils/check_repo.py @@ -141,18 +141,20 @@ def get_model_doc_files(): # for the all_model_classes variable. def find_tested_models(test_file): """ Parse the content of test_file to detect what's in all_model_classes""" + # This is a bit hacky but I didn't find a way to import the test_file as a module and read inside the class with open(os.path.join(PATH_TO_TESTS, test_file)) as f: content = f.read() - all_models = re.search(r"all_model_classes\s+=\s+\(\s*\(([^\)]*)\)", content) + all_models = re.findall(r"all_model_classes\s+=\s+\(\s*\(([^\)]*)\)", content) # Check with one less parenthesis - if all_models is None: - all_models = re.search(r"all_model_classes\s+=\s+\(([^\)]*)\)", content) - if all_models is not None: + if len(all_models) == 0: + all_models = re.findall(r"all_model_classes\s+=\s+\(([^\)]*)\)", content) + if len(all_models) > 0: model_tested = [] - for line in all_models.groups()[0].split(","): - name = line.strip() - if len(name) > 0: - model_tested.append(name) + for entry in all_models: + for line in entry.split(","): + name = line.strip() + if len(name) > 0: + model_tested.append(name) return model_tested