Convert rst files (#14888)
* Convert all tutorials and guides * Convert all remaining rst to mdx * Track and fix bad links
This commit is contained in:
154
docs/source/model_doc/xlnet.mdx
Normal file
154
docs/source/model_doc/xlnet.mdx
Normal file
@@ -0,0 +1,154 @@
|
||||
<!--Copyright 2020 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# XLNet
|
||||
|
||||
## Overview
|
||||
|
||||
The XLNet model was proposed in [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) by Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov,
|
||||
Quoc V. Le. XLnet is an extension of the Transformer-XL model pre-trained using an autoregressive method to learn
|
||||
bidirectional contexts by maximizing the expected likelihood over all permutations of the input sequence factorization
|
||||
order.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves
|
||||
better performance than pretraining approaches based on autoregressive language modeling. However, relying on
|
||||
corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a
|
||||
pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive
|
||||
pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all
|
||||
permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive
|
||||
formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into
|
||||
pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large
|
||||
margin, including question answering, natural language inference, sentiment analysis, and document ranking.*
|
||||
|
||||
Tips:
|
||||
|
||||
- The specific attention pattern can be controlled at training and test time using the `perm_mask` input.
|
||||
- Due to the difficulty of training a fully auto-regressive model over various factorization order, XLNet is pretrained
|
||||
using only a sub-set of the output tokens as target which are selected with the `target_mapping` input.
|
||||
- To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and
|
||||
`target_mapping` inputs to control the attention span and outputs (see examples in
|
||||
*examples/pytorch/text-generation/run_generation.py*)
|
||||
- XLNet is one of the few models that has no sequence length limit.
|
||||
|
||||
This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/zihangdai/xlnet/).
|
||||
|
||||
|
||||
## XLNetConfig
|
||||
|
||||
[[autodoc]] XLNetConfig
|
||||
|
||||
## XLNetTokenizer
|
||||
|
||||
[[autodoc]] XLNetTokenizer
|
||||
- build_inputs_with_special_tokens
|
||||
- get_special_tokens_mask
|
||||
- create_token_type_ids_from_sequences
|
||||
- save_vocabulary
|
||||
|
||||
## XLNetTokenizerFast
|
||||
|
||||
[[autodoc]] XLNetTokenizerFast
|
||||
|
||||
## XLNet specific outputs
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_xlnet.XLNetModelOutput
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_xlnet.XLNetLMHeadModelOutput
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_xlnet.XLNetForSequenceClassificationOutput
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_xlnet.XLNetForMultipleChoiceOutput
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_xlnet.XLNetForTokenClassificationOutput
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringSimpleOutput
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringOutput
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetModelOutput
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetLMHeadModelOutput
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForSequenceClassificationOutput
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForMultipleChoiceOutput
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForTokenClassificationOutput
|
||||
|
||||
[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForQuestionAnsweringSimpleOutput
|
||||
|
||||
## XLNetModel
|
||||
|
||||
[[autodoc]] XLNetModel
|
||||
- forward
|
||||
|
||||
## XLNetLMHeadModel
|
||||
|
||||
[[autodoc]] XLNetLMHeadModel
|
||||
- forward
|
||||
|
||||
## XLNetForSequenceClassification
|
||||
|
||||
[[autodoc]] XLNetForSequenceClassification
|
||||
- forward
|
||||
|
||||
## XLNetForMultipleChoice
|
||||
|
||||
[[autodoc]] XLNetForMultipleChoice
|
||||
- forward
|
||||
|
||||
## XLNetForTokenClassification
|
||||
|
||||
[[autodoc]] XLNetForTokenClassification
|
||||
- forward
|
||||
|
||||
## XLNetForQuestionAnsweringSimple
|
||||
|
||||
[[autodoc]] XLNetForQuestionAnsweringSimple
|
||||
- forward
|
||||
|
||||
## XLNetForQuestionAnswering
|
||||
|
||||
[[autodoc]] XLNetForQuestionAnswering
|
||||
- forward
|
||||
|
||||
## TFXLNetModel
|
||||
|
||||
[[autodoc]] TFXLNetModel
|
||||
- call
|
||||
|
||||
## TFXLNetLMHeadModel
|
||||
|
||||
[[autodoc]] TFXLNetLMHeadModel
|
||||
- call
|
||||
|
||||
## TFXLNetForSequenceClassification
|
||||
|
||||
[[autodoc]] TFXLNetForSequenceClassification
|
||||
- call
|
||||
|
||||
## TFLNetForMultipleChoice
|
||||
|
||||
[[autodoc]] TFXLNetForMultipleChoice
|
||||
- call
|
||||
|
||||
## TFXLNetForTokenClassification
|
||||
|
||||
[[autodoc]] TFXLNetForTokenClassification
|
||||
- call
|
||||
|
||||
## TFXLNetForQuestionAnsweringSimple
|
||||
|
||||
[[autodoc]] TFXLNetForQuestionAnsweringSimple
|
||||
- call
|
||||
Reference in New Issue
Block a user