Enable doc in Spanish (#16518)
* Reorganize doc for multilingual support * Fix style * Style * Toc trees * Adapt templates
This commit is contained in:
Sylvain Gugger
46
docs/source/en/internal/file_utils.mdx
Normal file
46
docs/source/en/internal/file_utils.mdx
Normal file
@@ -0,0 +1,46 @@
|
||||
<!--Copyright 2021 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# General Utilities
|
||||
|
||||
This page lists all of Transformers general utility functions that are found in the file `utils.py`.
|
||||
|
||||
Most of those are only useful if you are studying the general code in the library.
|
||||
|
||||
|
||||
## Enums and namedtuples
|
||||
|
||||
[[autodoc]] utils.ExplicitEnum
|
||||
|
||||
[[autodoc]] utils.PaddingStrategy
|
||||
|
||||
[[autodoc]] utils.TensorType
|
||||
|
||||
## Special Decorators
|
||||
|
||||
[[autodoc]] utils.add_start_docstrings
|
||||
|
||||
[[autodoc]] utils.add_start_docstrings_to_model_forward
|
||||
|
||||
[[autodoc]] utils.add_end_docstrings
|
||||
|
||||
[[autodoc]] utils.add_code_sample_docstrings
|
||||
|
||||
[[autodoc]] utils.replace_return_docstrings
|
||||
|
||||
## Special Properties
|
||||
|
||||
[[autodoc]] utils.cached_property
|
||||
|
||||
## Other Utilities
|
||||
|
||||
[[autodoc]] utils._LazyModule
|
||||
254
docs/source/en/internal/generation_utils.mdx
Normal file
254
docs/source/en/internal/generation_utils.mdx
Normal file
@@ -0,0 +1,254 @@
|
||||
<!--Copyright 2020 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# Utilities for Generation
|
||||
|
||||
This page lists all the utility functions used by [`~generation_utils.GenerationMixin.generate`],
|
||||
[`~generation_utils.GenerationMixin.greedy_search`],
|
||||
[`~generation_utils.GenerationMixin.sample`],
|
||||
[`~generation_utils.GenerationMixin.beam_search`],
|
||||
[`~generation_utils.GenerationMixin.beam_sample`],
|
||||
[`~generation_utils.GenerationMixin.group_beam_search`], and
|
||||
[`~generation_utils.GenerationMixin.constrained_beam_search`].
|
||||
|
||||
Most of those are only useful if you are studying the code of the generate methods in the library.
|
||||
|
||||
## Generate Outputs
|
||||
|
||||
The output of [`~generation_utils.GenerationMixin.generate`] is an instance of a subclass of
|
||||
[`~utils.ModelOutput`]. This output is a data structure containing all the information returned
|
||||
by [`~generation_utils.GenerationMixin.generate`], but that can also be used as tuple or dictionary.
|
||||
|
||||
Here's an example:
|
||||
|
||||
```python
|
||||
from transformers import GPT2Tokenizer, GPT2LMHeadModel
|
||||
|
||||
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
|
||||
model = GPT2LMHeadModel.from_pretrained("gpt2")
|
||||
|
||||
inputs = tokenizer("Hello, my dog is cute and ", return_tensors="pt")
|
||||
generation_output = model.generate(**inputs, return_dict_in_generate=True, output_scores=True)
|
||||
```
|
||||
|
||||
The `generation_output` object is a [`~generation_utils.GreedySearchDecoderOnlyOutput`], as we can
|
||||
see in the documentation of that class below, it means it has the following attributes:
|
||||
|
||||
- `sequences`: the generated sequences of tokens
|
||||
- `scores` (optional): the prediction scores of the language modelling head, for each generation step
|
||||
- `hidden_states` (optional): the hidden states of the model, for each generation step
|
||||
- `attentions` (optional): the attention weights of the model, for each generation step
|
||||
|
||||
Here we have the `scores` since we passed along `output_scores=True`, but we don't have `hidden_states` and
|
||||
`attentions` because we didn't pass `output_hidden_states=True` or `output_attentions=True`.
|
||||
|
||||
You can access each attribute as you would usually do, and if that attribute has not been returned by the model, you
|
||||
will get `None`. Here for instance `generation_output.scores` are all the generated prediction scores of the
|
||||
language modeling head, and `generation_output.attentions` is `None`.
|
||||
|
||||
When using our `generation_output` object as a tuple, it only keeps the attributes that don't have `None` values.
|
||||
Here, for instance, it has two elements, `loss` then `logits`, so
|
||||
|
||||
```python
|
||||
generation_output[:2]
|
||||
```
|
||||
|
||||
will return the tuple `(generation_output.sequences, generation_output.scores)` for instance.
|
||||
|
||||
When using our `generation_output` object as a dictionary, it only keeps the attributes that don't have `None`
|
||||
values. Here, for instance, it has two keys that are `sequences` and `scores`.
|
||||
|
||||
We document here all output types.
|
||||
|
||||
|
||||
### GreedySearchOutput
|
||||
|
||||
[[autodoc]] generation_utils.GreedySearchDecoderOnlyOutput
|
||||
|
||||
[[autodoc]] generation_utils.GreedySearchEncoderDecoderOutput
|
||||
|
||||
[[autodoc]] generation_flax_utils.FlaxGreedySearchOutput
|
||||
|
||||
### SampleOutput
|
||||
|
||||
[[autodoc]] generation_utils.SampleDecoderOnlyOutput
|
||||
|
||||
[[autodoc]] generation_utils.SampleEncoderDecoderOutput
|
||||
|
||||
[[autodoc]] generation_flax_utils.FlaxSampleOutput
|
||||
|
||||
### BeamSearchOutput
|
||||
|
||||
[[autodoc]] generation_utils.BeamSearchDecoderOnlyOutput
|
||||
|
||||
[[autodoc]] generation_utils.BeamSearchEncoderDecoderOutput
|
||||
|
||||
### BeamSampleOutput
|
||||
|
||||
[[autodoc]] generation_utils.BeamSampleDecoderOnlyOutput
|
||||
|
||||
[[autodoc]] generation_utils.BeamSampleEncoderDecoderOutput
|
||||
|
||||
## LogitsProcessor
|
||||
|
||||
A [`LogitsProcessor`] can be used to modify the prediction scores of a language model head for
|
||||
generation.
|
||||
|
||||
[[autodoc]] LogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] LogitsProcessorList
|
||||
- __call__
|
||||
|
||||
[[autodoc]] LogitsWarper
|
||||
- __call__
|
||||
|
||||
[[autodoc]] MinLengthLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TemperatureLogitsWarper
|
||||
- __call__
|
||||
|
||||
[[autodoc]] RepetitionPenaltyLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TopPLogitsWarper
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TopKLogitsWarper
|
||||
- __call__
|
||||
|
||||
[[autodoc]] NoRepeatNGramLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] NoBadWordsLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] PrefixConstrainedLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] HammingDiversityLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] ForcedBOSTokenLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] ForcedEOSTokenLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] InfNanRemoveLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TFLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TFLogitsProcessorList
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TFLogitsWarper
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TFTemperatureLogitsWarper
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TFTopPLogitsWarper
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TFTopKLogitsWarper
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TFMinLengthLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TFNoBadWordsLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TFNoRepeatNGramLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] TFRepetitionPenaltyLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] FlaxLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] FlaxLogitsProcessorList
|
||||
- __call__
|
||||
|
||||
[[autodoc]] FlaxLogitsWarper
|
||||
- __call__
|
||||
|
||||
[[autodoc]] FlaxTemperatureLogitsWarper
|
||||
- __call__
|
||||
|
||||
[[autodoc]] FlaxTopPLogitsWarper
|
||||
- __call__
|
||||
|
||||
[[autodoc]] FlaxTopKLogitsWarper
|
||||
- __call__
|
||||
|
||||
[[autodoc]] FlaxForcedBOSTokenLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] FlaxForcedEOSTokenLogitsProcessor
|
||||
- __call__
|
||||
|
||||
[[autodoc]] FlaxMinLengthLogitsProcessor
|
||||
- __call__
|
||||
|
||||
## StoppingCriteria
|
||||
|
||||
A [`StoppingCriteria`] can be used to change when to stop generation (other than EOS token).
|
||||
|
||||
[[autodoc]] StoppingCriteria
|
||||
- __call__
|
||||
|
||||
[[autodoc]] StoppingCriteriaList
|
||||
- __call__
|
||||
|
||||
[[autodoc]] MaxLengthCriteria
|
||||
- __call__
|
||||
|
||||
[[autodoc]] MaxTimeCriteria
|
||||
- __call__
|
||||
|
||||
## Constraints
|
||||
|
||||
A [`Constraint`] can be used to force the generation to include specific tokens or sequences in the output.
|
||||
|
||||
[[autodoc]] Constraint
|
||||
|
||||
[[autodoc]] PhrasalConstraint
|
||||
|
||||
[[autodoc]] DisjunctiveConstraint
|
||||
|
||||
[[autodoc]] ConstraintListState
|
||||
|
||||
## BeamSearch
|
||||
|
||||
[[autodoc]] BeamScorer
|
||||
- process
|
||||
- finalize
|
||||
|
||||
[[autodoc]] BeamSearchScorer
|
||||
- process
|
||||
- finalize
|
||||
|
||||
[[autodoc]] ConstrainedBeamSearchScorer
|
||||
- process
|
||||
- finalize
|
||||
|
||||
## Utilities
|
||||
|
||||
[[autodoc]] top_k_top_p_filtering
|
||||
|
||||
[[autodoc]] tf_top_k_top_p_filtering
|
||||
82
docs/source/en/internal/modeling_utils.mdx
Normal file
82
docs/source/en/internal/modeling_utils.mdx
Normal file
@@ -0,0 +1,82 @@
|
||||
<!--Copyright 2020 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# Custom Layers and Utilities
|
||||
|
||||
This page lists all the custom layers used by the library, as well as the utility functions it provides for modeling.
|
||||
|
||||
Most of those are only useful if you are studying the code of the models in the library.
|
||||
|
||||
|
||||
## Pytorch custom modules
|
||||
|
||||
[[autodoc]] modeling_utils.Conv1D
|
||||
|
||||
[[autodoc]] modeling_utils.PoolerStartLogits
|
||||
- forward
|
||||
|
||||
[[autodoc]] modeling_utils.PoolerEndLogits
|
||||
- forward
|
||||
|
||||
[[autodoc]] modeling_utils.PoolerAnswerClass
|
||||
- forward
|
||||
|
||||
[[autodoc]] modeling_utils.SquadHeadOutput
|
||||
|
||||
[[autodoc]] modeling_utils.SQuADHead
|
||||
- forward
|
||||
|
||||
[[autodoc]] modeling_utils.SequenceSummary
|
||||
- forward
|
||||
|
||||
## PyTorch Helper Functions
|
||||
|
||||
[[autodoc]] apply_chunking_to_forward
|
||||
|
||||
[[autodoc]] modeling_utils.find_pruneable_heads_and_indices
|
||||
|
||||
[[autodoc]] modeling_utils.prune_layer
|
||||
|
||||
[[autodoc]] modeling_utils.prune_conv1d_layer
|
||||
|
||||
[[autodoc]] modeling_utils.prune_linear_layer
|
||||
|
||||
## TensorFlow custom layers
|
||||
|
||||
[[autodoc]] modeling_tf_utils.TFConv1D
|
||||
|
||||
[[autodoc]] modeling_tf_utils.TFSharedEmbeddings
|
||||
- call
|
||||
|
||||
[[autodoc]] modeling_tf_utils.TFSequenceSummary
|
||||
|
||||
## TensorFlow loss functions
|
||||
|
||||
[[autodoc]] modeling_tf_utils.TFCausalLanguageModelingLoss
|
||||
|
||||
[[autodoc]] modeling_tf_utils.TFMaskedLanguageModelingLoss
|
||||
|
||||
[[autodoc]] modeling_tf_utils.TFMultipleChoiceLoss
|
||||
|
||||
[[autodoc]] modeling_tf_utils.TFQuestionAnsweringLoss
|
||||
|
||||
[[autodoc]] modeling_tf_utils.TFSequenceClassificationLoss
|
||||
|
||||
[[autodoc]] modeling_tf_utils.TFTokenClassificationLoss
|
||||
|
||||
## TensorFlow Helper Functions
|
||||
|
||||
[[autodoc]] modeling_tf_utils.get_initializer
|
||||
|
||||
[[autodoc]] modeling_tf_utils.keras_serializable
|
||||
|
||||
[[autodoc]] modeling_tf_utils.shape_list
|
||||
40
docs/source/en/internal/pipelines_utils.mdx
Normal file
40
docs/source/en/internal/pipelines_utils.mdx
Normal file
@@ -0,0 +1,40 @@
|
||||
<!--Copyright 2020 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# Utilities for pipelines
|
||||
|
||||
This page lists all the utility functions the library provides for pipelines.
|
||||
|
||||
Most of those are only useful if you are studying the code of the models in the library.
|
||||
|
||||
|
||||
## Argument handling
|
||||
|
||||
[[autodoc]] pipelines.ArgumentHandler
|
||||
|
||||
[[autodoc]] pipelines.ZeroShotClassificationArgumentHandler
|
||||
|
||||
[[autodoc]] pipelines.QuestionAnsweringArgumentHandler
|
||||
|
||||
## Data format
|
||||
|
||||
[[autodoc]] pipelines.PipelineDataFormat
|
||||
|
||||
[[autodoc]] pipelines.CsvPipelineDataFormat
|
||||
|
||||
[[autodoc]] pipelines.JsonPipelineDataFormat
|
||||
|
||||
[[autodoc]] pipelines.PipedPipelineDataFormat
|
||||
|
||||
## Utilities
|
||||
|
||||
[[autodoc]] pipelines.PipelineException
|
||||
38
docs/source/en/internal/tokenization_utils.mdx
Normal file
38
docs/source/en/internal/tokenization_utils.mdx
Normal file
@@ -0,0 +1,38 @@
|
||||
<!--Copyright 2020 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# Utilities for Tokenizers
|
||||
|
||||
This page lists all the utility functions used by the tokenizers, mainly the class
|
||||
[`~tokenization_utils_base.PreTrainedTokenizerBase`] that implements the common methods between
|
||||
[`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`] and the mixin
|
||||
[`~tokenization_utils_base.SpecialTokensMixin`].
|
||||
|
||||
Most of those are only useful if you are studying the code of the tokenizers in the library.
|
||||
|
||||
## PreTrainedTokenizerBase
|
||||
|
||||
[[autodoc]] tokenization_utils_base.PreTrainedTokenizerBase
|
||||
- __call__
|
||||
- all
|
||||
|
||||
## SpecialTokensMixin
|
||||
|
||||
[[autodoc]] tokenization_utils_base.SpecialTokensMixin
|
||||
|
||||
## Enums and namedtuples
|
||||
|
||||
[[autodoc]] tokenization_utils_base.TruncationStrategy
|
||||
|
||||
[[autodoc]] tokenization_utils_base.CharSpan
|
||||
|
||||
[[autodoc]] tokenization_utils_base.TokenSpan
|
||||
43
docs/source/en/internal/trainer_utils.mdx
Normal file
43
docs/source/en/internal/trainer_utils.mdx
Normal file
@@ -0,0 +1,43 @@
|
||||
<!--Copyright 2020 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# Utilities for Trainer
|
||||
|
||||
This page lists all the utility functions used by [`Trainer`].
|
||||
|
||||
Most of those are only useful if you are studying the code of the Trainer in the library.
|
||||
|
||||
## Utilities
|
||||
|
||||
[[autodoc]] EvalPrediction
|
||||
|
||||
[[autodoc]] IntervalStrategy
|
||||
|
||||
[[autodoc]] set_seed
|
||||
|
||||
[[autodoc]] torch_distributed_zero_first
|
||||
|
||||
## Callbacks internals
|
||||
|
||||
[[autodoc]] trainer_callback.CallbackHandler
|
||||
|
||||
## Distributed Evaluation
|
||||
|
||||
[[autodoc]] trainer_pt_utils.DistributedTensorGatherer
|
||||
|
||||
## Distributed Evaluation
|
||||
|
||||
[[autodoc]] HfArgumentParser
|
||||
|
||||
## Debug Utilities
|
||||
|
||||
[[autodoc]] debug_utils.DebugUnderflowOverflow
|
||||
Reference in New Issue
Block a user