SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Changed processor documentation architecture. Added documentation for GLUE

Browse Source
This commit is contained in:
LysandreJik
2019-09-25 08:30:07 -04:00
committed by Lysandre Debut
parent c4ac7a76db
commit ad4a393e2e
2 changed files with 55 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

62
docs/source/main_classes/processors.rst
View File

@@ -4,42 +4,46 @@ Processors
This library includes processors for several traditional tasks. These processors can be used to process a dataset into
examples that can be fed to a model.
``GLUE``
Processors
~~~~~~~~~~~~~~~~~~~~~
`General Language Understanding Evaluation (GLUE)<https://gluebenchmark.com/>`__ is a benchmark that evaluates
All processors follow the same architecture which is that of the
:class:`~pytorch_transformers.data.processors.utils.DataProcessor`. The processor returns a list
of :class:`~pytorch_transformers.data.processors.utils.InputExample`.
.. autoclass:: pytorch_transformers.data.processors.utils.DataProcessor
:members:
.. autoclass:: pytorch_transformers.data.processors.utils.InputExample
:members:
GLUE
~~~~~~~~~~~~~~~~~~~~~
`General Language Understanding Evaluation (GLUE) <https://gluebenchmark.com/>`__ is a benchmark that evaluates
the performance of models across a diverse set of existing NLU tasks. It was released together with the paper
`GLUE: A multi-task benchmark and analysis platform for natural language understanding<https://openreview.net/pdf?id=rJ4km2R5t7>`__
`GLUE: A multi-task benchmark and analysis platform for natural language understanding <https://openreview.net/pdf?id=rJ4km2R5t7>`__
This library hosts a total of 10 processors for the following tasks: MRPC, MNLI, MNLI (mismatched),
CoLA, SST2, STSB, QQP, QNLI, RTE and WNLI.
.. autoclass:: pytorch_transformers.data.processors.glue.MrpcProcessor
:members:
Those processors are:
- :class:`~pytorch_transformers.data.processors.utils.MrpcProcessor`
- :class:`~pytorch_transformers.data.processors.utils.MnliProcessor`
- :class:`~pytorch_transformers.data.processors.utils.MnliMismatchedProcessor`
- :class:`~pytorch_transformers.data.processors.utils.Sst2Processor`
- :class:`~pytorch_transformers.data.processors.utils.StsbProcessor`
- :class:`~pytorch_transformers.data.processors.utils.QqpProcessor`
- :class:`~pytorch_transformers.data.processors.utils.QnliProcessor`
- :class:`~pytorch_transformers.data.processors.utils.RteProcessor`
- :class:`~pytorch_transformers.data.processors.utils.WnliProcessor`
.. autoclass:: pytorch_transformers.data.processors.glue.MnliProcessor
:members:
Additionally, the following method can be used to load values from a data file and convert them to a list of
:class:`~pytorch_transformers.data.processors.utils.InputExample`.
.. autoclass:: pytorch_transformers.data.processors.glue.MnliMismatchedProcessor
:members:
.. automethod:: pytorch_transformers.data.processors.glue.glue_convert_examples_to_features
.. autoclass:: pytorch_transformers.data.processors.glue.ColaProcessor
:members:
.. autoclass:: pytorch_transformers.data.processors.glue.Sst2Processor
:members:
.. autoclass:: pytorch_transformers.data.processors.glue.StsbProcessor
:members:
.. autoclass:: pytorch_transformers.data.processors.glue.QqpProcessor
:members:
.. autoclass:: pytorch_transformers.data.processors.glue.QnliProcessor
:members:
.. autoclass:: pytorch_transformers.data.processors.glue.RteProcessor
:members:
.. autoclass:: pytorch_transformers.data.processors.glue.WnliProcessor
:members:
Example usage
^^^^^^^^^^^^^^^^^^^^^^^^^