SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Tokenizers and Config classes are referenced.

Browse Source
This commit is contained in:
LysandreJik
2019-07-05 17:44:59 -04:00
parent df759114c9
commit 64fd986376
6 changed files with 43 additions and 95 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
docs/source/model_doc/transformerxl.rst
View File

@@ -2,14 +2,18 @@ Transformer XL
----------------------------------------------------
``TransfoXLConfig``
~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: pytorch_pretrained_bert.TransfoXLConfig
:members:
``TransfoXLTokenizer``
~~~~~~~~~~~~~~~~~~~~~~~~~~
``TransfoXLTokenizer`` perform word tokenization. This tokenizer can be used for adaptive softmax and has utilities for counting tokens in a corpus to create a vocabulary ordered by toekn frequency (for adaptive softmax). See the adaptive softmax paper (\ `Efficient softmax approximation for GPUs <http://arxiv.org/abs/1609.04309>`_\ ) for more details.
The API is similar to the API of ``BertTokenizer`` (see above).
Please refer to the doc strings and code in `\ ``tokenization_transfo_xl.py`` <./pytorch_pretrained_bert/tokenization_transfo_xl.py>`_ for the details of these additional methods in ``TransfoXLTokenizer``.
.. autoclass:: pytorch_pretrained_bert.TransfoXLTokenizer
:members:
12. ``TransfoXLModel``