updating docs - adding few tests to tokenizers

This commit is contained in:
thomwolf
2019-08-04 22:42:55 +02:00
parent 009273dbdd
commit 00132b7a7a
10 changed files with 390 additions and 521 deletions

View File

@@ -1,6 +1,13 @@
Models
----------------------------------------------------
The base class ``PreTrainedModel`` implements the common methods for loading/saving a model either from a local file or directory, or from a pretrained model configuration provided by the library (downloaded from HuggingFace's AWS S3 repository).
``PreTrainedModel`` also implements a few methods which are common among all the models to:
- resize the input token embeddings when new tokens are added to the vocabulary
- prune the attention heads of the model.
``PreTrainedModel``
~~~~~~~~~~~~~~~~~~~~~