updating docs - adding few tests to tokenizers

docs/source/main_classes/configuration.rst

@@ -1,7 +1,7 @@
 Configuration
 ----------------------------------------------------
 
-We provide a base class, ``PretrainedConfig``, which can load a pretrained instance either from a local file or directory or from a pretrained model configuration provided by the library (downloaded from HuggingFace AWS S3 repository).
+The base class ``PretrainedConfig`` implements the common methods for loading/saving a configuration either from a local file or directory, or from a pretrained model configuration provided by the library (downloaded from HuggingFace's AWS S3 repository).
 
 ``PretrainedConfig``
 ~~~~~~~~~~~~~~~~~~~~~

docs/source/main_classes/model.rst

@@ -1,6 +1,13 @@
 Models
 ----------------------------------------------------
 
+The base class ``PreTrainedModel`` implements the common methods for loading/saving a model either from a local file or directory, or from a pretrained model configuration provided by the library (downloaded from HuggingFace's AWS S3 repository).
+
+``PreTrainedModel`` also implements a few methods which are common among all the models to:
+
+- resize the input token embeddings when new tokens are added to the vocabulary
+- prune the attention heads of the model.
+
 ``PreTrainedModel``
 ~~~~~~~~~~~~~~~~~~~~~
 

docs/source/main_classes/optimizer_schedules.rst

@@ -1,6 +1,11 @@
 Optimizer
 ----------------------------------------------------
 
+The ``.optimization`` module provides:
+
+- an optimizer with weight decay fixed that can be used to fine-tuned models, and
+- several schedules in the form of schedule objects that inherit from ``_LRSchedule``:
+
 ``AdamW``
 ~~~~~~~~~~~~~~~~
 
@@ -10,17 +15,41 @@ Optimizer
 Schedules
 ----------------------------------------------------
 
+Learning Rate Schedules
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
 .. autoclass:: pytorch_transformers.ConstantLRSchedule
     :members:
 
+
 .. autoclass:: pytorch_transformers.WarmupConstantSchedule
     :members:
 
+.. image:: /imgs/warmup_constant_schedule.png
+    :target: /imgs/warmup_constant_schedule.png
+    :alt:
+
+
 .. autoclass:: pytorch_transformers.WarmupCosineSchedule
     :members:
 
+.. image:: /imgs/warmup_cosine_schedule.png
+    :target: /imgs/warmup_cosine_schedule.png
+    :alt:
+
+
 .. autoclass:: pytorch_transformers.WarmupCosineWithHardRestartsSchedule
     :members:
 
+.. image:: /imgs/warmup_cosine_hard_restarts_schedule.png
+    :target: /imgs/warmup_cosine_hard_restarts_schedule.png
+    :alt:
+
+
+
 .. autoclass:: pytorch_transformers.WarmupLinearSchedule
     :members:
+
+.. image:: /imgs/warmup_linear_schedule.png
+    :target: /imgs/warmup_linear_schedule.png
+    :alt:

docs/source/main_classes/tokenizer.rst

@@ -1,6 +1,14 @@
 Tokenizer
 ----------------------------------------------------
 
+The base class ``PreTrainedTokenizer`` implements the common methods for loading/saving a tokenizer either from a local file or directory, or from a pretrained tokenizer provided by the library (downloaded from HuggingFace's AWS S3 repository).
+
+``PreTrainedTokenizer`` is the main entry point into tokenizers as it also implements the main methods for using all the tokenizers:
+
+- tokenizing, converting tokens to ids and back and encoding/decoding,
+- adding new tokens to the vocabulary in a way that is independant of the underlying structure (BPE, SentencePiece...),
+- managing special tokens (adding them, assigning them to roles, making sure they are not split during tokenization)
+
 ``PreTrainedTokenizer``
 ~~~~~~~~~~~~~~~~~~~~~~~~