big doc update [WIP]

2019-08-04 12:14:57 +02:00
parent bfbe52ec39
commit 009273dbdd
19 changed files with 189 additions and 60 deletions
--- a/docs/source/main_classes/tokenizer.rst
+++ b/docs/source/main_classes/tokenizer.rst
@@ -0,0 +1,8 @@
+Tokenizer
+----------------------------------------------------
+
+``PreTrainedTokenizer``
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: pytorch_transformers.PreTrainedTokenizer
+    :members: