Add FastTokenizer to REALM (#15211)

* Remove BertTokenizer abstraction * Add FastTokenizer to REALM * Fix config archive map * Fix copies * Update realm.mdx * Apply suggestions from code review
2022-01-19 22:19:36 +08:00
parent 021b52e7a8
commit 841d979190
10 changed files with 824 additions and 46 deletions
--- a/docs/source/model_doc/realm.mdx
+++ b/docs/source/model_doc/realm.mdx
@@ -49,6 +49,11 @@ This model was contributed by [qqaatw](https://huggingface.co/qqaatw). The origi
    - save_vocabulary
    - batch_encode_candidates

+## RealmTokenizerFast
+
+[[autodoc]] RealmTokenizerFast
+    - batch_encode_candidates
+
 ## RealmRetriever

 [[autodoc]] RealmRetriever