From 6dc41d9f8eeefd6af3a57c26bbf33edcb71b3d99 Mon Sep 17 00:00:00 2001 From: Suraj Patil Date: Thu, 23 Sep 2021 02:48:13 +0530 Subject: [PATCH] add a note about tokenizer (#13696) --- docs/source/model_doc/gptj.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/source/model_doc/gptj.rst b/docs/source/model_doc/gptj.rst index 1c296d453a..dc2d804520 100644 --- a/docs/source/model_doc/gptj.rst +++ b/docs/source/model_doc/gptj.rst @@ -34,6 +34,11 @@ Tips: >>> model = GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B", torch_dtype=torch.float16) +- Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT-2 tokenizer. These extra + tokens are added for the sake of efficiency on TPUs. To avoid the mis-match between embedding matrix size and vocab + size, the tokenizer for [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B) contains 143 extra tokens + ``<|extratoken_1|>... <|extratoken_143|>``, so the ``vocab_size`` of tokenizer also becomes 50400. + Generation _______________________________________________________________________________________________________________________