From 4114c9a75b8a0821b08cb786a6fb376646caa009 Mon Sep 17 00:00:00 2001 From: NielsRogge <48327001+NielsRogge@users.noreply.github.com> Date: Thu, 2 Sep 2021 09:46:05 +0200 Subject: [PATCH] Add tokenizer docs (#13373) --- docs/source/model_doc/layoutxlm.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/source/model_doc/layoutxlm.rst b/docs/source/model_doc/layoutxlm.rst index 82a6dc8a76..2635eb4cb0 100644 --- a/docs/source/model_doc/layoutxlm.rst +++ b/docs/source/model_doc/layoutxlm.rst @@ -40,6 +40,15 @@ One can directly plug in the weights of LayoutXLM into a LayoutLMv2 model, like model = LayoutLMv2Model.from_pretrained('microsoft/layoutxlm-base') +Note that LayoutXLM requires a different tokenizer, based on :class:`~transformers.XLMRobertaTokenizer`. You can +initialize it as follows: + +.. code-block:: + + from transformers import AutoTokenizer + + tokenizer = AutoTokenizer.from_pretrained('microsoft/layoutxlm-base') + As LayoutXLM's architecture is equivalent to that of LayoutLMv2, one can refer to :doc:`LayoutLMv2's documentation page ` for all tips, code examples and notebooks.