From 34334662df1a01567bcbf6859c2b03ac637147d4 Mon Sep 17 00:00:00 2001 From: Hayden Housen Date: Wed, 16 Dec 2020 11:06:14 -0500 Subject: [PATCH] Add message to documentation that longformer doesn't support token_type_ids (#9152) * Add message to documentation that longformer doesn't support token_type_ids * Format changes --- docs/source/model_doc/longformer.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/source/model_doc/longformer.rst b/docs/source/model_doc/longformer.rst index 9640851d50..e9c5b5054c 100644 --- a/docs/source/model_doc/longformer.rst +++ b/docs/source/model_doc/longformer.rst @@ -34,6 +34,12 @@ contrast to most prior work, we also pretrain Longformer and finetune it on a va pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA.* +Tips: + +- Since the Longformer is based on RoBERTa, it doesn't have :obj:`token_type_ids`. You don't need to indicate which + token belongs to which segment. Just separate your segments with the separation token :obj:`tokenizer.sep_token` (or + :obj:``). + The Authors' code can be found `here `__. Longformer Self Attention