Add message to documentation that longformer doesn't support token_type_ids (#9152)

* Add message to documentation that longformer doesn't support token_type_ids

* Format changes
This commit is contained in:
Hayden Housen
2020-12-16 11:06:14 -05:00
committed by GitHub
parent 2f918defa8
commit 34334662df

View File

@@ -34,6 +34,12 @@ contrast to most prior work, we also pretrain Longformer and finetune it on a va
pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on
WikiHop and TriviaQA.*
Tips:
- Since the Longformer is based on RoBERTa, it doesn't have :obj:`token_type_ids`. You don't need to indicate which
token belongs to which segment. Just separate your segments with the separation token :obj:`tokenizer.sep_token` (or
:obj:`</s>`).
The Authors' code can be found `here <https://github.com/allenai/longformer>`__.
Longformer Self Attention