@@ -54,7 +54,7 @@ Then pass your sentence to the tokenizer:
|
||||
'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
|
||||
```
|
||||
|
||||
The tokenizer returns a dictionary with three important itmes:
|
||||
The tokenizer returns a dictionary with three important items:
|
||||
|
||||
* [input_ids](glossary#input-ids) are the indices corresponding to each token in the sentence.
|
||||
* [attention_mask](glossary#attention-mask) indicates whether a token should be attended to or not.
|
||||
|
||||
Reference in New Issue
Block a user