@@ -193,8 +193,8 @@ Pass your text to the tokenizer:
|
|||||||
|
|
||||||
The tokenizer returns a dictionary containing:
|
The tokenizer returns a dictionary containing:
|
||||||
|
|
||||||
* [input_ids](./glossary#input-ids): numerical representions of your tokens.
|
* [input_ids](./glossary#input-ids): numerical representations of your tokens.
|
||||||
* [atttention_mask](.glossary#attention-mask): indicates which tokens should be attended to.
|
* [attention_mask](.glossary#attention-mask): indicates which tokens should be attended to.
|
||||||
|
|
||||||
A tokenizer can also accept a list of inputs, and pad and truncate the text to return a batch with uniform length:
|
A tokenizer can also accept a list of inputs, and pad and truncate the text to return a batch with uniform length:
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user