[tokenizers] Updates data processors, docstring, examples and model cards to the new API (#5308)
* remove references to old API in docstring - update data processors * style * fix tests - better type checking error messages * better type checking * include awesome fix by @LysandreJik for #5310 * updated doc and examples
This commit is contained in:
@@ -278,7 +278,7 @@ PT_MULTIPLE_CHOICE_SAMPLE = r"""
|
||||
>>> choice1 = "It is eaten while held in the hand."
|
||||
>>> labels = torch.tensor(0).unsqueeze(0) # choice0 is correct (according to Wikipedia ;)), batch size 1
|
||||
|
||||
>>> encoding = tokenizer([[prompt, prompt], [choice0, choice1]], return_tensors='pt', pad_to_max_length=True)
|
||||
>>> encoding = tokenizer([[prompt, prompt], [choice0, choice1]], return_tensors='pt', padding=True)
|
||||
>>> outputs = model(**{{k: v.unsqueeze(0) for k,v in encoding.items()}}, labels=labels) # batch size is 1
|
||||
|
||||
>>> # the linear classifier still needs to be trained
|
||||
@@ -391,7 +391,7 @@ TF_MULTIPLE_CHOICE_SAMPLE = r"""
|
||||
>>> choice0 = "It is eaten with a fork and a knife."
|
||||
>>> choice1 = "It is eaten while held in the hand."
|
||||
|
||||
>>> encoding = tokenizer([[prompt, prompt], [choice0, choice1]], return_tensors='tf', pad_to_max_length=True)
|
||||
>>> encoding = tokenizer([[prompt, prompt], [choice0, choice1]], return_tensors='tf', padding=True)
|
||||
>>> inputs = {{k: tf.expand_dims(v, 0) for k, v in encoding.items()}}
|
||||
>>> outputs = model(inputs) # batch size is 1
|
||||
|
||||
|
||||
Reference in New Issue
Block a user