SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Convert tokenizer outputs for Keras in doc example (#20732)

Browse Source 
* Convert tokenizer outputs for Keras in doc example

* Das deutsche Beispiel auch korrigieren
This commit is contained in:
Matt
2022-12-12 16:14:04 +00:00
committed by GitHub
parent 0ba94aceb6
commit c1b9a11dd4
2 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/source/de/training.mdx
View File

@@ -185,6 +185,8 @@ from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased") tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
tokenized_data = tokenizer(dataset["text"], return_tensors="np", padding=True) tokenized_data = tokenizer(dataset["text"], return_tensors="np", padding=True)
# Tokenizer returns a BatchEncoding, but we convert that to a dict for Keras
tokenized_data = dict(tokenized_data)
labels = np.array(dataset["label"]) # Label is already an array of 0 and 1 labels = np.array(dataset["label"]) # Label is already an array of 0 and 1
``` ```

2
docs/source/en/training.mdx
View File

@@ -185,6 +185,8 @@ from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased") tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
tokenized_data = tokenizer(dataset["text"], return_tensors="np", padding=True) tokenized_data = tokenizer(dataset["text"], return_tensors="np", padding=True)
# Tokenizer returns a BatchEncoding, but we convert that to a dict for Keras
tokenized_data = dict(tokenized_data)
labels = np.array(dataset["label"]) # Label is already an array of 0 and 1 labels = np.array(dataset["label"]) # Label is already an array of 0 and 1
``` ```