SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Added missing code in exemplary notebook - custom datasets fine-tuning (#15300)

Browse Source 
* Added missing code in exemplary notebook - custom datasets fine-tuning

Added missing code in tokenize_and_align_labels function in the exemplary notebook on custom datasets - token classification.
The missing code concerns adding labels for all but first token in a single word.
The added code was taken directly from huggingface official example - this [colab notebook](https://github.com/huggingface/notebooks/blob/master/transformers_doc/custom_datasets.ipynb).

* Changes requested in the review - keep the code as simple as possible
This commit is contained in:
Maciej Pawłowski
2022-01-25 23:26:17 +01:00
committed by GitHub
parent 0501beb846
commit e79a0faeae
1 changed files with 3 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/source/custom_datasets.mdx
View File

@@ -326,7 +326,9 @@ def tokenize_and_align_labels(examples):
label_ids.append(-100) label_ids.append(-100)
elif word_idx != previous_word_idx: # Only label the first token of a given word. elif word_idx != previous_word_idx: # Only label the first token of a given word.
label_ids.append(label[word_idx]) label_ids.append(label[word_idx])
else:
label_ids.append(-100)
previous_word_idx = word_idx
labels.append(label_ids) labels.append(label_ids)
tokenized_inputs["labels"] = labels tokenized_inputs["labels"] = labels