SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

fix bug in group_texts function, that was inserting short batches (#23429)

Browse Source 
* fix bug in group_texts function, that was inserting short batches

* fully exclude short batches and return empty dict instead

* fix style
This commit is contained in:
Boda Sadallah
2023-05-18 21:22:30 +03:00
committed by GitHub
parent b7b81d9344
commit a7920065f2
5 changed files with 15 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
examples/pytorch/language-modeling/run_plm.py
View File

@@ -450,10 +450,9 @@ def main():
# Concatenate all texts.
concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}
total_length = len(concatenated_examples[list(examples.keys())[0]])
# We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
# customize this part to your needs.
if total_length >= max_seq_length:
total_length = (total_length // max_seq_length) * max_seq_length
# We drop the small remainder, and if the total_length < max_seq_length we exclude this batch and return an empty dict.
# We could add padding if the model supported it instead of this drop, you can customize this part to your needs.
total_length = (total_length // max_seq_length) * max_seq_length
# Split by chunks of max_len.
result = {
k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]