SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Fix to prevent crashing on assert len(tokens_b)>=1

Browse Source
This commit is contained in:
searchivarius
2019-09-09 19:58:08 -04:00
parent a95ced6260
commit eab980fd68
1 changed files with 2 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
examples/lm_finetuning/pregenerate_training_data.py
View File

@@ -329,7 +329,8 @@ def main():
doc = [] doc = []
else: else:
tokens = tokenizer.tokenize(line) tokens = tokenizer.tokenize(line)
doc.append(tokens) if tokens:
doc.append(tokens)
if doc: if doc:
docs.add_document(doc) # If the last doc didn't end on a newline, make sure it still gets added docs.add_document(doc) # If the last doc didn't end on a newline, make sure it still gets added
if len(docs) <= 1: if len(docs) <= 1: