SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Reduced memory usage for pregenerating the data a lot by writing it

Browse Source 
out on the fly without shuffling - the Sampler in the finetuning script
will shuffle for us.
This commit is contained in:
Matthew Carrigan
2019-03-21 14:05:52 +00:00
parent 0ae59e662d
commit a8a577ba93
1 changed files with 0 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
examples/lm_finetuning/finetune_on_pregenerated.py
View File

@@ -74,8 +74,6 @@ class PregeneratedDataset(Dataset):
with data_file.open() as f: with data_file.open() as f:
for i, line in enumerate(tqdm(f, total=num_samples, desc="Training examples")): for i, line in enumerate(tqdm(f, total=num_samples, desc="Training examples")):
line = line.strip() line = line.strip()
if not line:
continue # Skip trailing blank lines etc.
example = json.loads(line) example = json.loads(line)
features = convert_example_to_features(example, tokenizer, seq_len) features = convert_example_to_features(example, tokenizer, seq_len)
input_ids[i] = features.input_ids input_ids[i] = features.input_ids