SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Removing a couple of other old unnecessary comments

Browse Source
This commit is contained in:
Matthew Carrigan
2019-03-21 14:09:57 +00:00
parent 8a861048dd
commit 8733ffcb5e
1 changed files with 0 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
examples/lm_finetuning/pregenerate_training_data.py
View File

@@ -1,7 +1,3 @@
# Step 1: Slurp the dataset up, tokenize each sentence, and store as docs -> sentences -> tokens
# Step 2: Walk over the dataset, using the Google BERT logic to concatenate sentences into training examples
# Step 3: Write out the examples, possibly as Torch tensors?
from argparse import ArgumentParser from argparse import ArgumentParser
from pathlib import Path from pathlib import Path
from tqdm import tqdm, trange from tqdm import tqdm, trange