Files
HuggingFace_transformer/examples
jeonsworld 60005f464d Update pregenerate_training_data.py
If the value of rand_end is returned from the randint function, the value of sampled_doc_index that matches current_idx is returned from searchsorted.

example:
cumsum_max = {int64} 30
doc_cumsum = {ndarray} [ 5  7 11 19 30]
doc_lengths = {list} <class 'list'>: [5, 2, 4, 8, 11]
if current_idx  = 1,
rand_start = 7
rand_end = 35
sentence_index = randint(7, 35) % cumsum_max
if randint return 35, sentence_index becomes 5.
if sentence_index is 5, np.searchsorted returns 1 equal to current_index.
2019-03-30 14:50:17 +09:00
..
2019-03-06 10:05:21 +01:00
2019-03-08 11:59:08 -05:00
2019-03-14 17:32:15 +08:00
2019-03-21 20:59:33 +08:00
2019-03-06 10:05:21 +01:00