SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

improve device usage

Browse Source
This commit is contained in:
Rémi Louf
2019-12-06 15:45:09 +01:00
committed by Julien Chaumond
parent c0707a85d2
commit 2a64107e44
4 changed files with 21 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
examples/summarization/README.md
View File

@@ -29,7 +29,7 @@ And move all the stories to the same folder. We will refer as `$DATA_PATH` the p
python run_summarization.py \
--documents_dir $DATA_PATH \
--summaries_output_dir $SUMMARIES_PATH \ # optional
--visible_gpus 0,1,2 \
--to_cpu false \
--batch_size 4 \
--min_length 50 \
--max_length 200 \
@@ -39,7 +39,7 @@ python run_summarization.py \
--compute_rouge true
```
The ROUGE scores will be displayed in the console at the end of evaluation and written in a `rouge_scores.txt` file.
The scripts executes on GPU if one is available and if `to_cpu` is not set to `true`. Inference on multiple GPUs is not suported yet. The ROUGE scores will be displayed in the console at the end of evaluation and written in a `rouge_scores.txt` file. The script takes 30 hours to compute with a single Tesla V100 GPU and a batch size of 10 (300,000 texts to summarize).
## Summarize any text
@@ -49,7 +49,7 @@ Put the documents that you would like to summarize in a folder (the path to whic
python run_summarization.py \
--documents_dir $DATA_PATH \
--summaries_output_dir $SUMMARIES_PATH \ # optional
--visible_gpus 0,1,2 \
--to_cpu false \
--batch_size 4 \
--min_length 50 \
--max_length 200 \
@@ -58,4 +58,4 @@ python run_summarization.py \
--block_trigram true \
```
If you want to compute ROUGE on another dataset you will need to tweak the stories/summaries import in `utils_summarization.py`
You may want to play around with `min_length`, `max_length` and `alpha` to suit your use case. If you want to compute ROUGE on another dataset you will need to tweak the stories/summaries import in `utils_summarization.py` and tell it where to fetch the reference summaries.