SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

mbert reproducibility results

Browse Source
This commit is contained in:
VictorSanh
2019-10-29 18:53:45 -04:00
committed by Lysandre Debut
parent c4336ecbbd
commit 84a0b522cf
1 changed files with 8 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
examples/README.md
View File

@@ -604,13 +604,13 @@ python run_summarization_finetuning.py \
## XNLI ## XNLI
Based on the script [`run_xnli.py`](TODO). Based on the script [`run_xnli.py`](https://github.com/huggingface/transformers/blob/master/examples/run_xnli.py).
[XNLI](https://www.nyu.edu/projects/bowman/xnli/) is crowd-sourced dataset based on [MultiNLI](http://www.nyu.edu/projects/bowman/multinli/). It is an evaluation benchmark for cross-lingual text representations. Pairs of text are labeled with textual entailment annotations for 15 different languages (including both high-ressource language such as English and low-ressource languages such as Swahili). [XNLI](https://www.nyu.edu/projects/bowman/xnli/) is crowd-sourced dataset based on [MultiNLI](http://www.nyu.edu/projects/bowman/multinli/). It is an evaluation benchmark for cross-lingual text representations. Pairs of text are labeled with textual entailment annotations for 15 different languages (including both high-ressource language such as English and low-ressource languages such as Swahili).
#### Fine-tuning on XNLI #### Fine-tuning on XNLI
This example code fine-tunes mBERT (multi-lingual BERT) on the XNLI dataset. It runs in TODO min This example code fine-tunes mBERT (multi-lingual BERT) on the XNLI dataset. It runs in 106 mins
on a single tesla V100 16GB. The data for XNLI can be downloaded with the following links and should be both saved (and un-zipped) in a on a single tesla V100 16GB. The data for XNLI can be downloaded with the following links and should be both saved (and un-zipped) in a
`$XNLI_DIR` directory. `$XNLI_DIR` directory.
@@ -623,20 +623,21 @@ export XNLI_DIR=/path/to/XNLI
python run_xnli.py \ python run_xnli.py \
--model_type bert \ --model_type bert \
--model_name_or_path bert-base-multilingual-cased \ --model_name_or_path bert-base-multilingual-cased \
--language en \ --language es \
--train_language en \ --train_language en \
--do_train \ --do_train \
--do_eval \ --do_eval \
--data_dir $SQUAD_DIR \ --data_dir $XNLI_DIR \
--per_gpu_train_batch_size 32 \ --per_gpu_train_batch_size 32 \
--learning_rate 5e-5 \ --learning_rate 5e-5 \
--num_train_epochs 2.0 \ --num_train_epochs 2.0 \
--max_seq_length 128 \ --max_seq_length 128 \
--output_dir /tmp/debug_xnli/ --output_dir /tmp/debug_xnli/ \
--save_steps -1
``` ```
Training with the previously defined hyper-parameters yields the following results: Training with the previously defined hyper-parameters yields the following results on the dev set:
```bash ```bash
TODO acc = 0.738152610441767
``` ```