SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

update README.md (#17657)

Browse Source 
- use CodeParrot scores of v1.1
- change evaluation command to use accelerate
This commit is contained in:
Loubna Ben Allal
2022-06-10 15:55:24 +02:00
committed by GitHub
parent c99ddcc441
commit 3114df41f4
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
examples/research_projects/codeparrot/README.md
View File

@@ -165,7 +165,7 @@ python scripts/validation_loss.py \
In addition we evaluate the model on OpenAI's _HumanEval_ benchmark. You can run the evaluation with the following command: In addition we evaluate the model on OpenAI's _HumanEval_ benchmark. You can run the evaluation with the following command:
```bash ```bash
python scripts/human_eval.py --model_ckpt lvwerra/codeparrot \ accelerate launch scripts/human_eval.py --model_ckpt lvwerra/codeparrot \
--do_sample True \ --do_sample True \
--temperature 0.2 \ --temperature 0.2 \
--top_p 0.95 \ --top_p 0.95 \
@@ -178,7 +178,7 @@ The results as well as reference values are shown in the following table:
| Model | pass@1 | pass@10 | pass@100| | Model | pass@1 | pass@10 | pass@100|
|-------|--------|---------|---------| |-------|--------|---------|---------|
|CodeParrot 🦜 (110M) | 3.80% | 6.57% | 12.78% | |CodeParrot 🦜 (110M) | 3.80% | 6.57% | 12.78% |
|CodeParrot 🦜 (1.5B) | 3.58% | 8.03% | 14.96% | |CodeParrot 🦜 (1.5B) | 3.99% | 8.69% | 17.88% |
||||| |||||
|Codex (25M)| 3.21% | 7.1% | 12.89%| |Codex (25M)| 3.21% | 7.1% | 12.89%|
|Codex (85M)| 8.22% | 12.81% | 22.40% | |Codex (85M)| 8.22% | 12.81% | 22.40% |