From 3114df41f448d8d4b0018e0629a9f66c53df224c Mon Sep 17 00:00:00 2001
From: Loubna Ben Allal <44069155+loubnabnl@users.noreply.github.com>
Date: Fri, 10 Jun 2022 15:55:24 +0200
Subject: [PATCH] update README.md (#17657)

- use CodeParrot scores of v1.1
- change evaluation command to use accelerate
---
 examples/research_projects/codeparrot/README.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/examples/research_projects/codeparrot/README.md b/examples/research_projects/codeparrot/README.md
index d93415d7ce..761b77a6df 100644
--- a/examples/research_projects/codeparrot/README.md
+++ b/examples/research_projects/codeparrot/README.md
@@ -165,7 +165,7 @@ python scripts/validation_loss.py \
 In addition we evaluate the model on OpenAI's _HumanEval_ benchmark. You can run the evaluation with the following command:
 
 ```bash
-python scripts/human_eval.py --model_ckpt lvwerra/codeparrot \
+accelerate launch  scripts/human_eval.py --model_ckpt lvwerra/codeparrot \
 --do_sample True \
 --temperature 0.2 \
 --top_p 0.95 \
@@ -178,7 +178,7 @@ The results as well as reference values are shown in the following table:
 | Model | pass@1 | pass@10 | pass@100|
 |-------|--------|---------|---------|
 |CodeParrot 🦜 (110M) | 3.80% | 6.57% | 12.78% |
-|CodeParrot 🦜 (1.5B) | 3.58% | 8.03% | 14.96% |
+|CodeParrot 🦜 (1.5B) | 3.99% | 8.69% | 17.88% |
 |||||
 |Codex (25M)| 3.21% | 7.1% |	12.89%|
 |Codex (85M)| 8.22%	| 12.81% | 22.40% |