Set cache_dir for evaluate.load() in example scripts (#28422)

While using `run_clm.py`,[^1] I noticed that some files were being added
to my global cache, not the local cache. I set the `cache_dir` parameter
for the one call to `evaluate.load()`, which partially solved the
problem. I figured that while I was fixing the one script upstream, I
might as well fix the problem in all other example scripts that I could.

There are still some files being added to my global cache, but this
appears to be a bug in `evaluate` itself. This commit at least moves
some of the files into the local cache, which is better than before.

To create this PR, I made the following regex-based transformation:
`evaluate\.load\((.*?)\)` -> `evaluate\.load\($1,
cache_dir=model_args.cache_dir\)`. After using that, I manually fixed
all modified files with `ruff` serving as useful guidance. During the
process, I removed one existing usage of the `cache_dir` parameter in a
script that did not have a corresponding `--cache-dir` argument
declared.

[^1]: I specifically used `pytorch/language-modeling/run_clm.py` from
v4.34.1 of the library. For the original code, see the following URL:
acc394c4f5/examples/pytorch/language-modeling/run_clm.py.
This commit is contained in:
Alex Hedges
2024-01-11 09:38:44 -05:00
committed by GitHub
parent 5fd5ef7624
commit 95091e1582
31 changed files with 47 additions and 38 deletions

View File

@@ -633,23 +633,23 @@ def main():
if data_args.metric_name is not None:
metric = (
evaluate.load(data_args.metric_name, config_name="multilabel")
evaluate.load(data_args.metric_name, config_name="multilabel", cache_dir=model_args.cache_dir)
if is_multi_label
else evaluate.load(data_args.metric_name)
else evaluate.load(data_args.metric_name, cache_dir=model_args.cache_dir)
)
logger.info(f"Using metric {data_args.metric_name} for evaluation.")
else:
if is_regression:
metric = evaluate.load("mse")
metric = evaluate.load("mse", cache_dir=model_args.cache_dir)
logger.info("Using mean squared error (mse) as regression score, you can use --metric_name to overwrite.")
else:
if is_multi_label:
metric = evaluate.load("f1", config_name="multilabel")
metric = evaluate.load("f1", config_name="multilabel", cache_dir=model_args.cache_dir)
logger.info(
"Using multilabel F1 for multi-label classification task, you can use --metric_name to overwrite."
)
else:
metric = evaluate.load("accuracy")
metric = evaluate.load("accuracy", cache_dir=model_args.cache_dir)
logger.info("Using accuracy as classification score, you can use --metric_name to overwrite.")
def compute_metrics(p: EvalPrediction):

View File

@@ -514,11 +514,11 @@ def main():
# Get the metric function
if data_args.task_name is not None:
metric = evaluate.load("glue", data_args.task_name)
metric = evaluate.load("glue", data_args.task_name, cache_dir=model_args.cache_dir)
elif is_regression:
metric = evaluate.load("mse")
metric = evaluate.load("mse", cache_dir=model_args.cache_dir)
else:
metric = evaluate.load("accuracy")
metric = evaluate.load("accuracy", cache_dir=model_args.cache_dir)
# You can define your custom compute_metrics function. It takes an `EvalPrediction` object (a namedtuple with a
# predictions and label_ids field) and has to return a dictionary string to float.

View File

@@ -385,7 +385,7 @@ def main():
)
# Get the metric function
metric = evaluate.load("xnli")
metric = evaluate.load("xnli", cache_dir=model_args.cache_dir)
# You can define your custom compute_metrics function. It takes an `EvalPrediction` object (a namedtuple with a
# predictions and label_ids field) and has to return a dictionary string to float.