[docs] Redesign (#31757)
* toctree * not-doctested.txt * collapse sections * feedback * update * rewrite get started sections * fixes * fix * loading models * fix * customize models * share * fix link * contribute part 1 * contribute pt 2 * fix toctree * tokenization pt 1 * Add new model (#32615) * v1 - working version * fix * fix * fix * fix * rename to correct name * fix title * fixup * rename files * fix * add copied from on tests * rename to `FalconMamba` everywhere and fix bugs * fix quantization + accelerate * fix copies * add `torch.compile` support * fix tests * fix tests and add slow tests * copies on config * merge the latest changes * fix tests * add few lines about instruct * Apply suggestions from code review Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * fix * fix tests --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * "to be not" -> "not to be" (#32636) * "to be not" -> "not to be" * Update sam.md * Update trainer.py * Update modeling_utils.py * Update test_modeling_utils.py * Update test_modeling_utils.py * fix hfoption tag * tokenization pt. 2 * image processor * fix toctree * backbones * feature extractor * fix file name * processor * update not-doctested * update * make style * fix toctree * revision * make fixup * fix toctree * fix * make style * fix hfoption tag * pipeline * pipeline gradio * pipeline web server * add pipeline * fix toctree * not-doctested * prompting * llm optims * fix toctree * fixes * cache * text generation * fix * chat pipeline * chat stuff * xla * torch.compile * cpu inference * toctree * gpu inference * agents and tools * gguf/tiktoken * finetune * toctree * trainer * trainer pt 2 * optims * optimizers * accelerate * parallelism * fsdp * update * distributed cpu * hardware training * gpu training * gpu training 2 * peft * distrib debug * deepspeed 1 * deepspeed 2 * chat toctree * quant pt 1 * quant pt 2 * fix toctree * fix * fix * quant pt 3 * quant pt 4 * serialization * torchscript * scripts * tpu * review * model addition timeline * modular * more reviews * reviews * fix toctree * reviews reviews * continue reviews * more reviews * modular transformers * more review * zamba2 * fix * all frameworks * pytorch * supported model frameworks * flashattention * rm check_table * not-doctested.txt * rm check_support_list.py * feedback * updates/feedback * review * feedback * fix * update * feedback * updates * update --------- Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>
This commit is contained in:
Steven Liu
@@ -13,124 +13,155 @@ rendered properly in your Markdown viewer.
|
||||
|
||||
-->
|
||||
|
||||
# Hyperparameter Search using Trainer API
|
||||
# Hyperparameter search
|
||||
|
||||
🤗 Transformers provides a [`Trainer`] class optimized for training 🤗 Transformers models, making it easier to start training without manually writing your own training loop. The [`Trainer`] provides API for hyperparameter search. This doc shows how to enable it in example.
|
||||
Hyperparameter search discovers an optimal set of hyperparameters that produces the best model performance. [`Trainer`] supports several hyperparameter search backends - [Optuna](https://optuna.readthedocs.io/en/stable/index.html), [SigOpt](https://docs.sigopt.com/), [Weights & Biases](https://docs.wandb.ai/), [Ray Tune](https://docs.ray.io/en/latest/tune/index.html) - through [`~Trainer.hyperparameter_search`] to optimize an objective or even multiple objectives.
|
||||
|
||||
## Hyperparameter Search backend
|
||||
This guide will go over how to set up a hyperparameter search for each of the backends.
|
||||
|
||||
[`Trainer`] supports four hyperparameter search backends currently:
|
||||
[optuna](https://optuna.org/), [sigopt](https://sigopt.com/), [raytune](https://docs.ray.io/en/latest/tune/index.html) and [wandb](https://wandb.ai/site/sweeps).
|
||||
|
||||
you should install them before using them as the hyperparameter search backend
|
||||
```bash
|
||||
pip install optuna/sigopt/wandb/ray[tune]
|
||||
```
|
||||
|
||||
## How to enable Hyperparameter search in example
|
||||
To use [`~Trainer.hyperparameter_search`], you need to create a `model_init` function. This function includes basic model information (arguments and configuration) because it needs to be reinitialized for each search trial in the run.
|
||||
|
||||
Define the hyperparameter search space, different backends need different format.
|
||||
> [!WARNING]
|
||||
> The `model_init` function is incompatible with the [optimizers](./main_classes/trainer#transformers.Trainer.optimizers) parameter. Subclass [`Trainer`] and override the [`~Trainer.create_optimizer_and_scheduler`] method to create a custom optimizer and scheduler.
|
||||
|
||||
For sigopt, see sigopt [object_parameter](https://docs.sigopt.com/ai-module-api-references/api_reference/objects/object_parameter), it's like following:
|
||||
```py
|
||||
>>> def sigopt_hp_space(trial):
|
||||
... return [
|
||||
... {"bounds": {"min": 1e-6, "max": 1e-4}, "name": "learning_rate", "type": "double"},
|
||||
... {
|
||||
... "categorical_values": ["16", "32", "64", "128"],
|
||||
... "name": "per_device_train_batch_size",
|
||||
... "type": "categorical",
|
||||
... },
|
||||
... ]
|
||||
```
|
||||
|
||||
For optuna, see optuna [object_parameter](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html#sphx-glr-tutorial-10-key-features-002-configurations-py), it's like following:
|
||||
An example `model_init` function is shown below.
|
||||
|
||||
```py
|
||||
>>> def optuna_hp_space(trial):
|
||||
... return {
|
||||
... "learning_rate": trial.suggest_float("learning_rate", 1e-6, 1e-4, log=True),
|
||||
... "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [16, 32, 64, 128]),
|
||||
... }
|
||||
def model_init(trial):
|
||||
return AutoModelForSequenceClassification.from_pretrained(
|
||||
model_args.model_name_or_path,
|
||||
from_tf=bool(".ckpt" in model_args.model_name_or_path),
|
||||
config=config,
|
||||
cache_dir=model_args.cache_dir,
|
||||
revision=model_args.model_revision,
|
||||
token=True if model_args.use_auth_token else None,
|
||||
)
|
||||
```
|
||||
|
||||
Optuna provides multi-objective HPO. You can pass `direction` in `hyperparameter_search` and define your own compute_objective to return multiple objective values. The Pareto Front (`List[BestRun]`) will be returned in hyperparameter_search, you should refer to the test case `TrainerHyperParameterMultiObjectOptunaIntegrationTest` in [test_trainer](https://github.com/huggingface/transformers/blob/main/tests/trainer/test_trainer.py). It's like following
|
||||
Pass `model_init` to [`Trainer`] along with everything else you need for training. Then you can call [`~Trainer.hyperparameter_search`] to start the search.
|
||||
|
||||
[`~Trainer.hyperparameter_search`] accepts a [direction](./main_classes/trainer#transformers.Trainer.hyperparameter_search.direction) parameter to specify whether to minimize, maximize, or minimize and maximize multiple objectives. You'll also need to set the [backend](./main_classes/trainer#transformers.Trainer.hyperparameter_search.backend) you're using, an [object](./main_classes/trainer#transformers.Trainer.hyperparameter_search.hp_space) containing the hyperparameters to optimize for, the [number of trials](./main_classes/trainer#transformers.Trainer.hyperparameter_search.n_trials) to run, and a [compute_objective](./main_classes/trainer#transformers.Trainer.hyperparameter_search.compute_objective) to return the objective values.
|
||||
|
||||
> [!TIP]
|
||||
> If [compute_objective](./main_classes/trainer#transformers.Trainer.hyperparameter_search.compute_objective) isn't defined, the default [compute_objective](./main_classes/trainer#transformers.Trainer.hyperparameter_search.compute_objective) is called which is the sum of an evaluation metric like F1.
|
||||
|
||||
```py
|
||||
>>> best_trials = trainer.hyperparameter_search(
|
||||
... direction=["minimize", "maximize"],
|
||||
... backend="optuna",
|
||||
... hp_space=optuna_hp_space,
|
||||
... n_trials=20,
|
||||
... compute_objective=compute_objective,
|
||||
... )
|
||||
from transformers import Trainer
|
||||
|
||||
trainer = Trainer(
|
||||
model=None,
|
||||
args=training_args,
|
||||
train_dataset=small_train_dataset,
|
||||
eval_dataset=small_eval_dataset,
|
||||
compute_metrics=compute_metrics,
|
||||
processing_class=tokenizer,
|
||||
model_init=model_init,
|
||||
data_collator=data_collator,
|
||||
)
|
||||
trainer.hyperparameter_search(...)
|
||||
```
|
||||
|
||||
For raytune, see raytune [object_parameter](https://docs.ray.io/en/latest/tune/api/search_space.html), it's like following:
|
||||
The following examples demonstrate how to perform a hyperparameter search for the learning rate and training batch size using the different backends.
|
||||
|
||||
<hfoptions id="backends">
|
||||
<hfoption id="Optuna">
|
||||
|
||||
[Optuna](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html#sphx-glr-tutorial-10-key-features-002-configurations-py) optimizes categories, integers, and floats.
|
||||
|
||||
```py
|
||||
>>> def ray_hp_space(trial):
|
||||
... return {
|
||||
... "learning_rate": tune.loguniform(1e-6, 1e-4),
|
||||
... "per_device_train_batch_size": tune.choice([16, 32, 64, 128]),
|
||||
... }
|
||||
def optuna_hp_space(trial):
|
||||
return {
|
||||
"learning_rate": trial.suggest_float("learning_rate", 1e-6, 1e-4, log=True),
|
||||
"per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [16, 32, 64, 128]),
|
||||
}
|
||||
|
||||
best_trials = trainer.hyperparameter_search(
|
||||
direction=["minimize", "maximize"],
|
||||
backend="optuna",
|
||||
hp_space=optuna_hp_space,
|
||||
n_trials=20,
|
||||
compute_objective=compute_objective,
|
||||
)
|
||||
```
|
||||
|
||||
For wandb, see wandb [object_parameter](https://docs.wandb.ai/guides/sweeps/configuration), it's like following:
|
||||
</hfoption>
|
||||
<hfoption id="Ray Tune">
|
||||
|
||||
[Ray Tune](https://docs.ray.io/en/latest/tune/api/search_space.html) optimizes floats, integers, and categorical parameters. It also offers multiple sampling distributions for each parameter such as uniform and log-uniform.
|
||||
|
||||
```py
|
||||
>>> def wandb_hp_space(trial):
|
||||
... return {
|
||||
... "method": "random",
|
||||
... "metric": {"name": "objective", "goal": "minimize"},
|
||||
... "parameters": {
|
||||
... "learning_rate": {"distribution": "uniform", "min": 1e-6, "max": 1e-4},
|
||||
... "per_device_train_batch_size": {"values": [16, 32, 64, 128]},
|
||||
... },
|
||||
... }
|
||||
def ray_hp_space(trial):
|
||||
return {
|
||||
"learning_rate": tune.loguniform(1e-6, 1e-4),
|
||||
"per_device_train_batch_size": tune.choice([16, 32, 64, 128]),
|
||||
}
|
||||
|
||||
best_trials = trainer.hyperparameter_search(
|
||||
direction=["minimize", "maximize"],
|
||||
backend="ray",
|
||||
hp_space=ray_hp_space,
|
||||
n_trials=20,
|
||||
compute_objective=compute_objective,
|
||||
)
|
||||
```
|
||||
|
||||
Define a `model_init` function and pass it to the [`Trainer`], as an example:
|
||||
```py
|
||||
>>> def model_init(trial):
|
||||
... return AutoModelForSequenceClassification.from_pretrained(
|
||||
... model_args.model_name_or_path,
|
||||
... from_tf=bool(".ckpt" in model_args.model_name_or_path),
|
||||
... config=config,
|
||||
... cache_dir=model_args.cache_dir,
|
||||
... revision=model_args.model_revision,
|
||||
... token=True if model_args.use_auth_token else None,
|
||||
... )
|
||||
```
|
||||
</hfoption>
|
||||
<hfoption id="SigOpt">
|
||||
|
||||
Create a [`Trainer`] with your `model_init` function, training arguments, training and test datasets, and evaluation function:
|
||||
[SigOpt](https://docs.sigopt.com/ai-module-api-references/api_reference/objects/object_parameter) optimizes double, integer, and categorical parameters.
|
||||
|
||||
```py
|
||||
>>> trainer = Trainer(
|
||||
... model=None,
|
||||
... args=training_args,
|
||||
... train_dataset=small_train_dataset,
|
||||
... eval_dataset=small_eval_dataset,
|
||||
... compute_metrics=compute_metrics,
|
||||
... processing_class=tokenizer,
|
||||
... model_init=model_init,
|
||||
... data_collator=data_collator,
|
||||
... )
|
||||
def sigopt_hp_space(trial):
|
||||
return [
|
||||
{"bounds": {"min": 1e-6, "max": 1e-4}, "name": "learning_rate", "type": "double"},
|
||||
{
|
||||
"categorical_values": ["16", "32", "64", "128"],
|
||||
"name": "per_device_train_batch_size",
|
||||
"type": "categorical",
|
||||
},
|
||||
]
|
||||
|
||||
best_trials = trainer.hyperparameter_search(
|
||||
direction=["minimize", "maximize"],
|
||||
backend="sigopt",
|
||||
hp_space=sigopt_hp_space,
|
||||
n_trials=20,
|
||||
compute_objective=compute_objective,
|
||||
)
|
||||
```
|
||||
|
||||
Call hyperparameter search, get the best trial parameters, backend could be `"optuna"`/`"sigopt"`/`"wandb"`/`"ray"`. direction can be`"minimize"` or `"maximize"`, which indicates whether to optimize greater or lower objective.
|
||||
</hfoption>
|
||||
<hfoption id="Weights & Biases">
|
||||
|
||||
You could define your own compute_objective function, if not defined, the default compute_objective will be called, and the sum of eval metric like f1 is returned as objective value.
|
||||
[Weights & Biases](https://docs.wandb.ai/guides/sweeps/sweep-config-keys) also optimizes integers, floats, and categorical parameters. It also includes support for different search strategies and distribution options.
|
||||
|
||||
```py
|
||||
>>> best_trial = trainer.hyperparameter_search(
|
||||
... direction="maximize",
|
||||
... backend="optuna",
|
||||
... hp_space=optuna_hp_space,
|
||||
... n_trials=20,
|
||||
... compute_objective=compute_objective,
|
||||
... )
|
||||
def wandb_hp_space(trial):
|
||||
return {
|
||||
"method": "random",
|
||||
"metric": {"name": "objective", "goal": "minimize"},
|
||||
"parameters": {
|
||||
"learning_rate": {"distribution": "uniform", "min": 1e-6, "max": 1e-4},
|
||||
"per_device_train_batch_size": {"values": [16, 32, 64, 128]},
|
||||
},
|
||||
}
|
||||
|
||||
best_trials = trainer.hyperparameter_search(
|
||||
direction=["minimize", "maximize"],
|
||||
backend="wandb",
|
||||
hp_space=wandb_hp_space,
|
||||
n_trials=20,
|
||||
compute_objective=compute_objective,
|
||||
)
|
||||
```
|
||||
|
||||
## Hyperparameter search For DDP finetune
|
||||
Currently, Hyperparameter search for DDP is enabled for optuna and sigopt. Only the rank-zero process will generate the search trial and pass the argument to other ranks.
|
||||
</hfoption>
|
||||
</hfoptions>
|
||||
|
||||
## Distributed Data Parallel
|
||||
|
||||
[`Trainer`] only supports hyperparameter search for distributed data parallel (DDP) on the Optuna and SigOpt backends. Only the rank-zero process is used to generate the search trial, and the resulting parameters are passed along to the other ranks.
|
||||
|
||||
Reference in New Issue
Block a user