Adding performer fine-tuning research exampke (#9239)
* added run_mlm_performer.py research example * make styke * make styke * Added a README !
This commit is contained in:
25
examples/research_projects/performer/README.md
Normal file
25
examples/research_projects/performer/README.md
Normal file
@@ -0,0 +1,25 @@
|
||||
# Performer fine-tuning
|
||||
|
||||
Example authors: @TevenLeScao, @Patrickvonplaten
|
||||
|
||||
Paper authors: Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, Adrian Weller
|
||||
|
||||
## Requirements
|
||||
|
||||
`datasets`, `flax` and `jax`. `wandb` integration is built-in if you want to use it.
|
||||
|
||||
## Examples
|
||||
|
||||
`sanity_script.sh` will launch performer fine-tuning from the bert-base-cased checkpoint on the Simple Wikipedia dataset (a small, easy-language English Wikipedia) from `datasets`.
|
||||
`full_script.sh` will launch performer fine-tuning from the bert-large-cased checkpoint on the English Wikipedia dataset from `datasets`.
|
||||
|
||||
Here are a few key arguments:
|
||||
- Remove the `--performer` argument to use a standard Bert model.
|
||||
|
||||
- Add `--reinitialize` to start from a blank model rather than a Bert checkpoint.
|
||||
|
||||
- You may change the Bert size by passing a different [checkpoint](https://huggingface.co/transformers/pretrained_models.html) to the `--model_name_or_path` argument.
|
||||
|
||||
- Passing your user name to the `--wandb_user_name` argument will trigger weights and biases logging.
|
||||
|
||||
- You can choose a dataset with `--dataset_name` and `--dataset_config`. Our [viewer](https://huggingface.co/datasets/viewer/) will help you find what you need.
|
||||
Reference in New Issue
Block a user