[doc] Model upload and sharing
ping @lysandrejik @thomwolf Is this clear enough? Anything we should add?
This commit is contained in:
41
README.md
41
README.md
@@ -56,6 +56,7 @@ Choose the right framework for every part of a model's lifetime
|
|||||||
| [Quick tour: Usage](#quick-tour) | Tokenizers & models usage: Bert and GPT-2 |
|
| [Quick tour: Usage](#quick-tour) | Tokenizers & models usage: Bert and GPT-2 |
|
||||||
| [Quick tour: TF 2.0 and PyTorch ](#Quick-tour-TF-20-training-and-PyTorch-interoperability) | Train a TF 2.0 model in 10 lines of code, load it in PyTorch |
|
| [Quick tour: TF 2.0 and PyTorch ](#Quick-tour-TF-20-training-and-PyTorch-interoperability) | Train a TF 2.0 model in 10 lines of code, load it in PyTorch |
|
||||||
| [Quick tour: Fine-tuning/usage scripts](#quick-tour-of-the-fine-tuningusage-scripts) | Using provided scripts: GLUE, SQuAD and Text generation |
|
| [Quick tour: Fine-tuning/usage scripts](#quick-tour-of-the-fine-tuningusage-scripts) | Using provided scripts: GLUE, SQuAD and Text generation |
|
||||||
|
| [Quick tour: Share your models ](#Quick-tour-of-model-sharing) | Upload and share your fine-tuned models with the community |
|
||||||
| [Migrating from pytorch-transformers to transformers](#Migrating-from-pytorch-transformers-to-transformers) | Migrating your code from pytorch-transformers to transformers |
|
| [Migrating from pytorch-transformers to transformers](#Migrating-from-pytorch-transformers-to-transformers) | Migrating your code from pytorch-transformers to transformers |
|
||||||
| [Migrating from pytorch-pretrained-bert to pytorch-transformers](#Migrating-from-pytorch-pretrained-bert-to-transformers) | Migrating your code from pytorch-pretrained-bert to transformers |
|
| [Migrating from pytorch-pretrained-bert to pytorch-transformers](#Migrating-from-pytorch-pretrained-bert-to-transformers) | Migrating your code from pytorch-pretrained-bert to transformers |
|
||||||
| [Documentation][(v2.2.0/v2.2.1/v2.2.2)](https://huggingface.co/transformers/v2.2.0) [(v2.1.1)](https://huggingface.co/transformers/v2.1.1) [(v2.0.0)](https://huggingface.co/transformers/v2.0.0) [(v1.2.0)](https://huggingface.co/transformers/v1.2.0) [(v1.1.0)](https://huggingface.co/transformers/v1.1.0) [(v1.0.0)](https://huggingface.co/transformers/v1.0.0) [(master)](https://huggingface.co/transformers) | Full API documentation and more |
|
| [Documentation][(v2.2.0/v2.2.1/v2.2.2)](https://huggingface.co/transformers/v2.2.0) [(v2.1.1)](https://huggingface.co/transformers/v2.1.1) [(v2.0.0)](https://huggingface.co/transformers/v2.0.0) [(v1.2.0)](https://huggingface.co/transformers/v1.2.0) [(v1.1.0)](https://huggingface.co/transformers/v1.1.0) [(v1.0.0)](https://huggingface.co/transformers/v1.0.0) [(master)](https://huggingface.co/transformers) | Full API documentation and more |
|
||||||
@@ -446,6 +447,46 @@ python ./examples/run_generation.py \
|
|||||||
--repetition_penalty=1.2 \
|
--repetition_penalty=1.2 \
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Quick tour of model sharing
|
||||||
|
|
||||||
|
New in `v2.2.2`: you can now upload and share your fine-tuned models with the community, using the <abbr title="Command-line interface">CLI</abbr> that's built-in to the library.
|
||||||
|
|
||||||
|
**First, create an account on [https://huggingface.co/join](https://huggingface.co/join)**. Then:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
transformers-cli login
|
||||||
|
# log in using the same credentials as on huggingface.co
|
||||||
|
```
|
||||||
|
Upload your model:
|
||||||
|
```shell
|
||||||
|
transformers-cli upload ./path/to/pretrained_model/
|
||||||
|
|
||||||
|
# ^^ Upload folder containing weights/tokenizer/config
|
||||||
|
# saved via `.save_pretrained()`
|
||||||
|
|
||||||
|
transformers-cli upload ./config.json [--filename foobar.json]
|
||||||
|
|
||||||
|
# ^^ Upload a single file
|
||||||
|
# (you can optionally override its filename)
|
||||||
|
```
|
||||||
|
|
||||||
|
Your model will then be accessible through its identifier:
|
||||||
|
```python
|
||||||
|
"username/model_name"
|
||||||
|
```
|
||||||
|
|
||||||
|
Anyone can load it from code:
|
||||||
|
```python
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained("username/model_name")
|
||||||
|
model = AutoModel.from_pretrained("username/model_name")
|
||||||
|
```
|
||||||
|
|
||||||
|
Finally, list all your files on S3:
|
||||||
|
```shell
|
||||||
|
transformers-cli ls
|
||||||
|
# List all your S3 objects.
|
||||||
|
```
|
||||||
|
|
||||||
## Migrating from pytorch-transformers to transformers
|
## Migrating from pytorch-transformers to transformers
|
||||||
|
|
||||||
Here is a quick summary of what you should take care of when migrating from `pytorch-transformers` to `transformers`.
|
Here is a quick summary of what you should take care of when migrating from `pytorch-transformers` to `transformers`.
|
||||||
|
|||||||
@@ -58,6 +58,7 @@ The library currently contains PyTorch and Tensorflow implementations, pre-train
|
|||||||
installation
|
installation
|
||||||
quickstart
|
quickstart
|
||||||
pretrained_models
|
pretrained_models
|
||||||
|
model_sharing
|
||||||
examples
|
examples
|
||||||
notebooks
|
notebooks
|
||||||
serialization
|
serialization
|
||||||
|
|||||||
40
docs/source/model_sharing.md
Normal file
40
docs/source/model_sharing.md
Normal file
@@ -0,0 +1,40 @@
|
|||||||
|
# Model upload and sharing
|
||||||
|
|
||||||
|
Starting with `v2.2.2`, you can now upload and share your fine-tuned models with the community, using the <abbr title="Command-line interface">CLI</abbr> that's built-in to the library.
|
||||||
|
|
||||||
|
**First, create an account on [https://huggingface.co/join](https://huggingface.co/join)**. Then:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
transformers-cli login
|
||||||
|
# log in using the same credentials as on huggingface.co
|
||||||
|
```
|
||||||
|
Upload your model:
|
||||||
|
```shell
|
||||||
|
transformers-cli upload ./path/to/pretrained_model/
|
||||||
|
|
||||||
|
# ^^ Upload folder containing weights/tokenizer/config
|
||||||
|
# saved via `.save_pretrained()`
|
||||||
|
|
||||||
|
transformers-cli upload ./config.json [--filename foobar.json]
|
||||||
|
|
||||||
|
# ^^ Upload a single file
|
||||||
|
# (you can optionally override its filename)
|
||||||
|
```
|
||||||
|
|
||||||
|
Your model will then be accessible through its identifier:
|
||||||
|
```python
|
||||||
|
"username/model_name"
|
||||||
|
```
|
||||||
|
|
||||||
|
Anyone can load it from code:
|
||||||
|
```python
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained("username/model_name")
|
||||||
|
model = AutoModel.from_pretrained("username/model_name")
|
||||||
|
```
|
||||||
|
|
||||||
|
Finally, list all your files on S3:
|
||||||
|
```shell
|
||||||
|
transformers-cli ls
|
||||||
|
# List all your S3 objects.
|
||||||
|
```
|
||||||
|
|
||||||
Reference in New Issue
Block a user