Tokenizers: ability to load from model subfolder (#8586)

* <small>tiny typo</small>

* Tokenizers: ability to load from model subfolder

* use subfolder for local files as well

* Uniformize model shortcut name => model id

* from s3 => from huggingface.co

Co-authored-by: Quentin Lhoest <lhoest.q@gmail.com>
This commit is contained in:
Julien Chaumond
2020-11-17 14:58:45 +01:00
committed by GitHub
parent 48395d6b8e
commit 042a6aa777
54 changed files with 210 additions and 186 deletions

View File

@@ -620,7 +620,7 @@ def main():
"--cache_dir",
default="",
type=str,
help="Where do you want to store the pre-trained models downloaded from s3",
help="Where do you want to store the pre-trained models downloaded from huggingface.co",
)
parser.add_argument(
"--max_seq_length",

View File

@@ -725,7 +725,7 @@ def main():
"--cache_dir",
default="",
type=str,
help="Where do you want to store the pre-trained models downloaded from s3",
help="Where do you want to store the pre-trained models downloaded from huggingface.co",
)
parser.add_argument(