From 153ec2f1542f1d0c5b11fcd2afee0d58b1129776 Mon Sep 17 00:00:00 2001 From: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> Date: Tue, 15 Sep 2020 10:40:57 -0400 Subject: [PATCH] Funnel model cards (#7147) --- .../intermediate-base/README.md | 94 +++++++++++++++++++ .../funnel-transformer/intermediate/README.md | 90 ++++++++++++++++++ .../funnel-transformer/large-base/README.md | 94 +++++++++++++++++++ .../funnel-transformer/large/README.md | 90 ++++++++++++++++++ .../funnel-transformer/medium-base/README.md | 94 +++++++++++++++++++ .../funnel-transformer/medium/README.md | 90 ++++++++++++++++++ .../funnel-transformer/small-base/README.md | 94 +++++++++++++++++++ .../funnel-transformer/small/README.md | 90 ++++++++++++++++++ .../funnel-transformer/xlarge-base/README.md | 94 +++++++++++++++++++ .../funnel-transformer/xlarge/README.md | 90 ++++++++++++++++++ 10 files changed, 920 insertions(+) create mode 100644 model_cards/funnel-transformer/intermediate-base/README.md create mode 100644 model_cards/funnel-transformer/intermediate/README.md create mode 100644 model_cards/funnel-transformer/large-base/README.md create mode 100644 model_cards/funnel-transformer/large/README.md create mode 100644 model_cards/funnel-transformer/medium-base/README.md create mode 100644 model_cards/funnel-transformer/medium/README.md create mode 100644 model_cards/funnel-transformer/small-base/README.md create mode 100644 model_cards/funnel-transformer/small/README.md create mode 100644 model_cards/funnel-transformer/xlarge-base/README.md create mode 100644 model_cards/funnel-transformer/xlarge/README.md diff --git a/model_cards/funnel-transformer/intermediate-base/README.md b/model_cards/funnel-transformer/intermediate-base/README.md new file mode 100644 index 0000000000..68d3c4a9e1 --- /dev/null +++ b/model_cards/funnel-transformer/intermediate-base/README.md @@ -0,0 +1,94 @@ +--- +language: en +license: apache-2.0 +datasets: +- bookcorpus +- wikipedia +- gigaword +--- + +# Funnel Transformer intermediate model (B6-6-6 without decoder) + +Pretrained model on English language using a similar objective objective as [ELECTRA](https://huggingface.co/transformers/model_doc/electra.html). It was introduced in +[this paper](https://arxiv.org/pdf/2006.03236.pdf) and first released in +[this repository](https://github.com/laiguokun/Funnel-Transformer). This model is uncased: it does not make a difference +between english and English. + +Disclaimer: The team releasing Funnel Transformer did not write a model card for this model so this model card has been +written by the Hugging Face team. + +## Model description + +Funnel Transformer is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it +was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of +publicly available data) with an automatic process to generate inputs and labels from those texts. + +More precisely, a small language model corrupts the input texts and serves as a generator of inputs for this model, and +the pretraining objective is to predict which token is an original and which one has been replaced, a bit like a GAN training. + +This way, the model learns an inner representation of the English language that can then be used to extract features +useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard +classifier using the features produced by the BERT model as inputs. + +**Note:** This model does not contain the decoder, so it ouputs hidden states that have a sequence length of one fourth +of the inputs. It's good to use for tasks requiring a summary of the sentence (like sentence classification) but not if +you need one input per initial token. You should use the `intermediate` model in that case. + +## Intended uses & limitations + +You can use the raw model to extract a vector representation of a given text, but it's mostly intended to +be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=funnel-transformer) to look for +fine-tuned versions on a task that interests you. + +Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) +to make decisions, such as sequence classification, token classification or question answering. For tasks such as text +generation you should look at model like GPT2. + +### How to use + + +Here is how to use this model to get the features of a given text in PyTorch: + +```python +from transformers import FunnelTokenizer, FunnelBaseModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/intermediate-base") +model = FunnelBaseModel.from_pretrained("funnel-transformer/intermediate-base") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='pt') +output = model(**encoded_input) +``` + +and in TensorFlow: + +```python +from transformers import FunnelTokenizer, TFFunnelBaseModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/intermediate-base") +model = TFFunnelBaseModel.from_pretrained("funnel-transformer/intermediate-base") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='tf') +output = model(encoded_input) +``` + +## Training data + +The BERT model was pretrained on: +- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books, +- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers), +- [Clue Web](https://lemurproject.org/clueweb12/), a dataset of 733,019,372 English web pages, +- [GigaWord](https://catalog.ldc.upenn.edu/LDC2011T07), an archive of newswire text data, +- [Common Crawl](https://commoncrawl.org/), a dataset of raw web pages. + + +### BibTeX entry and citation info + +```bibtex +@misc{dai2020funneltransformer, + title={Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing}, + author={Zihang Dai and Guokun Lai and Yiming Yang and Quoc V. Le}, + year={2020}, + eprint={2006.03236}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} +``` + diff --git a/model_cards/funnel-transformer/intermediate/README.md b/model_cards/funnel-transformer/intermediate/README.md new file mode 100644 index 0000000000..5645505a7c --- /dev/null +++ b/model_cards/funnel-transformer/intermediate/README.md @@ -0,0 +1,90 @@ +--- +language: en +license: apache-2.0 +datasets: +- bookcorpus +- wikipedia +- gigaword +--- + +# Funnel Transformer intermediate model (B6-6-6 with decoder) + +Pretrained model on English language using a similar objective objective as [ELECTRA](https://huggingface.co/transformers/model_doc/electra.html). It was introduced in +[this paper](https://arxiv.org/pdf/2006.03236.pdf) and first released in +[this repository](https://github.com/laiguokun/Funnel-Transformer). This model is uncased: it does not make a difference +between english and English. + +Disclaimer: The team releasing Funnel Transformer did not write a model card for this model so this model card has been +written by the Hugging Face team. + +## Model description + +Funnel Transformer is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it +was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of +publicly available data) with an automatic process to generate inputs and labels from those texts. + +More precisely, a small language model corrupts the input texts and serves as a generator of inputs for this model, and +the pretraining objective is to predict which token is an original and which one has been replaced, a bit like a GAN training. + +This way, the model learns an inner representation of the English language that can then be used to extract features +useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard +classifier using the features produced by the BERT model as inputs. + +## Intended uses & limitations + +You can use the raw model to extract a vector representation of a given text, but it's mostly intended to +be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=funnel-transformer) to look for +fine-tuned versions on a task that interests you. + +Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) +to make decisions, such as sequence classification, token classification or question answering. For tasks such as text +generation you should look at model like GPT2. + +### How to use + + +Here is how to use this model to get the features of a given text in PyTorch: + +```python +from transformers import FunnelTokenizer, FunnelModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/intermediate") +model = FunneModel.from_pretrained("funnel-transformer/intermediate") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='pt') +output = model(**encoded_input) +``` + +and in TensorFlow: + +```python +from transformers import FunnelTokenizer, TFFunnelModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/intermediate") +model = TFFunnelModel.from_pretrained("funnel-transformer/intermediatesmall") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='tf') +output = model(encoded_input) +``` + +## Training data + +The BERT model was pretrained on: +- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books, +- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers), +- [Clue Web](https://lemurproject.org/clueweb12/), a dataset of 733,019,372 English web pages, +- [GigaWord](https://catalog.ldc.upenn.edu/LDC2011T07), an archive of newswire text data, +- [Common Crawl](https://commoncrawl.org/), a dataset of raw web pages. + + +### BibTeX entry and citation info + +```bibtex +@misc{dai2020funneltransformer, + title={Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing}, + author={Zihang Dai and Guokun Lai and Yiming Yang and Quoc V. Le}, + year={2020}, + eprint={2006.03236}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} +``` + diff --git a/model_cards/funnel-transformer/large-base/README.md b/model_cards/funnel-transformer/large-base/README.md new file mode 100644 index 0000000000..e8dd2f3e53 --- /dev/null +++ b/model_cards/funnel-transformer/large-base/README.md @@ -0,0 +1,94 @@ +--- +language: en +license: apache-2.0 +datasets: +- bookcorpus +- wikipedia +- gigaword +--- + +# Funnel Transformer large model (B8-8-8 without decoder) + +Pretrained model on English language using a similar objective objective as [ELECTRA](https://huggingface.co/transformers/model_doc/electra.html). It was introduced in +[this paper](https://arxiv.org/pdf/2006.03236.pdf) and first released in +[this repository](https://github.com/laiguokun/Funnel-Transformer). This model is uncased: it does not make a difference +between english and English. + +Disclaimer: The team releasing Funnel Transformer did not write a model card for this model so this model card has been +written by the Hugging Face team. + +## Model description + +Funnel Transformer is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it +was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of +publicly available data) with an automatic process to generate inputs and labels from those texts. + +More precisely, a small language model corrupts the input texts and serves as a generator of inputs for this model, and +the pretraining objective is to predict which token is an original and which one has been replaced, a bit like a GAN training. + +This way, the model learns an inner representation of the English language that can then be used to extract features +useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard +classifier using the features produced by the BERT model as inputs. + +**Note:** This model does not contain the decoder, so it ouputs hidden states that have a sequence length of one fourth +of the inputs. It's good to use for tasks requiring a summary of the sentence (like sentence classification) but not if +you need one input per initial token. You should use the `large` model in that case. + +## Intended uses & limitations + +You can use the raw model to extract a vector representation of a given text, but it's mostly intended to +be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=funnel-transformer) to look for +fine-tuned versions on a task that interests you. + +Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) +to make decisions, such as sequence classification, token classification or question answering. For tasks such as text +generation you should look at model like GPT2. + +### How to use + + +Here is how to use this model to get the features of a given text in PyTorch: + +```python +from transformers import FunnelTokenizer, FunnelBaseModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/large-base") +model = FunnelBaseModel.from_pretrained("funnel-transformer/large-base") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='pt') +output = model(**encoded_input) +``` + +and in TensorFlow: + +```python +from transformers import FunnelTokenizer, TFFunnelBaseModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/large-base") +model = TFFunnelBaseModel.from_pretrained("funnel-transformer/large-base") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='tf') +output = model(encoded_input) +``` + +## Training data + +The BERT model was pretrained on: +- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books, +- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers), +- [Clue Web](https://lemurproject.org/clueweb12/), a dataset of 733,019,372 English web pages, +- [GigaWord](https://catalog.ldc.upenn.edu/LDC2011T07), an archive of newswire text data, +- [Common Crawl](https://commoncrawl.org/), a dataset of raw web pages. + + +### BibTeX entry and citation info + +```bibtex +@misc{dai2020funneltransformer, + title={Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing}, + author={Zihang Dai and Guokun Lai and Yiming Yang and Quoc V. Le}, + year={2020}, + eprint={2006.03236}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} +``` + diff --git a/model_cards/funnel-transformer/large/README.md b/model_cards/funnel-transformer/large/README.md new file mode 100644 index 0000000000..9c8128e96d --- /dev/null +++ b/model_cards/funnel-transformer/large/README.md @@ -0,0 +1,90 @@ +--- +language: en +license: apache-2.0 +datasets: +- bookcorpus +- wikipedia +- gigaword +--- + +# Funnel Transformer large model (B8-8-8 with decoder) + +Pretrained model on English language using a similar objective objective as [ELECTRA](https://huggingface.co/transformers/model_doc/electra.html). It was introduced in +[this paper](https://arxiv.org/pdf/2006.03236.pdf) and first released in +[this repository](https://github.com/laiguokun/Funnel-Transformer). This model is uncased: it does not make a difference +between english and English. + +Disclaimer: The team releasing Funnel Transformer did not write a model card for this model so this model card has been +written by the Hugging Face team. + +## Model description + +Funnel Transformer is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it +was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of +publicly available data) with an automatic process to generate inputs and labels from those texts. + +More precisely, a small language model corrupts the input texts and serves as a generator of inputs for this model, and +the pretraining objective is to predict which token is an original and which one has been replaced, a bit like a GAN training. + +This way, the model learns an inner representation of the English language that can then be used to extract features +useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard +classifier using the features produced by the BERT model as inputs. + +## Intended uses & limitations + +You can use the raw model to extract a vector representation of a given text, but it's mostly intended to +be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=funnel-transformer) to look for +fine-tuned versions on a task that interests you. + +Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) +to make decisions, such as sequence classification, token classification or question answering. For tasks such as text +generation you should look at model like GPT2. + +### How to use + + +Here is how to use this model to get the features of a given text in PyTorch: + +```python +from transformers import FunnelTokenizer, FunnelModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/large") +model = FunneModel.from_pretrained("funnel-transformer/large") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='pt') +output = model(**encoded_input) +``` + +and in TensorFlow: + +```python +from transformers import FunnelTokenizer, TFFunnelModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/large") +model = TFFunnelModel.from_pretrained("funnel-transformer/large") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='tf') +output = model(encoded_input) +``` + +## Training data + +The BERT model was pretrained on: +- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books, +- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers), +- [Clue Web](https://lemurproject.org/clueweb12/), a dataset of 733,019,372 English web pages, +- [GigaWord](https://catalog.ldc.upenn.edu/LDC2011T07), an archive of newswire text data, +- [Common Crawl](https://commoncrawl.org/), a dataset of raw web pages. + + +### BibTeX entry and citation info + +```bibtex +@misc{dai2020funneltransformer, + title={Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing}, + author={Zihang Dai and Guokun Lai and Yiming Yang and Quoc V. Le}, + year={2020}, + eprint={2006.03236}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} +``` + diff --git a/model_cards/funnel-transformer/medium-base/README.md b/model_cards/funnel-transformer/medium-base/README.md new file mode 100644 index 0000000000..414c94eed9 --- /dev/null +++ b/model_cards/funnel-transformer/medium-base/README.md @@ -0,0 +1,94 @@ +--- +language: en +license: apache-2.0 +datasets: +- bookcorpus +- wikipedia +- gigaword +--- + +# Funnel Transformer medium model (B6-3x2-3x2 without decoder) + +Pretrained model on English language using a similar objective objective as [ELECTRA](https://huggingface.co/transformers/model_doc/electra.html). It was introduced in +[this paper](https://arxiv.org/pdf/2006.03236.pdf) and first released in +[this repository](https://github.com/laiguokun/Funnel-Transformer). This model is uncased: it does not make a difference +between english and English. + +Disclaimer: The team releasing Funnel Transformer did not write a model card for this model so this model card has been +written by the Hugging Face team. + +## Model description + +Funnel Transformer is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it +was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of +publicly available data) with an automatic process to generate inputs and labels from those texts. + +More precisely, a small language model corrupts the input texts and serves as a generator of inputs for this model, and +the pretraining objective is to predict which token is an original and which one has been replaced, a bit like a GAN training. + +This way, the model learns an inner representation of the English language that can then be used to extract features +useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard +classifier using the features produced by the BERT model as inputs. + +**Note:** This model does not contain the decoder, so it ouputs hidden states that have a sequence length of one fourth +of the inputs. It's good to use for tasks requiring a summary of the sentence (like sentence classification) but not if +you need one input per initial token. You should use the `medium` model in that case. + +## Intended uses & limitations + +You can use the raw model to extract a vector representation of a given text, but it's mostly intended to +be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=funnel-transformer) to look for +fine-tuned versions on a task that interests you. + +Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) +to make decisions, such as sequence classification, token classification or question answering. For tasks such as text +generation you should look at model like GPT2. + +### How to use + + +Here is how to use this model to get the features of a given text in PyTorch: + +```python +from transformers import FunnelTokenizer, FunnelBaseModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/medium-base") +model = FunnelBaseModel.from_pretrained("funnel-transformer/medium-base") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='pt') +output = model(**encoded_input) +``` + +and in TensorFlow: + +```python +from transformers import FunnelTokenizer, TFFunnelBaseModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/medium-base") +model = TFFunnelBaseModel.from_pretrained("funnel-transformer/medium-base") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='tf') +output = model(encoded_input) +``` + +## Training data + +The BERT model was pretrained on: +- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books, +- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers), +- [Clue Web](https://lemurproject.org/clueweb12/), a dataset of 733,019,372 English web pages, +- [GigaWord](https://catalog.ldc.upenn.edu/LDC2011T07), an archive of newswire text data, +- [Common Crawl](https://commoncrawl.org/), a dataset of raw web pages. + + +### BibTeX entry and citation info + +```bibtex +@misc{dai2020funneltransformer, + title={Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing}, + author={Zihang Dai and Guokun Lai and Yiming Yang and Quoc V. Le}, + year={2020}, + eprint={2006.03236}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} +``` + diff --git a/model_cards/funnel-transformer/medium/README.md b/model_cards/funnel-transformer/medium/README.md new file mode 100644 index 0000000000..d5db54b89d --- /dev/null +++ b/model_cards/funnel-transformer/medium/README.md @@ -0,0 +1,90 @@ +--- +language: en +license: apache-2.0 +datasets: +- bookcorpus +- wikipedia +- gigaword +--- + +# Funnel Transformer medium model (B6-3x2-3x2 with decoder) + +Pretrained model on English language using a similar objective objective as [ELECTRA](https://huggingface.co/transformers/model_doc/electra.html). It was introduced in +[this paper](https://arxiv.org/pdf/2006.03236.pdf) and first released in +[this repository](https://github.com/laiguokun/Funnel-Transformer). This model is uncased: it does not make a difference +between english and English. + +Disclaimer: The team releasing Funnel Transformer did not write a model card for this model so this model card has been +written by the Hugging Face team. + +## Model description + +Funnel Transformer is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it +was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of +publicly available data) with an automatic process to generate inputs and labels from those texts. + +More precisely, a small language model corrupts the input texts and serves as a generator of inputs for this model, and +the pretraining objective is to predict which token is an original and which one has been replaced, a bit like a GAN training. + +This way, the model learns an inner representation of the English language that can then be used to extract features +useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard +classifier using the features produced by the BERT model as inputs. + +## Intended uses & limitations + +You can use the raw model to extract a vector representation of a given text, but it's mostly intended to +be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=funnel-transformer) to look for +fine-tuned versions on a task that interests you. + +Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) +to make decisions, such as sequence classification, token classification or question answering. For tasks such as text +generation you should look at model like GPT2. + +### How to use + + +Here is how to use this model to get the features of a given text in PyTorch: + +```python +from transformers import FunnelTokenizer, FunnelModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/medium") +model = FunneModel.from_pretrained("funnel-transformer/medium") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='pt') +output = model(**encoded_input) +``` + +and in TensorFlow: + +```python +from transformers import FunnelTokenizer, TFFunnelModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/medium") +model = TFFunnelModel.from_pretrained("funnel-transformer/medium") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='tf') +output = model(encoded_input) +``` + +## Training data + +The BERT model was pretrained on: +- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books, +- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers), +- [Clue Web](https://lemurproject.org/clueweb12/), a dataset of 733,019,372 English web pages, +- [GigaWord](https://catalog.ldc.upenn.edu/LDC2011T07), an archive of newswire text data, +- [Common Crawl](https://commoncrawl.org/), a dataset of raw web pages. + + +### BibTeX entry and citation info + +```bibtex +@misc{dai2020funneltransformer, + title={Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing}, + author={Zihang Dai and Guokun Lai and Yiming Yang and Quoc V. Le}, + year={2020}, + eprint={2006.03236}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} +``` + diff --git a/model_cards/funnel-transformer/small-base/README.md b/model_cards/funnel-transformer/small-base/README.md new file mode 100644 index 0000000000..30ba16c717 --- /dev/null +++ b/model_cards/funnel-transformer/small-base/README.md @@ -0,0 +1,94 @@ +--- +language: en +license: apache-2.0 +datasets: +- bookcorpus +- wikipedia +- gigaword +--- + +# Funnel Transformer small model (B4-4-4 without decoder) + +Pretrained model on English language using a similar objective objective as [ELECTRA](https://huggingface.co/transformers/model_doc/electra.html). It was introduced in +[this paper](https://arxiv.org/pdf/2006.03236.pdf) and first released in +[this repository](https://github.com/laiguokun/Funnel-Transformer). This model is uncased: it does not make a difference +between english and English. + +Disclaimer: The team releasing Funnel Transformer did not write a model card for this model so this model card has been +written by the Hugging Face team. + +## Model description + +Funnel Transformer is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it +was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of +publicly available data) with an automatic process to generate inputs and labels from those texts. + +More precisely, a small language model corrupts the input texts and serves as a generator of inputs for this model, and +the pretraining objective is to predict which token is an original and which one has been replaced, a bit like a GAN training. + +This way, the model learns an inner representation of the English language that can then be used to extract features +useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard +classifier using the features produced by the BERT model as inputs. + +**Note:** This model does not contain the decoder, so it ouputs hidden states that have a sequence length of one fourth +of the inputs. It's good to use for tasks requiring a summary of the sentence (like sentence classification) but not if +you need one input per initial token. You should use the `small` model in that case. + +## Intended uses & limitations + +You can use the raw model to extract a vector representation of a given text, but it's mostly intended to +be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=funnel-transformer) to look for +fine-tuned versions on a task that interests you. + +Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) +to make decisions, such as sequence classification, token classification or question answering. For tasks such as text +generation you should look at model like GPT2. + +### How to use + + +Here is how to use this model to get the features of a given text in PyTorch: + +```python +from transformers import FunnelTokenizer, FunnelBaseModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/small-base") +model = FunnelBaseModel.from_pretrained("funnel-transformer/small-base") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='pt') +output = model(**encoded_input) +``` + +and in TensorFlow: + +```python +from transformers import FunnelTokenizer, TFFunnelBaseModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/small-base") +model = TFFunnelBaseModel.from_pretrained("funnel-transformer/small-base") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='tf') +output = model(encoded_input) +``` + +## Training data + +The BERT model was pretrained on: +- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books, +- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers), +- [Clue Web](https://lemurproject.org/clueweb12/), a dataset of 733,019,372 English web pages, +- [GigaWord](https://catalog.ldc.upenn.edu/LDC2011T07), an archive of newswire text data, +- [Common Crawl](https://commoncrawl.org/), a dataset of raw web pages. + + +### BibTeX entry and citation info + +```bibtex +@misc{dai2020funneltransformer, + title={Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing}, + author={Zihang Dai and Guokun Lai and Yiming Yang and Quoc V. Le}, + year={2020}, + eprint={2006.03236}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} +``` + diff --git a/model_cards/funnel-transformer/small/README.md b/model_cards/funnel-transformer/small/README.md new file mode 100644 index 0000000000..3a53ca0510 --- /dev/null +++ b/model_cards/funnel-transformer/small/README.md @@ -0,0 +1,90 @@ +--- +language: en +license: apache-2.0 +datasets: +- bookcorpus +- wikipedia +- gigaword +--- + +# Funnel Transformer small model (B4-4-4 with decoder) + +Pretrained model on English language using a similar objective objective as [ELECTRA](https://huggingface.co/transformers/model_doc/electra.html). It was introduced in +[this paper](https://arxiv.org/pdf/2006.03236.pdf) and first released in +[this repository](https://github.com/laiguokun/Funnel-Transformer). This model is uncased: it does not make a difference +between english and English. + +Disclaimer: The team releasing Funnel Transformer did not write a model card for this model so this model card has been +written by the Hugging Face team. + +## Model description + +Funnel Transformer is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it +was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of +publicly available data) with an automatic process to generate inputs and labels from those texts. + +More precisely, a small language model corrupts the input texts and serves as a generator of inputs for this model, and +the pretraining objective is to predict which token is an original and which one has been replaced, a bit like a GAN training. + +This way, the model learns an inner representation of the English language that can then be used to extract features +useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard +classifier using the features produced by the BERT model as inputs. + +## Intended uses & limitations + +You can use the raw model to extract a vector representation of a given text, but it's mostly intended to +be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=funnel-transformer) to look for +fine-tuned versions on a task that interests you. + +Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) +to make decisions, such as sequence classification, token classification or question answering. For tasks such as text +generation you should look at model like GPT2. + +### How to use + + +Here is how to use this model to get the features of a given text in PyTorch: + +```python +from transformers import FunnelTokenizer, FunnelModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/small") +model = FunneModel.from_pretrained("funnel-transformer/small") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='pt') +output = model(**encoded_input) +``` + +and in TensorFlow: + +```python +from transformers import FunnelTokenizer, TFFunnelModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/small") +model = TFFunnelModel.from_pretrained("funnel-transformer/small") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='tf') +output = model(encoded_input) +``` + +## Training data + +The BERT model was pretrained on: +- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books, +- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers), +- [Clue Web](https://lemurproject.org/clueweb12/), a dataset of 733,019,372 English web pages, +- [GigaWord](https://catalog.ldc.upenn.edu/LDC2011T07), an archive of newswire text data, +- [Common Crawl](https://commoncrawl.org/), a dataset of raw web pages. + + +### BibTeX entry and citation info + +```bibtex +@misc{dai2020funneltransformer, + title={Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing}, + author={Zihang Dai and Guokun Lai and Yiming Yang and Quoc V. Le}, + year={2020}, + eprint={2006.03236}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} +``` + diff --git a/model_cards/funnel-transformer/xlarge-base/README.md b/model_cards/funnel-transformer/xlarge-base/README.md new file mode 100644 index 0000000000..7461fb5112 --- /dev/null +++ b/model_cards/funnel-transformer/xlarge-base/README.md @@ -0,0 +1,94 @@ +--- +language: en +license: apache-2.0 +datasets: +- bookcorpus +- wikipedia +- gigaword +--- + +# Funnel Transformer xlarge model (B10-10-10 without decoder) + +Pretrained model on English language using a similar objective objective as [ELECTRA](https://huggingface.co/transformers/model_doc/electra.html). It was introduced in +[this paper](https://arxiv.org/pdf/2006.03236.pdf) and first released in +[this repository](https://github.com/laiguokun/Funnel-Transformer). This model is uncased: it does not make a difference +between english and English. + +Disclaimer: The team releasing Funnel Transformer did not write a model card for this model so this model card has been +written by the Hugging Face team. + +## Model description + +Funnel Transformer is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it +was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of +publicly available data) with an automatic process to generate inputs and labels from those texts. + +More precisely, a small language model corrupts the input texts and serves as a generator of inputs for this model, and +the pretraining objective is to predict which token is an original and which one has been replaced, a bit like a GAN training. + +This way, the model learns an inner representation of the English language that can then be used to extract features +useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard +classifier using the features produced by the BERT model as inputs. + +**Note:** This model does not contain the decoder, so it ouputs hidden states that have a sequence length of one fourth +of the inputs. It's good to use for tasks requiring a summary of the sentence (like sentence classification) but not if +you need one input per initial token. You should use the `xlarge` model in that case. + +## Intended uses & limitations + +You can use the raw model to extract a vector representation of a given text, but it's mostly intended to +be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=funnel-transformer) to look for +fine-tuned versions on a task that interests you. + +Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) +to make decisions, such as sequence classification, token classification or question answering. For tasks such as text +generation you should look at model like GPT2. + +### How to use + + +Here is how to use this model to get the features of a given text in PyTorch: + +```python +from transformers import FunnelTokenizer, FunnelBaseModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/xlarge-base") +model = FunnelBaseModel.from_pretrained("funnel-transformer/xlarge-base") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='pt') +output = model(**encoded_input) +``` + +and in TensorFlow: + +```python +from transformers import FunnelTokenizer, TFFunnelBaseModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/xlarge-base") +model = TFFunnelBaseModel.from_pretrained("funnel-transformer/xlarge-base") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='tf') +output = model(encoded_input) +``` + +## Training data + +The BERT model was pretrained on: +- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books, +- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers), +- [Clue Web](https://lemurproject.org/clueweb12/), a dataset of 733,019,372 English web pages, +- [GigaWord](https://catalog.ldc.upenn.edu/LDC2011T07), an archive of newswire text data, +- [Common Crawl](https://commoncrawl.org/), a dataset of raw web pages. + + +### BibTeX entry and citation info + +```bibtex +@misc{dai2020funneltransformer, + title={Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing}, + author={Zihang Dai and Guokun Lai and Yiming Yang and Quoc V. Le}, + year={2020}, + eprint={2006.03236}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} +``` + diff --git a/model_cards/funnel-transformer/xlarge/README.md b/model_cards/funnel-transformer/xlarge/README.md new file mode 100644 index 0000000000..a1a0a69f24 --- /dev/null +++ b/model_cards/funnel-transformer/xlarge/README.md @@ -0,0 +1,90 @@ +--- +language: en +license: apache-2.0 +datasets: +- bookcorpus +- wikipedia +- gigaword +--- + +# Funnel Transformer xlarge model (B10-10-10 with decoder) + +Pretrained model on English language using a similar objective objective as [ELECTRA](https://huggingface.co/transformers/model_doc/electra.html). It was introduced in +[this paper](https://arxiv.org/pdf/2006.03236.pdf) and first released in +[this repository](https://github.com/laiguokun/Funnel-Transformer). This model is uncased: it does not make a difference +between english and English. + +Disclaimer: The team releasing Funnel Transformer did not write a model card for this model so this model card has been +written by the Hugging Face team. + +## Model description + +Funnel Transformer is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it +was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of +publicly available data) with an automatic process to generate inputs and labels from those texts. + +More precisely, a small language model corrupts the input texts and serves as a generator of inputs for this model, and +the pretraining objective is to predict which token is an original and which one has been replaced, a bit like a GAN training. + +This way, the model learns an inner representation of the English language that can then be used to extract features +useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard +classifier using the features produced by the BERT model as inputs. + +## Intended uses & limitations + +You can use the raw model to extract a vector representation of a given text, but it's mostly intended to +be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=funnel-transformer) to look for +fine-tuned versions on a task that interests you. + +Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) +to make decisions, such as sequence classification, token classification or question answering. For tasks such as text +generation you should look at model like GPT2. + +### How to use + + +Here is how to use this model to get the features of a given text in PyTorch: + +```python +from transformers import FunnelTokenizer, FunnelModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/xlarge") +model = FunneModel.from_pretrained("funnel-transformer/xlarge") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='pt') +output = model(**encoded_input) +``` + +and in TensorFlow: + +```python +from transformers import FunnelTokenizer, TFFunnelModel +tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/xlarge") +model = TFFunnelModel.from_pretrained("funnel-transformer/xlarge") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='tf') +output = model(encoded_input) +``` + +## Training data + +The BERT model was pretrained on: +- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books, +- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers), +- [Clue Web](https://lemurproject.org/clueweb12/), a dataset of 733,019,372 English web pages, +- [GigaWord](https://catalog.ldc.upenn.edu/LDC2011T07), an archive of newswire text data, +- [Common Crawl](https://commoncrawl.org/), a dataset of raw web pages. + + +### BibTeX entry and citation info + +```bibtex +@misc{dai2020funneltransformer, + title={Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing}, + author={Zihang Dai and Guokun Lai and Yiming Yang and Quoc V. Le}, + year={2020}, + eprint={2006.03236}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} +``` +