Add LongT5 model (#16792)
* Initial commit * Make some fixes * Make PT model full forward pass * Drop TF & Flax implementation, fix copies etc * Add Flax model and update some corresponding stuff * Drop some TF things * Update config and flax local attn * Add encoder_attention_type to config * . * Update docs * Do some cleansing * Fix some issues -> make style; add some docs * Fix position_bias + mask addition + Update tests * Fix repo consistency * Fix model consistency by removing flax operation over attn_mask * [WIP] Add PT TGlobal LongT5 * . * [WIP] Add flax tglobal model * [WIP] Update flax model to use the right attention type in the encoder * Fix flax tglobal model forward pass * Make the use of global_relative_attention_bias * Add test suites for TGlobal model * Fix minor bugs, clean code * Fix pt-flax equivalence though not convinced with correctness * Fix LocalAttn implementation to match the original impl. + update READMEs * Few updates * Update: [Flax] improve large model init and loading #16148 * Add ckpt conversion script accoring to #16853 + handle torch device placement * Minor updates to conversion script. * Typo: AutoModelForSeq2SeqLM -> FlaxAutoModelForSeq2SeqLM * gpu support + dtype fix * Apply some suggestions from code review Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * * Remove (de)parallelize stuff * Edit shape comments * Update README.md * make fix-copies * Remove caching logic for local & tglobal attention * Apply another batch of suggestions from code review * Add missing checkpoints * Format converting scripts * Drop (de)parallelize links from longT5 mdx * Fix converting script + revert config file change * Revert "Remove caching logic for local & tglobal attention" This reverts commit 2a619828f6ddc3e65bd9bb1725a12b77fa883a46. * Stash caching logic in Flax model * Make side relative bias used always * Drop caching logic in PT model * Return side bias as it was * Drop all remaining model parallel logic * Remove clamp statements * Move test files to the proper place * Update docs with new version of hf-doc-builder * Fix test imports * Make some minor improvements * Add missing checkpoints to docs * Make TGlobal model compatible with torch.onnx.export * Replace some np.ndarray with jnp.ndarray * Fix TGlobal for ONNX conversion + update docs * fix _make_global_fixed_block_ids and masked neg value * update flax model * style and quality * fix imports * remove load_tf_weights_in_longt5 from init and fix copies * add slow test for TGlobal model * typo fix * Drop obsolete is_parallelizable and one warning * Update __init__ files to fix repo-consistency * fix pipeline test * Fix some device placements * [wip]: Update tests -- need to generate summaries to update expected_summary * Fix quality * Update LongT5 model card * Update (slow) summarization tests * make style * rename checkpoitns * finish * fix flax tests Co-authored-by: phungvanduy <pvduy23@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: patil-suraj <surajp815@gmail.com>
This commit is contained in:
Daniel Stancl
@@ -256,6 +256,8 @@
|
||||
title: LeViT
|
||||
- local: model_doc/longformer
|
||||
title: Longformer
|
||||
- local: model_doc/longt5
|
||||
title: LongT5
|
||||
- local: model_doc/luke
|
||||
title: LUKE
|
||||
- local: model_doc/lxmert
|
||||
|
||||
@@ -107,6 +107,7 @@ The library currently contains JAX, PyTorch and TensorFlow implementations, pret
|
||||
1. **[LED](model_doc/led)** (from AllenAI) released with the paper [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150) by Iz Beltagy, Matthew E. Peters, Arman Cohan.
|
||||
1. **[LeViT](model_doc/levit)** (from Meta AI) released with the paper [LeViT: A Vision Transformer in ConvNet's Clothing for Faster Inference](https://arxiv.org/abs/2104.01136) by Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, Matthijs Douze.
|
||||
1. **[Longformer](model_doc/longformer)** (from AllenAI) released with the paper [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150) by Iz Beltagy, Matthew E. Peters, Arman Cohan.
|
||||
1. **[LongT5](model_doc/longt5)** (from Google AI) released with the paper [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://arxiv.org/abs/2112.07916) by Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang.
|
||||
1. **[LUKE](model_doc/luke)** (from Studio Ousia) released with the paper [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://arxiv.org/abs/2010.01057) by Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, Yuji Matsumoto.
|
||||
1. **[LXMERT](model_doc/lxmert)** (from UNC Chapel Hill) released with the paper [LXMERT: Learning Cross-Modality Encoder Representations from Transformers for Open-Domain Question Answering](https://arxiv.org/abs/1908.07490) by Hao Tan and Mohit Bansal.
|
||||
1. **[M-CTC-T](model_doc/mctct)** (from Facebook) released with the paper [Pseudo-Labeling For Massively Multilingual Speech Recognition](https://arxiv.org/abs/2111.00161) by Loren Lugosch, Tatiana Likhomanenko, Gabriel Synnaeve, and Ronan Collobert.
|
||||
@@ -233,6 +234,7 @@ Flax), PyTorch, and/or TensorFlow.
|
||||
| LED | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| LeViT | ❌ | ❌ | ✅ | ❌ | ❌ |
|
||||
| Longformer | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| LongT5 | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| LUKE | ✅ | ❌ | ✅ | ❌ | ❌ |
|
||||
| LXMERT | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| M-CTC-T | ❌ | ❌ | ✅ | ❌ | ❌ |
|
||||
|
||||
121
docs/source/en/model_doc/longt5.mdx
Normal file
121
docs/source/en/model_doc/longt5.mdx
Normal file
@@ -0,0 +1,121 @@
|
||||
<!--Copyright 2022 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# LongT5
|
||||
|
||||
## Overview
|
||||
|
||||
The LongT5 model was proposed in [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://arxiv.org/abs/2112.07916)
|
||||
by Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung and Yinfei Yang. It's an
|
||||
encoder-decoder transformer pre-trained in a text-to-text denoising generative setting. LongT5 model is an extension of
|
||||
T5 model, and it enables using one of the two different efficient attention mechanisms - (1) Local attention, or (2)
|
||||
Transient-Global attention.
|
||||
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the
|
||||
performance of Transformer-based neural models. In this paper, we present a new model, called LongT5, with which we
|
||||
explore the effects of scaling both the input length and model size at the same time. Specifically, we integrated
|
||||
attention ideas from long-input transformers (ETC), and adopted pre-training strategies from summarization pre-training
|
||||
(PEGASUS) into the scalable T5 architecture. The result is a new attention mechanism we call {\em Transient Global}
|
||||
(TGlobal), which mimics ETC's local/global attention mechanism, but without requiring additional side-inputs. We are
|
||||
able to achieve state-of-the-art results on several summarization tasks and outperform the original T5 models on
|
||||
question answering tasks.*
|
||||
|
||||
Tips:
|
||||
|
||||
- [`LongT5ForConditionalGeneration`] is an extension of [`T5ForConditionalGeneration`] exchanging the traditional
|
||||
encoder *self-attention* layer with efficient either *local* attention or *transient-global* (*tglobal*) attention.
|
||||
- Unlike the T5 model, LongT5 does not use a task prefix. Furthermore, it uses a different pre-training objective
|
||||
inspired by the pre-training of `[PegasusForConditionalGeneration]`.
|
||||
- LongT5 model is designed to work efficiently and very well on long-range *sequence-to-sequence* tasks where the
|
||||
input sequence exceeds commonly used 512 tokens. It is capable of handling input sequences of a length up to 16,384 tokens.
|
||||
- For *Local Attention*, the sparse sliding-window local attention operation allows a given token to attend only `r`
|
||||
tokens to the left and right of it (with `r=127` by default). *Local Attention* does not introduce any new parameters
|
||||
to the model. The complexity of the mechanism is linear in input sequence length `l`: `O(l*r)`.
|
||||
- *Transient Global Attention* is an extension of the *Local Attention*. It, furthermore, allows each input token to
|
||||
interact with all other tokens in the layer. This is achieved via splitting an input sequence into blocks of a fixed
|
||||
length `k` (with a default `k=16`). Then, a global token for such a block is obtained via summing and normalizing the embeddings of every token
|
||||
in the block. Thanks to this, the attention allows each token to attend to both nearby tokens like in Local attention, and
|
||||
also every global token like in the case of standard global attention (*transient* represents the fact the global tokens
|
||||
are constructed dynamically within each attention operation). As a consequence, *TGlobal* attention introduces
|
||||
a few new parameters -- global relative position biases and a layer normalization for global token's embedding.
|
||||
The complexity of this mechanism is `O(l(r + l/k))`.
|
||||
- An example showing how to evaluate a fine-tuned LongT5 model on the [pubmed dataset](https://huggingface.co/datasets/scientific_papers) is below.
|
||||
|
||||
```python
|
||||
>>> import evaluate
|
||||
>>> from datasets import load_dataset
|
||||
>>> from transformers import AutoTokenizer, LongT5ForConditionalGeneration
|
||||
|
||||
>>> dataset = load_dataset("scientific_papers", "pubmed", split="validation")
|
||||
>>> model = (
|
||||
... LongT5ForConditionalGeneration.from_pretrained("Stancld/longt5-tglobal-large-16384-pubmed-3k_steps")
|
||||
... .to("cuda")
|
||||
... .half()
|
||||
... )
|
||||
>>> tokenizer = AutoTokenizer.from_pretrained("Stancld/longt5-tglobal-large-16384-pubmed-3k_steps")
|
||||
|
||||
|
||||
>>> def generate_answers(batch):
|
||||
... inputs_dict = tokenizer(
|
||||
... batch["article"], max_length=16384, padding="max_length", truncation=True, return_tensors="pt"
|
||||
... )
|
||||
... input_ids = inputs_dict.input_ids.to("cuda")
|
||||
... attention_mask = inputs_dict.attention_mask.to("cuda")
|
||||
... output_ids = model.generate(input_ids, attention_mask=attention_mask, max_length=512, num_beams=2)
|
||||
... batch["predicted_abstract"] = tokenizer.batch_decode(output_ids, skip_special_tokens=True)
|
||||
... return batch
|
||||
|
||||
|
||||
>>> result = dataset.map(generate_answer, batched=True, batch_size=2)
|
||||
>>> rouge = evaluate.load("rouge")
|
||||
>>> rouge.compute(predictions=result["predicted_abstract"], references=result["abstract"])
|
||||
```
|
||||
|
||||
This model was contributed by [stancld](https://huggingface.co/stancld).
|
||||
The original code can be found [here](https://github.com/google-research/longt5).
|
||||
|
||||
|
||||
## LongT5Config
|
||||
|
||||
[[autodoc]] LongT5Config
|
||||
|
||||
## LongT5Model
|
||||
|
||||
[[autodoc]] LongT5Model
|
||||
- forward
|
||||
|
||||
## LongT5ForConditionalGeneration
|
||||
|
||||
[[autodoc]] LongT5ForConditionalGeneration
|
||||
- forward
|
||||
|
||||
## LongT5EncoderModel
|
||||
|
||||
[[autodoc]] LongT5EncoderModel
|
||||
- forward
|
||||
|
||||
## FlaxLongT5Model
|
||||
|
||||
[[autodoc]] FlaxLongT5Model
|
||||
- __call__
|
||||
- encode
|
||||
- decode
|
||||
|
||||
## FlaxLongT5ForConditionalGeneration
|
||||
|
||||
[[autodoc]] FlaxLongT5ForConditionalGeneration
|
||||
- __call__
|
||||
- encode
|
||||
- decode
|
||||
@@ -66,6 +66,7 @@ Ready-made configurations include the following architectures:
|
||||
- GPT-J
|
||||
- I-BERT
|
||||
- LayoutLM
|
||||
- LongT5
|
||||
- M2M100
|
||||
- Marian
|
||||
- mBART
|
||||
|
||||
Reference in New Issue
Block a user