diff --git a/README.md b/README.md index 853353ecc3..3e3d261226 100644 --- a/README.md +++ b/README.md @@ -363,6 +363,7 @@ Current number of checkpoints: ![](https://img.shields.io/endpoint?url=https://h 1. **[FNet](https://huggingface.co/docs/transformers/model_doc/fnet)** (from Google Research) released with the paper [FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/abs/2105.03824) by James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. 1. **[FocalNet](https://huggingface.co/docs/transformers/model_doc/focalnet)** (from Microsoft Research) released with the paper [Focal Modulation Networks](https://arxiv.org/abs/2203.11926) by Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, Jianfeng Gao. 1. **[Funnel Transformer](https://huggingface.co/docs/transformers/model_doc/funnel)** (from CMU/Google Brain) released with the paper [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://arxiv.org/abs/2006.03236) by Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le. +1. **[Fuyu](https://huggingface.co/docs/transformers/model_doc/fuyu)** (from ADEPT) Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, Sağnak Taşırlar. Released with the paper [blog post](https://www.adept.ai/blog/fuyu-8b) 1. **[GIT](https://huggingface.co/docs/transformers/model_doc/git)** (from Microsoft Research) released with the paper [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) by Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. 1. **[GLPN](https://huggingface.co/docs/transformers/model_doc/glpn)** (from KAIST) released with the paper [Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth](https://arxiv.org/abs/2201.07436) by Doyeon Kim, Woonghyun Ga, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, Junmo Kim. 1. **[GPT](https://huggingface.co/docs/transformers/model_doc/openai-gpt)** (from OpenAI) released with the paper [Improving Language Understanding by Generative Pre-Training](https://openai.com/research/language-unsupervised/) by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. diff --git a/README_es.md b/README_es.md index e74485a2fc..dee98fa4ae 100644 --- a/README_es.md +++ b/README_es.md @@ -338,6 +338,7 @@ Número actual de puntos de control: ![](https://img.shields.io/endpoint?url=htt 1. **[FNet](https://huggingface.co/docs/transformers/model_doc/fnet)** (from Google Research) released with the paper [FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/abs/2105.03824) by James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. 1. **[FocalNet](https://huggingface.co/docs/transformers/model_doc/focalnet)** (from Microsoft Research) released with the paper [Focal Modulation Networks](https://arxiv.org/abs/2203.11926) by Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, Jianfeng Gao. 1. **[Funnel Transformer](https://huggingface.co/docs/transformers/model_doc/funnel)** (from CMU/Google Brain) released with the paper [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://arxiv.org/abs/2006.03236) by Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le. +1. **[Fuyu](https://huggingface.co/docs/transformers/model_doc/fuyu)** (from ADEPT) Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, Sağnak Taşırlar. Released with the paper [blog post](https://www.adept.ai/blog/fuyu-8b) 1. **[GIT](https://huggingface.co/docs/transformers/model_doc/git)** (from Microsoft Research) released with the paper [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) by Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. 1. **[GLPN](https://huggingface.co/docs/transformers/model_doc/glpn)** (from KAIST) released with the paper [Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth](https://arxiv.org/abs/2201.07436) by Doyeon Kim, Woonghyun Ga, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, Junmo Kim. 1. **[GPT](https://huggingface.co/docs/transformers/model_doc/openai-gpt)** (from OpenAI) released with the paper [Improving Language Understanding by Generative Pre-Training](https://blog.openai.com/language-unsupervised/) by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. diff --git a/README_hd.md b/README_hd.md index 96c70ce393..895d440f33 100644 --- a/README_hd.md +++ b/README_hd.md @@ -310,6 +310,7 @@ conda install -c huggingface transformers 1. **[FNet](https://huggingface.co/docs/transformers/model_doc/fnet)** (गूगल रिसर्च से) साथ वाला पेपर [FNet: मिक्सिंग टोकन विद फूरियर ट्रांसफॉर्म्स](https://arxiv.org /abs/2105.03824) जेम्स ली-थॉर्प, जोशुआ आइंस्ली, इल्या एकस्टीन, सैंटियागो ओंटानन द्वारा। 1. **[FocalNet](https://huggingface.co/docs/transformers/model_doc/focalnet)** (Microsoft Research से) Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, Jianfeng Gao. द्वाराअनुसंधान पत्र [Focal Modulation Networks](https://arxiv.org/abs/2203.11926) के साथ जारी किया गया 1. **[Funnel Transformer](https://huggingface.co/docs/transformers/model_doc/funnel)** (सीएमयू/गूगल ब्रेन से) साथ में कागज [फ़नल-ट्रांसफॉर्मर: कुशल भाषा प्रसंस्करण के लिए अनुक्रमिक अतिरेक को छानना](https://arxiv.org/abs/2006.03236) जिहांग दाई, गुओकुन लाई, यिमिंग यांग, क्वोक वी. ले ​​द्वारा रिहाई। +1. **[Fuyu](https://huggingface.co/docs/transformers/model_doc/fuyu)** (ADEPT से) रोहन बाविशी, एरिच एलसेन, कर्टिस हॉथोर्न, मैक्सवेल नी, ऑगस्टस ओडेना, अरुशी सोमानी, सागनाक तासिरलार [blog post](https://www.adept.ai/blog/fuyu-8b) 1. **[GIT](https://huggingface.co/docs/transformers/model_doc/git)** (from Microsoft Research) released with the paper [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) by Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. 1. **[GLPN](https://huggingface.co/docs/transformers/model_doc/glpn)** (KAIST से) साथ वाला पेपर [वर्टिकल कटडेप्थ के साथ मोनोकुलर डेप्थ एस्टीमेशन के लिए ग्लोबल-लोकल पाथ नेटवर्क्स](https:/ /arxiv.org/abs/2201.07436) डोयोन किम, वूंगह्युन गा, प्युंगवान आह, डोंगग्यू जू, सेहवान चुन, जुनमो किम द्वारा। 1. **[GPT](https://huggingface.co/docs/transformers/model_doc/openai-gpt)** (OpenAI से) साथ में दिया गया पेपर [जेनरेटिव प्री-ट्रेनिंग द्वारा भाषा की समझ में सुधार](https://blog .openai.com/language-unsupervised/) एलेक रैडफोर्ड, कार्तिक नरसिम्हन, टिम सालिमन्स और इल्या सुत्स्केवर द्वारा। diff --git a/README_ja.md b/README_ja.md index 108755ba18..0944a9a36b 100644 --- a/README_ja.md +++ b/README_ja.md @@ -372,6 +372,7 @@ Flax、PyTorch、TensorFlowをcondaでインストールする方法は、それ 1. **[FNet](https://huggingface.co/docs/transformers/model_doc/fnet)** (Google Research から) James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon から公開された研究論文: [FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/abs/2105.03824) 1. **[FocalNet](https://huggingface.co/docs/transformers/model_doc/focalnet)** (Microsoft Research から) Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, Jianfeng Gao. から公開された研究論文 [Focal Modulation Networks](https://arxiv.org/abs/2203.11926) 1. **[Funnel Transformer](https://huggingface.co/docs/transformers/model_doc/funnel)** (CMU/Google Brain から) Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le から公開された研究論文: [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://arxiv.org/abs/2006.03236) +1. **[Fuyu](https://huggingface.co/docs/transformers/model_doc/fuyu)** (ADEPT から) Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, Sağnak Taşırlar. から公開された研究論文 [blog post](https://www.adept.ai/blog/fuyu-8b) 1. **[GIT](https://huggingface.co/docs/transformers/model_doc/git)** (Microsoft Research から) Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. から公開された研究論文 [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) 1. **[GLPN](https://huggingface.co/docs/transformers/model_doc/glpn)** (KAIST から) Doyeon Kim, Woonghyun Ga, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, Junmo Kim から公開された研究論文: [Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth](https://arxiv.org/abs/2201.07436) 1. **[GPT](https://huggingface.co/docs/transformers/model_doc/openai-gpt)** (OpenAI から) Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever から公開された研究論文: [Improving Language Understanding by Generative Pre-Training](https://blog.openai.com/language-unsupervised/) diff --git a/README_ko.md b/README_ko.md index 60a46aefe5..2e325be61c 100644 --- a/README_ko.md +++ b/README_ko.md @@ -287,6 +287,7 @@ Flax, PyTorch, TensorFlow 설치 페이지에서 이들을 conda로 설치하는 1. **[FNet](https://huggingface.co/docs/transformers/model_doc/fnet)** (from Google Research) released with the paper [FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/abs/2105.03824) by James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. 1. **[FocalNet](https://huggingface.co/docs/transformers/model_doc/focalnet)** (from Microsoft Research) released with the paper [Focal Modulation Networks](https://arxiv.org/abs/2203.11926) by Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, Jianfeng Gao. 1. **[Funnel Transformer](https://huggingface.co/docs/transformers/model_doc/funnel)** (from CMU/Google Brain) released with the paper [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://arxiv.org/abs/2006.03236) by Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le. +1. **[Fuyu](https://huggingface.co/docs/transformers/model_doc/fuyu)** (from ADEPT) Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, Sağnak Taşırlar. 논문과 함께 공개 [blog post](https://www.adept.ai/blog/fuyu-8b) 1. **[GIT](https://huggingface.co/docs/transformers/model_doc/git)** (from Microsoft Research) released with the paper [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) by Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. 1. **[GLPN](https://huggingface.co/docs/transformers/model_doc/glpn)** (from KAIST) released with the paper [Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth](https://arxiv.org/abs/2201.07436) by Doyeon Kim, Woonghyun Ga, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, Junmo Kim. 1. **[GPT](https://huggingface.co/docs/transformers/model_doc/openai-gpt)** (from OpenAI) released with the paper [Improving Language Understanding by Generative Pre-Training](https://blog.openai.com/language-unsupervised/) by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. diff --git a/README_ru.md b/README_ru.md index 8a15bf871e..84184733e6 100644 --- a/README_ru.md +++ b/README_ru.md @@ -361,6 +361,7 @@ conda install -c huggingface transformers 1. **[FNet](https://huggingface.co/docs/transformers/model_doc/fnet)** (from Google Research) released with the paper [FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/abs/2105.03824) by James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. 1. **[FocalNet](https://huggingface.co/docs/transformers/model_doc/focalnet)** (from Microsoft Research) released with the paper [Focal Modulation Networks](https://arxiv.org/abs/2203.11926) by Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, Jianfeng Gao. 1. **[Funnel Transformer](https://huggingface.co/docs/transformers/model_doc/funnel)** (from CMU/Google Brain) released with the paper [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://arxiv.org/abs/2006.03236) by Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le. +1. **[Fuyu](https://huggingface.co/docs/transformers/model_doc/fuyu)** (from ADEPT) Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, Sağnak Taşırlar. Released with the paper [blog post](https://www.adept.ai/blog/fuyu-8b) 1. **[GIT](https://huggingface.co/docs/transformers/model_doc/git)** (from Microsoft Research) released with the paper [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) by Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. 1. **[GLPN](https://huggingface.co/docs/transformers/model_doc/glpn)** (from KAIST) released with the paper [Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth](https://arxiv.org/abs/2201.07436) by Doyeon Kim, Woonghyun Ga, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, Junmo Kim. 1. **[GPT](https://huggingface.co/docs/transformers/model_doc/openai-gpt)** (from OpenAI) released with the paper [Improving Language Understanding by Generative Pre-Training](https://blog.openai.com/language-unsupervised/) by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. diff --git a/README_zh-hans.md b/README_zh-hans.md index 7b55646bb2..53bef31231 100644 --- a/README_zh-hans.md +++ b/README_zh-hans.md @@ -311,6 +311,7 @@ conda install -c huggingface transformers 1. **[FNet](https://huggingface.co/docs/transformers/model_doc/fnet)** (来自 Google Research) 伴随论文 [FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/abs/2105.03824) 由 James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon 发布。 1. **[FocalNet](https://huggingface.co/docs/transformers/model_doc/focalnet)** (来自 Microsoft Research) 伴随论文 [Focal Modulation Networks](https://arxiv.org/abs/2203.11926) 由 Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, Jianfeng Gao 发布。 1. **[Funnel Transformer](https://huggingface.co/docs/transformers/model_doc/funnel)** (来自 CMU/Google Brain) 伴随论文 [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://arxiv.org/abs/2006.03236) 由 Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le 发布。 +1. **[Fuyu](https://huggingface.co/docs/transformers/model_doc/fuyu)** (来自 ADEPT) 伴随论文 [blog post](https://www.adept.ai/blog/fuyu-8b 由 Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, Sağnak Taşırlar 发布。) 1. **[GIT](https://huggingface.co/docs/transformers/model_doc/git)** (来自 Microsoft Research) 伴随论文 [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) 由 Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang 发布。 1. **[GLPN](https://huggingface.co/docs/transformers/model_doc/glpn)** (来自 KAIST) 伴随论文 [Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth](https://arxiv.org/abs/2201.07436) 由 Doyeon Kim, Woonghyun Ga, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, Junmo Kim 发布。 1. **[GPT](https://huggingface.co/docs/transformers/model_doc/openai-gpt)** (来自 OpenAI) 伴随论文 [Improving Language Understanding by Generative Pre-Training](https://blog.openai.com/language-unsupervised/) 由 Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever 发布。 diff --git a/README_zh-hant.md b/README_zh-hant.md index 15f56c6688..5ba42b77ef 100644 --- a/README_zh-hant.md +++ b/README_zh-hant.md @@ -323,6 +323,7 @@ conda install -c huggingface transformers 1. **[FNet](https://huggingface.co/docs/transformers/model_doc/fnet)** (from Google Research) released with the paper [FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/abs/2105.03824) by James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. 1. **[FocalNet](https://huggingface.co/docs/transformers/model_doc/focalnet)** (from Microsoft Research) released with the paper [Focal Modulation Networks](https://arxiv.org/abs/2203.11926) by Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, Jianfeng Gao. 1. **[Funnel Transformer](https://huggingface.co/docs/transformers/model_doc/funnel)** (from CMU/Google Brain) released with the paper [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://arxiv.org/abs/2006.03236) by Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le. +1. **[Fuyu](https://huggingface.co/docs/transformers/model_doc/fuyu)** (from ADEPT) Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, Sağnak Taşırlar. Released with the paper [blog post](https://www.adept.ai/blog/fuyu-8b) 1. **[GIT](https://huggingface.co/docs/transformers/model_doc/git)** (from Microsoft Research) released with the paper [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) by Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. 1. **[GLPN](https://huggingface.co/docs/transformers/model_doc/glpn)** (from KAIST) released with the paper [Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth](https://arxiv.org/abs/2201.07436) by Doyeon Kim, Woonghyun Ga, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, Junmo Kim. 1. **[GPT](https://huggingface.co/docs/transformers/model_doc/openai-gpt)** (from OpenAI) released with the paper [Improving Language Understanding by Generative Pre-Training](https://blog.openai.com/language-unsupervised/) by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. diff --git a/docs/source/en/_toctree.yml b/docs/source/en/_toctree.yml index e57d45f9a0..11fa956db8 100644 --- a/docs/source/en/_toctree.yml +++ b/docs/source/en/_toctree.yml @@ -342,6 +342,8 @@ title: FSMT - local: model_doc/funnel title: Funnel Transformer + - local: model_doc/fuyu + title: Fuyu - local: model_doc/openai-gpt title: GPT - local: model_doc/gpt_neo diff --git a/docs/source/en/index.md b/docs/source/en/index.md index a1fbc63c7c..97fdcf55e1 100644 --- a/docs/source/en/index.md +++ b/docs/source/en/index.md @@ -138,6 +138,7 @@ Flax), PyTorch, and/or TensorFlow. | [FNet](model_doc/fnet) | ✅ | ❌ | ❌ | | [FocalNet](model_doc/focalnet) | ✅ | ❌ | ❌ | | [Funnel Transformer](model_doc/funnel) | ✅ | ✅ | ❌ | +| [Fuyu](model_doc/fuyu) | ✅ | ❌ | ❌ | | [GIT](model_doc/git) | ✅ | ❌ | ❌ | | [GLPN](model_doc/glpn) | ✅ | ❌ | ❌ | | [GPT Neo](model_doc/gpt_neo) | ✅ | ❌ | ✅ | diff --git a/docs/source/en/model_doc/fuyu.md b/docs/source/en/model_doc/fuyu.md new file mode 100644 index 0000000000..9d24e9951b --- /dev/null +++ b/docs/source/en/model_doc/fuyu.md @@ -0,0 +1,115 @@ + + +# Fuyu + +## Overview + +The Fuyu model was created by [ADEPT](https://www.adept.ai/blog/fuyu-8b), and authored by Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, Sağnak Taşırlar. + +The authors introduced Fuyu-8B, a decoder-only multimodal model based on the classic transformers architecture, with query and key normalization. A linear encoder is added to create multimodal embeddings from image inputs. + +By treating image tokens like text tokens and using a special image-newline character, the model knows when an image line ends. Image positional embeddings are removed. This avoids the need for different training phases for various image resolutions. With 8 billion parameters and licensed under Apache, Fuyu-8B is notable for its ability to handle both text and images, its impressive context size of 16K, and its overall performance. + + + +The `Fuyu` models were trained using `bfloat16`, but the original inference uses `float16` The checkpoints uploaded on the hub use `torch_dtype = 'float16'` which will be +used by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. + +The `dtype` of the online weights is mostly irrelevant, unless you are using `torch_dtype="auto"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained("path", torch_dtype = "auto")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online) then it will be cast to the default `dtype` of `torch` (becomes `torch.float32`). Users should specify the `torch_dtype` they want, and if they don't it will be `torch.float32`. + +Finetuning the model in `float16` is not recommended and known to produce `nan`, as such the model should be fine-tuned in `bfloat16`. + + + + +Tips: + +- To convert the model, you need to clone the original repository using `git clone https://github.com/persimmon-ai-labs/adept-inference`, then get the checkpoints: + +```bash +git clone https://github.com/persimmon-ai-labs/adept-inference +wget path/to/fuyu-8b-model-weights.tar +tar -xvf fuyu-8b-model-weights.tar +python src/transformers/models/fuyu/convert_fuyu_weights_to_hf.py --input_dir /path/to/downloaded/fuyu/weights/ --output_dir /output/path \ + --pt_model_path /path/to/fuyu_8b_release/iter_0001251/mp_rank_00/model_optim_rng.pt + --ada_lib_path /path/to/adept-inference +``` + +For the chat model: +```bash +wget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-data/o/8b_chat_model_release.tar +tar -xvf 8b_base_model_release.tar +``` +Then, model can be loaded via: + +```py +from transformers import FuyuConfig, FuyuForCausalLM +model_config = FuyuConfig() +model = FuyuForCausalLM(model_config).from_pretrained('/output/path') +``` + +Inputs need to be passed through a specific Processor to have the correct formats. +A processor requires an image_processor and a tokenizer. Hence, inputs can be loaded via: + +```py +from PIL import Image +from transformers import AutoTokenizer +from transformers.models.fuyu.processing_fuyu import FuyuProcessor +from transformers.models.fuyu.image_processing_fuyu import FuyuImageProcessor + + +tokenizer = AutoTokenizer.from_pretrained('adept-hf-collab/fuyu-8b') +image_processor = FuyuImageProcessor() + + +processor = FuyuProcessor(image_processor=image_processor, tokenizer=tokenizer) +text_prompt = "Generate a coco-style caption.\\n" + +bus_image_url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/bus.png" +bus_image_pil = Image.open(io.BytesIO(requests.get(bus_image_url).content)) +inputs_to_model = processor(text=text_prompt, images=image_pil) + + +``` + +This model was contributed by [Molbap](https://huggingface.co/Molbap). +The original code can be found [here](https://github.com/persimmon-ai-labs/adept-inference). + +- Fuyu uses a `sentencepiece` based tokenizer, with a `Unigram` model. It supports bytefallback, which is only available in `tokenizers==0.14.0` for the fast tokenizer. +The `LlamaTokenizer` is used as it is a standard wrapper around sentencepiece. + +- The authors suggest to use the following prompt for image captioning: `f"Generate a coco-style caption.\\n"` + + +## FuyuConfig + +[[autodoc]] FuyuConfig + +## FuyuForCausalLM + +[[autodoc]] FuyuForCausalLM + - forward + +## FuyuImageProcessor + +[[autodoc]] FuyuImageProcessor + - __call__ + +## FuyuProcessor + +[[autodoc]] FuyuProcessor + - __call__ \ No newline at end of file diff --git a/docs/source/en/tasks/language_modeling.md b/docs/source/en/tasks/language_modeling.md index 4240678821..c509899882 100644 --- a/docs/source/en/tasks/language_modeling.md +++ b/docs/source/en/tasks/language_modeling.md @@ -37,7 +37,7 @@ You can finetune other architectures for causal language modeling following the Choose one of the following architectures: -[BART](../model_doc/bart), [BERT](../model_doc/bert), [Bert Generation](../model_doc/bert-generation), [BigBird](../model_doc/big_bird), [BigBird-Pegasus](../model_doc/bigbird_pegasus), [BioGpt](../model_doc/biogpt), [Blenderbot](../model_doc/blenderbot), [BlenderbotSmall](../model_doc/blenderbot-small), [BLOOM](../model_doc/bloom), [CamemBERT](../model_doc/camembert), [CodeLlama](../model_doc/code_llama), [CodeGen](../model_doc/codegen), [CPM-Ant](../model_doc/cpmant), [CTRL](../model_doc/ctrl), [Data2VecText](../model_doc/data2vec-text), [ELECTRA](../model_doc/electra), [ERNIE](../model_doc/ernie), [Falcon](../model_doc/falcon), [GIT](../model_doc/git), [GPT-Sw3](../model_doc/gpt-sw3), [OpenAI GPT-2](../model_doc/gpt2), [GPTBigCode](../model_doc/gpt_bigcode), [GPT Neo](../model_doc/gpt_neo), [GPT NeoX](../model_doc/gpt_neox), [GPT NeoX Japanese](../model_doc/gpt_neox_japanese), [GPT-J](../model_doc/gptj), [LLaMA](../model_doc/llama), [Marian](../model_doc/marian), [mBART](../model_doc/mbart), [MEGA](../model_doc/mega), [Megatron-BERT](../model_doc/megatron-bert), [Mistral](../model_doc/mistral), [MPT](../model_doc/mpt), [MusicGen](../model_doc/musicgen), [MVP](../model_doc/mvp), [OpenLlama](../model_doc/open-llama), [OpenAI GPT](../model_doc/openai-gpt), [OPT](../model_doc/opt), [Pegasus](../model_doc/pegasus), [Persimmon](../model_doc/persimmon), [PLBart](../model_doc/plbart), [ProphetNet](../model_doc/prophetnet), [QDQBert](../model_doc/qdqbert), [Reformer](../model_doc/reformer), [RemBERT](../model_doc/rembert), [RoBERTa](../model_doc/roberta), [RoBERTa-PreLayerNorm](../model_doc/roberta-prelayernorm), [RoCBert](../model_doc/roc_bert), [RoFormer](../model_doc/roformer), [RWKV](../model_doc/rwkv), [Speech2Text2](../model_doc/speech_to_text_2), [Transformer-XL](../model_doc/transfo-xl), [TrOCR](../model_doc/trocr), [XGLM](../model_doc/xglm), [XLM](../model_doc/xlm), [XLM-ProphetNet](../model_doc/xlm-prophetnet), [XLM-RoBERTa](../model_doc/xlm-roberta), [XLM-RoBERTa-XL](../model_doc/xlm-roberta-xl), [XLNet](../model_doc/xlnet), [X-MOD](../model_doc/xmod) +[BART](../model_doc/bart), [BERT](../model_doc/bert), [Bert Generation](../model_doc/bert-generation), [BigBird](../model_doc/big_bird), [BigBird-Pegasus](../model_doc/bigbird_pegasus), [BioGpt](../model_doc/biogpt), [Blenderbot](../model_doc/blenderbot), [BlenderbotSmall](../model_doc/blenderbot-small), [BLOOM](../model_doc/bloom), [CamemBERT](../model_doc/camembert), [CodeLlama](../model_doc/code_llama), [CodeGen](../model_doc/codegen), [CPM-Ant](../model_doc/cpmant), [CTRL](../model_doc/ctrl), [Data2VecText](../model_doc/data2vec-text), [ELECTRA](../model_doc/electra), [ERNIE](../model_doc/ernie), [Falcon](../model_doc/falcon), [Fuyu](../model_doc/fuyu), [GIT](../model_doc/git), [GPT-Sw3](../model_doc/gpt-sw3), [OpenAI GPT-2](../model_doc/gpt2), [GPTBigCode](../model_doc/gpt_bigcode), [GPT Neo](../model_doc/gpt_neo), [GPT NeoX](../model_doc/gpt_neox), [GPT NeoX Japanese](../model_doc/gpt_neox_japanese), [GPT-J](../model_doc/gptj), [LLaMA](../model_doc/llama), [Marian](../model_doc/marian), [mBART](../model_doc/mbart), [MEGA](../model_doc/mega), [Megatron-BERT](../model_doc/megatron-bert), [Mistral](../model_doc/mistral), [MPT](../model_doc/mpt), [MusicGen](../model_doc/musicgen), [MVP](../model_doc/mvp), [OpenLlama](../model_doc/open-llama), [OpenAI GPT](../model_doc/openai-gpt), [OPT](../model_doc/opt), [Pegasus](../model_doc/pegasus), [Persimmon](../model_doc/persimmon), [PLBart](../model_doc/plbart), [ProphetNet](../model_doc/prophetnet), [QDQBert](../model_doc/qdqbert), [Reformer](../model_doc/reformer), [RemBERT](../model_doc/rembert), [RoBERTa](../model_doc/roberta), [RoBERTa-PreLayerNorm](../model_doc/roberta-prelayernorm), [RoCBert](../model_doc/roc_bert), [RoFormer](../model_doc/roformer), [RWKV](../model_doc/rwkv), [Speech2Text2](../model_doc/speech_to_text_2), [Transformer-XL](../model_doc/transfo-xl), [TrOCR](../model_doc/trocr), [XGLM](../model_doc/xglm), [XLM](../model_doc/xlm), [XLM-ProphetNet](../model_doc/xlm-prophetnet), [XLM-RoBERTa](../model_doc/xlm-roberta), [XLM-RoBERTa-XL](../model_doc/xlm-roberta-xl), [XLNet](../model_doc/xlnet), [X-MOD](../model_doc/xmod) diff --git a/src/transformers/__init__.py b/src/transformers/__init__.py index a68a492676..3325a2c77b 100644 --- a/src/transformers/__init__.py +++ b/src/transformers/__init__.py @@ -343,6 +343,7 @@ _import_structure = { "models.focalnet": ["FOCALNET_PRETRAINED_CONFIG_ARCHIVE_MAP", "FocalNetConfig"], "models.fsmt": ["FSMT_PRETRAINED_CONFIG_ARCHIVE_MAP", "FSMTConfig", "FSMTTokenizer"], "models.funnel": ["FUNNEL_PRETRAINED_CONFIG_ARCHIVE_MAP", "FunnelConfig", "FunnelTokenizer"], + "models.fuyu": ["FUYU_PRETRAINED_CONFIG_ARCHIVE_MAP", "FuyuConfig", "FuyuProcessor"], "models.git": ["GIT_PRETRAINED_CONFIG_ARCHIVE_MAP", "GitConfig", "GitProcessor", "GitVisionConfig"], "models.glpn": ["GLPN_PRETRAINED_CONFIG_ARCHIVE_MAP", "GLPNConfig"], "models.gpt2": ["GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP", "GPT2Config", "GPT2Tokenizer"], @@ -972,6 +973,7 @@ else: _import_structure["models.efficientformer"].append("EfficientFormerImageProcessor") _import_structure["models.efficientnet"].append("EfficientNetImageProcessor") _import_structure["models.flava"].extend(["FlavaFeatureExtractor", "FlavaImageProcessor", "FlavaProcessor"]) + _import_structure["models.fuyu"].append("FuyuImageProcessor") _import_structure["models.glpn"].extend(["GLPNFeatureExtractor", "GLPNImageProcessor"]) _import_structure["models.idefics"].extend(["IdeficsImageProcessor"]) _import_structure["models.imagegpt"].extend(["ImageGPTFeatureExtractor", "ImageGPTImageProcessor"]) @@ -1864,6 +1866,7 @@ else: "load_tf_weights_in_funnel", ] ) + _import_structure["models.fuyu"].extend(["FuyuForCausalLM", "FuyuPreTrainedModel"]) _import_structure["models.git"].extend( [ "GIT_PRETRAINED_MODEL_ARCHIVE_LIST", @@ -4489,6 +4492,7 @@ if TYPE_CHECKING: from .models.focalnet import FOCALNET_PRETRAINED_CONFIG_ARCHIVE_MAP, FocalNetConfig from .models.fsmt import FSMT_PRETRAINED_CONFIG_ARCHIVE_MAP, FSMTConfig, FSMTTokenizer from .models.funnel import FUNNEL_PRETRAINED_CONFIG_ARCHIVE_MAP, FunnelConfig, FunnelTokenizer + from .models.fuyu import FUYU_PRETRAINED_CONFIG_ARCHIVE_MAP, FuyuConfig, FuyuProcessor from .models.git import GIT_PRETRAINED_CONFIG_ARCHIVE_MAP, GitConfig, GitProcessor, GitVisionConfig from .models.glpn import GLPN_PRETRAINED_CONFIG_ARCHIVE_MAP, GLPNConfig from .models.gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config, GPT2Tokenizer @@ -5053,6 +5057,7 @@ if TYPE_CHECKING: from .models.efficientformer import EfficientFormerImageProcessor from .models.efficientnet import EfficientNetImageProcessor from .models.flava import FlavaFeatureExtractor, FlavaImageProcessor, FlavaProcessor + from .models.fuyu import FuyuImageProcessor from .models.glpn import GLPNFeatureExtractor, GLPNImageProcessor from .models.idefics import IdeficsImageProcessor from .models.imagegpt import ImageGPTFeatureExtractor, ImageGPTImageProcessor @@ -5807,6 +5812,10 @@ if TYPE_CHECKING: FunnelPreTrainedModel, load_tf_weights_in_funnel, ) + from .models.fuyu import ( + FuyuForCausalLM, + FuyuPreTrainedModel, + ) from .models.git import ( GIT_PRETRAINED_MODEL_ARCHIVE_LIST, GitForCausalLM, diff --git a/src/transformers/models/__init__.py b/src/transformers/models/__init__.py index b4486039b9..074bade692 100644 --- a/src/transformers/models/__init__.py +++ b/src/transformers/models/__init__.py @@ -88,6 +88,7 @@ from . import ( focalnet, fsmt, funnel, + fuyu, git, glpn, gpt2, diff --git a/src/transformers/models/auto/configuration_auto.py b/src/transformers/models/auto/configuration_auto.py index 5690359643..0328426d67 100755 --- a/src/transformers/models/auto/configuration_auto.py +++ b/src/transformers/models/auto/configuration_auto.py @@ -97,6 +97,7 @@ CONFIG_MAPPING_NAMES = OrderedDict( ("focalnet", "FocalNetConfig"), ("fsmt", "FSMTConfig"), ("funnel", "FunnelConfig"), + ("fuyu", "FuyuConfig"), ("git", "GitConfig"), ("glpn", "GLPNConfig"), ("gpt-sw3", "GPT2Config"), @@ -310,6 +311,7 @@ CONFIG_ARCHIVE_MAP_MAPPING_NAMES = OrderedDict( ("focalnet", "FOCALNET_PRETRAINED_CONFIG_ARCHIVE_MAP"), ("fsmt", "FSMT_PRETRAINED_CONFIG_ARCHIVE_MAP"), ("funnel", "FUNNEL_PRETRAINED_CONFIG_ARCHIVE_MAP"), + ("fuyu", "FUYU_PRETRAINED_CONFIG_ARCHIVE_MAP"), ("git", "GIT_PRETRAINED_CONFIG_ARCHIVE_MAP"), ("glpn", "GLPN_PRETRAINED_CONFIG_ARCHIVE_MAP"), ("gpt2", "GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP"), @@ -521,6 +523,7 @@ MODEL_NAMES_MAPPING = OrderedDict( ("focalnet", "FocalNet"), ("fsmt", "FairSeq Machine-Translation"), ("funnel", "Funnel Transformer"), + ("fuyu", "Fuyu"), ("git", "GIT"), ("glpn", "GLPN"), ("gpt-sw3", "GPT-Sw3"), diff --git a/src/transformers/models/auto/image_processing_auto.py b/src/transformers/models/auto/image_processing_auto.py index 13bb3a6e5d..0a9c95b3e5 100644 --- a/src/transformers/models/auto/image_processing_auto.py +++ b/src/transformers/models/auto/image_processing_auto.py @@ -64,6 +64,7 @@ IMAGE_PROCESSOR_MAPPING_NAMES = OrderedDict( ("efficientnet", "EfficientNetImageProcessor"), ("flava", "FlavaImageProcessor"), ("focalnet", "BitImageProcessor"), + ("fuyu", "FuyuImageProcessor"), ("git", "CLIPImageProcessor"), ("glpn", "GLPNImageProcessor"), ("groupvit", "CLIPImageProcessor"), diff --git a/src/transformers/models/auto/modeling_auto.py b/src/transformers/models/auto/modeling_auto.py index bbbaa58d6e..c6fc1dce1b 100755 --- a/src/transformers/models/auto/modeling_auto.py +++ b/src/transformers/models/auto/modeling_auto.py @@ -400,6 +400,7 @@ MODEL_FOR_CAUSAL_LM_MAPPING_NAMES = OrderedDict( ("electra", "ElectraForCausalLM"), ("ernie", "ErnieForCausalLM"), ("falcon", "FalconForCausalLM"), + ("fuyu", "FuyuForCausalLM"), ("git", "GitForCausalLM"), ("gpt-sw3", "GPT2LMHeadModel"), ("gpt2", "GPT2LMHeadModel"), diff --git a/src/transformers/models/auto/processing_auto.py b/src/transformers/models/auto/processing_auto.py index 36a93789e2..e7cb9aa11a 100644 --- a/src/transformers/models/auto/processing_auto.py +++ b/src/transformers/models/auto/processing_auto.py @@ -54,6 +54,7 @@ PROCESSOR_MAPPING_NAMES = OrderedDict( ("clip", "CLIPProcessor"), ("clipseg", "CLIPSegProcessor"), ("flava", "FlavaProcessor"), + ("fuyu", "FuyuProcessor"), ("git", "GitProcessor"), ("groupvit", "CLIPProcessor"), ("hubert", "Wav2Vec2Processor"), diff --git a/src/transformers/models/fuyu/__init__.py b/src/transformers/models/fuyu/__init__.py new file mode 100644 index 0000000000..51a72a5366 --- /dev/null +++ b/src/transformers/models/fuyu/__init__.py @@ -0,0 +1,73 @@ +# Copyright 2023 AdeptAI and The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +from typing import TYPE_CHECKING + +from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available, is_vision_available + + +_import_structure = { + "configuration_fuyu": ["FUYU_PRETRAINED_CONFIG_ARCHIVE_MAP", "FuyuConfig"], +} + + +try: + if not is_vision_available(): + raise OptionalDependencyNotAvailable() +except OptionalDependencyNotAvailable: + pass +else: + _import_structure["image_processing_fuyu"] = ["FuyuImageProcessor"] + _import_structure["processing_fuyu"] = ["FuyuProcessor"] + + +try: + if not is_torch_available(): + raise OptionalDependencyNotAvailable() +except OptionalDependencyNotAvailable: + pass +else: + _import_structure["modeling_fuyu"] = [ + "FuyuForCausalLM", + "FuyuPreTrainedModel", + ] + + +if TYPE_CHECKING: + from .configuration_fuyu import FUYU_PRETRAINED_CONFIG_ARCHIVE_MAP, FuyuConfig + + try: + if not is_vision_available(): + raise OptionalDependencyNotAvailable() + except OptionalDependencyNotAvailable: + pass + else: + from .image_processing_fuyu import FuyuImageProcessor + from .processing_fuyu import FuyuProcessor + + try: + if not is_torch_available(): + raise OptionalDependencyNotAvailable() + except OptionalDependencyNotAvailable: + pass + else: + from .modeling_fuyu import ( + FuyuForCausalLM, + FuyuPreTrainedModel, + ) + + +else: + import sys + + sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) diff --git a/src/transformers/models/fuyu/configuration_fuyu.py b/src/transformers/models/fuyu/configuration_fuyu.py new file mode 100644 index 0000000000..b031fff45a --- /dev/null +++ b/src/transformers/models/fuyu/configuration_fuyu.py @@ -0,0 +1,211 @@ +# coding=utf-8 +# Copyright 2023 Adept AI and the HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" Fuyu model configuration""" + +from ...configuration_utils import PretrainedConfig +from ...utils import logging +from ..auto import CONFIG_MAPPING + + +logger = logging.get_logger(__name__) + +FUYU_PRETRAINED_CONFIG_ARCHIVE_MAP = { + "adept/fuyu-8b-base": "https://huggingface.co/adept/fuyu-8b-base/resolve/main/config.json", +} + + +class FuyuConfig(PretrainedConfig): + r""" + This is the configuration class to store the configuration of a [`FuyuForCausalLM`]. It is used to instantiate an + Fuyu model according to the specified arguments, defining the model architecture. Instantiating a configuration + with the defaults will yield a similar configuration to that of the + [adept/fuyu-8b-base](https://huggingface.co/adept/fuyu-8b-base). + + Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the + documentation from [`PretrainedConfig`] for more information. + + + Args: + vocab_size (`int`, *optional*, defaults to 262144): + Vocabulary size of the Fuyu model. Defines the number of different tokens that can be represented by the + `inputs_ids` passed when calling [`FuyuForCausalLM`] + hidden_size (`int`, *optional*, defaults to 4096): + Dimension of the hidden representations. + intermediate_size (`int`, *optional*, defaults to 16384): + Dimension of the MLP representations. + num_hidden_layers (`int`, *optional*, defaults to 36): + Number of hidden layers in the Transformer encoder. + num_attention_heads (`int`, *optional*, defaults to 64): + Number of attention heads for each attention layer in the Transformer encoder. + hidden_act (`str` or `function`, *optional*, defaults to `"relu2"`): + The non-linear activation function (function or string) in the decoder. + max_position_embeddings (`int`, *optional*, defaults to 16384): + The maximum sequence length that this model might ever be used with. + image_size (`int`, *optional*, defaults to 300): + The input image size. + patch_size (`int`, *optional*, defaults to 30): + The input vision transformer encoding patch size. + num_channels (`int`, *optional*, defaults to 3): + The input image number of channels. + initializer_range (`float`, *optional*, defaults to 0.02): + The standard deviation of the truncated_normal_initializer for initializing all weight matrices. + layer_norm_eps (`float`, *optional*, defaults to 1e-05): + The epsilon used by the rms normalization layers. + use_cache (`bool`, *optional*, defaults to `True`): + Whether or not the model should return the last key/values attentions (not used by all models). Only + relevant if `config.is_decoder=True`. Whether to tie weight embeddings + tie_word_embeddings (`bool`, *optional*, defaults to `False`): + Whether to tie input and output embeddings. + rope_theta (`float`, *optional*, defaults to 25000.0): + The base period of the RoPE embeddings. + rope_scaling (`Dict`, *optional*): + Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling + strategies: linear and dynamic. Their scaling factor must be an float greater than 1. The expected format + is `{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update + `max_position_embeddings` to the expected new maximum. See the following thread for more information on how + these scaling strategies behave: + https://www.reddit.com/r/LocalFuyu/comments/14mrgpr/dynamically_scaled_rope_further_increases/. This is an + experimental feature, subject to breaking API changes in future versions. + qk_layernorm (`bool`, *optional*, defaults to `True`): + Whether or not to normalize the Queries and Keys after projecting the hidden states + hidden_dropout (`float`, *optional*, defaults to 0.0): + The dropout ratio after applying the MLP to the hidden states. + attention_dropout (`float`, *optional*, defaults to 0.0): + The dropout ratio after computing the attention scores. + partial_rotary_factor (`float`, *optional*, defaults to 0.5): + Percentage of the query and keys which will have rotary embedding. + + pad_token_id (`int`, *optional*): + The id of the *padding* token. + bos_token_id (`int`, *optional*, defaults to 1): + The id of the *beginning-of-sequence* token. + eos_token_id (`Union[int, List[int]]`, *optional*, defaults to 2): + The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. + text_config (`dict`, *optional*): + Dictionary of configuration options used to initialize the `language``[`Aut`]. + + ```python + >>> from transformers import FuyuConfig + + >>> # Initializing a Fuyu fuyu-7b style configuration + >>> configuration = FuyuConfig() + ```""" + model_type = "fuyu" + keys_to_ignore_at_inference = ["past_key_values"] + + def __init__( + self, + vocab_size=262144, + hidden_size=4096, + intermediate_size=16384, + num_hidden_layers=36, + num_attention_heads=64, + hidden_act="relu2", + max_position_embeddings=16384, + image_size=300, + patch_size=30, + num_channels=3, + initializer_range=0.02, + layer_norm_eps=1e-5, + use_cache=True, + tie_word_embeddings=False, + rope_theta=25000.0, + rope_scaling=None, + qk_layernorm=True, + hidden_dropout=0.0, + attention_dropout=0.0, + partial_rotary_factor=0.5, + pad_token_id=None, + bos_token_id=1, + eos_token_id=2, + text_config=None, + **kwargs, + ): + if text_config is None: + text_config = { + "vocab_size": vocab_size, + "max_position_embeddings": max_position_embeddings, + "hidden_size": hidden_size, + "intermediate_size": intermediate_size, + "num_hidden_layers": num_hidden_layers, + "num_attention_heads": num_attention_heads, + "hidden_act": hidden_act, + "initializer_range": initializer_range, + "layer_norm_eps": layer_norm_eps, + "use_cache": use_cache, + "rope_theta": rope_theta, + "rope_scaling": rope_scaling, + "qk_layernorm": qk_layernorm, + "hidden_dropout": hidden_dropout, + "attention_dropout": attention_dropout, + "partial_rotary_factor": partial_rotary_factor, + "pad_token_id": pad_token_id, + "bos_token_id": bos_token_id, + "eos_token_id": eos_token_id, + "tie_word_embeddings": tie_word_embeddings, + } + logger.info("text_config is None. initializing the text model with default values.") + text_model_type = text_config["model_type"] if "model_type" in text_config else "persimmon" + self.text_config = CONFIG_MAPPING[text_model_type](**text_config) + + self.vocab_size = vocab_size + self.max_position_embeddings = max_position_embeddings + self.image_size = image_size + self.patch_size = patch_size + self.num_channels = num_channels + self.hidden_size = hidden_size + self.intermediate_size = intermediate_size + self.num_hidden_layers = num_hidden_layers + self.num_attention_heads = num_attention_heads + self.hidden_act = hidden_act + self.initializer_range = initializer_range + self.layer_norm_eps = layer_norm_eps + self.use_cache = use_cache + self.rope_theta = rope_theta + self.rope_scaling = rope_scaling + self.qk_layernorm = qk_layernorm + self.hidden_dropout = hidden_dropout + self.attention_dropout = attention_dropout + self.partial_rotary_factor = partial_rotary_factor + self._rope_scaling_validation() + + super().__init__( + pad_token_id=pad_token_id, + bos_token_id=bos_token_id, + eos_token_id=eos_token_id, + tie_word_embeddings=tie_word_embeddings, + **kwargs, + ) + + def _rope_scaling_validation(self): + """ + Validate the `rope_scaling` configuration. + """ + if self.rope_scaling is None: + return + + if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2: + raise ValueError( + "`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, " + f"got {self.rope_scaling}" + ) + rope_scaling_type = self.rope_scaling.get("type", None) + rope_scaling_factor = self.rope_scaling.get("factor", None) + if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic"]: + raise ValueError( + f"`rope_scaling`'s type field must be one of ['linear', 'dynamic'], got {rope_scaling_type}" + ) + if rope_scaling_factor is None or not isinstance(rope_scaling_factor, float) or rope_scaling_factor <= 1.0: + raise ValueError(f"`rope_scaling`'s factor field must be an float > 1, got {rope_scaling_factor}") diff --git a/src/transformers/models/fuyu/convert_fuyu_model_weights_to_hf.py b/src/transformers/models/fuyu/convert_fuyu_model_weights_to_hf.py new file mode 100644 index 0000000000..6d029c0d13 --- /dev/null +++ b/src/transformers/models/fuyu/convert_fuyu_model_weights_to_hf.py @@ -0,0 +1,134 @@ +# Copyright 2023 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import argparse +import os +import sys +import warnings + +import flatdict +import torch + +from transformers import FuyuConfig, FuyuForCausalLM, LlamaTokenizer + + +try: + from transformers import LlamaTokenizerFast + + tokenizer_class = LlamaTokenizerFast +except ImportError as e: + warnings.warn(e) + warnings.warn( + "The converted tokenizer will be the `slow` tokenizer. To use the fast, update your `tokenizers` library and re-run the tokenizer conversion" + ) + tokenizer_class = LlamaTokenizer + +""" +Sample usage: # TODO fix clone links from persimmon to fuyu +``` +git clone https://github.com/adept-ai-labs/adept-inference +wget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-data/o/8b_base_model_release.tar +wget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-data/o/8b_chat_model_release.tar +python src/transformers/models/fuyu/convert_fuyu_weights_to_hf.py --input_dir /path/to/downloaded/fuyu/weights/ --output_dir /output/path +``` + +Thereafter, models can be loaded via: + +```py +from transformers import FuyuForCausalLM, FuyuTokenizer + +model = FuyuForCausalLM.from_pretrained("/output/path") +tokenizer = FuyuTokenizer.from_pretrained("/output/path") +``` + +Important note: you need to be able to host the whole model in RAM to execute this script (even if the biggest versions +come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). +""" + + +KEYS_TO_MODIFY_MAPPING = { + "self_attention": "self_attn", + "language_model.encoder": "language_model.model", + "word_embeddings_for_head": "language_model.lm_head", + "language_model.embedding.word_embeddings": "language_model.model.embed_tokens", + "vit_encoder.linear_encoder": "vision_embed_tokens", +} + +KEYS_TO_REMOVE = { + "rotary_emb.inv_freq", + "image_patch_projection", + "image_patch_projection.weight", + "image_patch_projection.bias", +} + + +def rename_state_dict(state_dict): + model_state_dict = {} + for key, value in state_dict.items(): + for key_to_modify, new_key in KEYS_TO_MODIFY_MAPPING.items(): + if key_to_modify in key: + key = key.replace(key_to_modify, new_key) + # if KEYS_TO_REMOVE in key: + if key in KEYS_TO_REMOVE: + continue + model_state_dict[key] = value + return model_state_dict + + +def convert_fuyu_checkpoint(pytorch_dump_folder_path, ada_lib_path, pt_model_path, safe_serialization=False): + sys.path.insert(0, ada_lib_path) + model_state_dict_base = torch.load(pt_model_path, map_location="cpu") + state_dict = flatdict.FlatDict(model_state_dict_base["model"], ".") + state_dict = rename_state_dict(state_dict) + + transformers_config = FuyuConfig() + model = FuyuForCausalLM(transformers_config).to(torch.bfloat16) + model.load_state_dict(state_dict) + model.save_pretrained(pytorch_dump_folder_path, safe_serialization=safe_serialization) + transformers_config.save_pretrained(pytorch_dump_folder_path) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--input_dir", + help="Location of Fuyu weights, which contains tokenizer.model and model folders", + ) + parser.add_argument( + "--pt_model_path", + help="Location of Fuyu `model_optim_rng.pt`", + ) + parser.add_argument( + "--output_dir", + help="Location to write HF model and tokenizer", + ) + parser.add_argument( + "--ada_lib_path", + help="Location of original source code from adept to deserialize .pt checkpoint", + ) + parser.add_argument("--safe_serialization", type=bool, help="Whether or not to save using `safetensors`.") + args = parser.parse_args() + spm_path = os.path.join(args.input_dir, "adept_vocab.model") + + convert_fuyu_checkpoint( + pytorch_dump_folder_path=args.output_dir, + pt_model_path=args.pt_model_path, + safe_serialization=args.safe_serialization, + ada_lib_path=args.ada_lib_path, + ) + tokenizer = tokenizer_class(spm_path, bos_token="|ENDOFTEXT|", eos_token="|ENDOFTEXT|") + tokenizer.save_pretrained(args.output_dir) + + +if __name__ == "__main__": + main() diff --git a/src/transformers/models/fuyu/image_processing_fuyu.py b/src/transformers/models/fuyu/image_processing_fuyu.py new file mode 100644 index 0000000000..3f145ae5cd --- /dev/null +++ b/src/transformers/models/fuyu/image_processing_fuyu.py @@ -0,0 +1,254 @@ +import math +from typing import List, Union + +import numpy as np + +from ...image_processing_utils import BaseImageProcessor +from ...image_transforms import ( + normalize, + pad, + resize, +) +from ...image_utils import to_numpy_array +from ...utils import is_torch_available, is_vision_available, logging, requires_backends + + +if is_vision_available(): + import PIL + +if is_torch_available(): + import torch + +logger = logging.get_logger(__name__) + + +class FuyuImageProcessor(BaseImageProcessor): + """ + This class should handle the image processing part before the main FuyuForCausalLM. In particular, it should + handle: + + - Processing Images: + Taking a batch of images as input. If the images are variable-sized, it resizes them based on the desired patch + dimensions. The image output is always img_h ........................................... 1080 img_w + ........................................... 1920 Then, it patches up these images using the patchify_image + function. + + - Creating Image Input IDs: + For each patch, a placeholder ID is given to identify where these patches belong in a token sequence. For + variable-sized images, each line of patches is terminated with a newline ID. + + - Image Patch Indices: + For each image patch, the code maintains an index where these patches should be inserted in a token stream. + + """ + + model_input_names = [ + "images", + "image_input_ids", + "image_patches", + "image_patch_indices_per_batch", + "image_patch_indices_per_subsequence", + ] + + def __init__( + self, target_height=1080, target_width=1920, padding_value=1.0, padding_mode: str = "constant", **kwargs + ): + super().__init__(**kwargs) + self.target_width = target_width + self.target_height = target_height + self.padding_value = padding_value + self.padding_mode = padding_mode + + def get_num_patches(self, img_h: int, img_w: int, patch_dim_h: int, patch_dim_w: int) -> int: + """Calculate number of patches required to encode an image.""" + if img_h % patch_dim_h != 0: + raise ValueError(f"{img_h=} must be divisible by {patch_dim_h=}") + if img_w % patch_dim_w != 0: + raise ValueError(f"{img_w=} must be divisible by {patch_dim_w=}") + + num_patches_per_dim_h = img_h // patch_dim_h + num_patches_per_dim_w = img_w // patch_dim_w + num_patches = num_patches_per_dim_h * num_patches_per_dim_w + + return num_patches + + def patchify_image(self, image: "torch.Tensor", patch_dim_h: int, patch_dim_w: int) -> "torch.Tensor": + """ + Convert an image into a tensor of patches. + + Args: + image: Image to convert. Shape: [batch, channels, height, width] + patch_dim_h: Height of each patch. + patch_dim_w: Width of each patch. + """ + requires_backends(self, ["torch"]) + + # TODO refer to https://github.com/ArthurZucker/transformers/blob/0f0a3fe5ca5697ee58faeb5b53f049af720b5e98/src/transformers/models/vit_mae/modeling_vit_mae.py#L871 + # torch implementation is faster but does not handle non-squares + + batch_size, channels, height, width = image.shape + unfolded_along_height = image.unfold(2, patch_dim_h, patch_dim_h) + patches = unfolded_along_height.unfold(3, patch_dim_w, patch_dim_w) + + patches_reshaped = patches.contiguous().view(batch_size, channels, -1, patch_dim_h, patch_dim_w) + + patches_final = patches_reshaped.permute(0, 2, 3, 4, 1).reshape( + batch_size, -1, channels * patch_dim_h * patch_dim_w + ) + + return patches_final + + def process_images_for_model_input( + self, + image_input: "torch.Tensor", + image_present: "torch.Tensor", + image_unpadded_h: "torch.Tensor", + image_unpadded_w: "torch.Tensor", + image_patch_dim_h: int, + image_patch_dim_w: int, + image_placeholder_id: int, + image_newline_id: int, + variable_sized: bool, + ) -> dict: + """Process images for model input. In particular, variable-sized images are handled here. + + Args: + image_input: [batch_size, 1, c, h, w] tensor of images padded to model input size. + image_present: [batch_size, 1] tensor of 1s and 0s indicating whether an image is present. + image_unpadded_h: [batch_size, 1] tensor of unpadded image heights. + image_unpadded_w: [batch_size, 1] tensor of unpadded image widths. + image_patch_dim_h: The height of the image patches. + image_patch_dim_w: The width of the image patches. + image_placeholder_id: The id of the image placeholder token. + image_newline_id: The id of the image newline token. + variable_sized: Whether to process images as variable-sized. + """ + requires_backends(self, ["torch"]) + # Only images that are present. + images: List[List[torch.Tensor]] = [] + image_patches: List[List[torch.Tensor]] = [] + # Image input ids for every subsequence, including ones with no image present. + image_input_ids: List[List[torch.Tensor]] = [] + for bi in range(image_input.shape[0]): + images.append([]) + image_input_ids.append([]) + image_patches.append([]) + for si in range(image_input.shape[1]): + if image_present[bi, si]: + image = image_input[bi, si] + if variable_sized: + # The min() is required here due to floating point issues: + # math.ceil(torch.tensor(300).cuda() / 30) == 11 + new_h = min( + image.shape[1], math.ceil(image_unpadded_h[bi, si] / image_patch_dim_h) * image_patch_dim_h + ) + new_w = min( + image.shape[2], math.ceil(image_unpadded_w[bi, si] / image_patch_dim_w) * image_patch_dim_w + ) + image = image[:, :new_h, :new_w] + images[bi].append(image) + num_patches = self.get_num_patches( + img_h=image.shape[1], + img_w=image.shape[2], + patch_dim_h=image_patch_dim_h, + patch_dim_w=image_patch_dim_w, + ) + ids = torch.full([num_patches], image_placeholder_id, dtype=torch.int32, device=image_input.device) + patches = self.patchify_image( + image=image.unsqueeze(0), patch_dim_h=image_patch_dim_h, patch_dim_w=image_patch_dim_w + ).squeeze(0) + if variable_sized: + # Now terminate each line with |NEWLINE|. + ids = ids.reshape(-1, new_w // image_patch_dim_w) + ids = torch.cat( + [ + ids, + torch.full( + [ids.shape[0], 1], image_newline_id, dtype=torch.int32, device=image_input.device + ), + ], + dim=1, + ) + ids = ids.reshape(-1) + image_input_ids[bi].append(ids) + image_patches[bi].append(patches) + else: + image_input_ids[bi].append(torch.tensor([], dtype=torch.int32, device=image_input.device)) + + # Create image_patch_input_indices, where non-negative values correspond to image patches to be inserted in + # the stream. + image_patch_indices_per_batch: List[List[torch.Tensor]] = [] + image_patch_indices_per_subsequence: List[List[torch.Tensor]] = [] + for bi in range(len(image_input_ids)): + image_patch_indices_per_batch.append([]) + image_patch_indices_per_subsequence.append([]) + index_offset = 0 + for si in range(len(image_input_ids[bi])): + # Indices of image patches. + num_patches = torch.count_nonzero(image_input_ids[bi][si] == image_placeholder_id) + indices = torch.arange( + num_patches, + dtype=image_input_ids[bi][si].dtype, + device=image_input_ids[bi][si].device, + ) + + # Place those indices in the image input ids token stream, with -1 representing non-index tokens. + indices_in_stream_per_batch = torch.full_like(image_input_ids[bi][si], -1) + indices_in_stream_per_subsequence = torch.full_like(image_input_ids[bi][si], -1) + indices_in_stream_per_batch[ + torch.nonzero(image_input_ids[bi][si] == image_placeholder_id, as_tuple=True)[0] + ] = (indices + index_offset) + indices_in_stream_per_subsequence[ + torch.nonzero(image_input_ids[bi][si] == image_placeholder_id, as_tuple=True)[0] + ] = indices + + image_patch_indices_per_batch[bi].append(indices_in_stream_per_batch) + image_patch_indices_per_subsequence[bi].append(indices_in_stream_per_subsequence) + index_offset += num_patches + + return { + "images": images, + "image_input_ids": image_input_ids, + "image_patches": image_patches, + "image_patch_indices_per_batch": image_patch_indices_per_batch, + "image_patch_indices_per_subsequence": image_patch_indices_per_subsequence, + } + + def _scale_to_target_aspect_ratio(self, image: np.ndarray) -> np.ndarray: + image_height, image_width, _ = image.shape + if image_width <= self.target_width and image_height <= self.target_height: + return image + + height_scale_factor = self.target_height / image_height + width_scale_factor = self.target_width / image_width + optimal_scale_factor = min(height_scale_factor, width_scale_factor) + + new_height = int(image_height * optimal_scale_factor) + new_width = int(image_width * optimal_scale_factor) + + scaled_image = resize(image=image, size=(new_width, new_height)) + return np.array(scaled_image) + + def _pad_to_target_size(self, image: np.ndarray) -> np.ndarray: + image_height, image_width, _ = image.shape + + padding_top = 0 + padding_left = 0 + padding_bottom = self.target_height - image_height + padding_right = self.target_width - image_width + + padded_image = pad( + image, + ((padding_top, padding_bottom), (padding_left, padding_right)), + mode=self.padding_mode, + constant_values=self.padding_value, + ) + return padded_image + + def apply_transformation(self, image: Union[np.ndarray, PIL.Image.Image]) -> np.ndarray: + if isinstance(image, PIL.Image.Image): + image = to_numpy_array(image) + scaled_image = self._scale_to_target_aspect_ratio(image) + padded_image = self._pad_to_target_size(scaled_image) + normalized_padded_image = normalize(padded_image, 0.5, 0.5) + return normalized_padded_image diff --git a/src/transformers/models/fuyu/modeling_fuyu.py b/src/transformers/models/fuyu/modeling_fuyu.py new file mode 100644 index 0000000000..b14b1b0b87 --- /dev/null +++ b/src/transformers/models/fuyu/modeling_fuyu.py @@ -0,0 +1,323 @@ +# coding=utf-8 +# Copyright 2023 HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" PyTorch Fuyu model.""" +from typing import List, Optional, Tuple, Union + +import torch +import torch.utils.checkpoint +from torch import nn + +from ...modeling_outputs import BaseModelOutputWithPast +from ...modeling_utils import PreTrainedModel +from ...models.auto.modeling_auto import AutoModelForCausalLM +from ...utils import add_start_docstrings, add_start_docstrings_to_model_forward, logging +from .configuration_fuyu import FuyuConfig + + +logger = logging.get_logger(__name__) + +_CONFIG_FOR_DOC = "FuyuConfig" + + +FUYU_START_DOCSTRING = r""" + This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the + library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads + etc.) + + This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. + Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage + and behavior. + + Parameters: + config ([`FuyuConfig`]): + Model configuration class with all the parameters of the model. Initializing with a config file does not + load the weights associated with the model, only the configuration. Check out the + [`~PreTrainedModel.from_pretrained`] method to load the model weights. +""" + + +@add_start_docstrings( + "The bare Fuyu Model outputting raw hidden-states without any specific head on top.", + FUYU_START_DOCSTRING, +) +class FuyuPreTrainedModel(PreTrainedModel): + config_class = FuyuConfig + base_model_prefix = "fuyu" + supports_gradient_checkpointing = True + _no_split_modules = [] + _skip_keys_device_placement = "past_key_values" + + def _init_weights(self, module): + std = self.config.initializer_range + if isinstance(module, nn.Linear): + module.weight.data.normal_(mean=0.0, std=std) + if module.bias is not None: + module.bias.data.zero_() + elif isinstance(module, nn.Embedding): + module.weight.data.normal_(mean=0.0, std=std) + if module.padding_idx is not None: + module.weight.data[module.padding_idx].zero_() + + def _set_gradient_checkpointing(self, module, value=False): + if isinstance(module, FuyuForCausalLM): + module.gradient_checkpointing = value + + +FUYU_INPUTS_DOCSTRING = r""" + Args: + input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): + Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide + it. + + Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and + [`PreTrainedTokenizer.__call__`] for details. + + [What are input IDs?](../glossary#input-ids) + attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): + Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: + + - 1 for tokens that are **not masked**, + - 0 for tokens that are **masked**. + + [What are attention masks?](../glossary#attention-mask) + + Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and + [`PreTrainedTokenizer.__call__`] for details. + + If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see + `past_key_values`). + + If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] + and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more + information on the default strategy. + + - 1 indicates the head is **not masked**, + - 0 indicates the head is **masked**. + position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): + Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, + config.n_positions - 1]`. + + [What are position IDs?](../glossary#position-ids) + past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): + Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape + `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape + `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. + + Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention + blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. + + If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that + don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all + `decoder_input_ids` of shape `(batch_size, sequence_length)`. + inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): + Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This + is useful if you want more control over how to convert `input_ids` indices into associated vectors than the + model's internal embedding lookup matrix. + use_cache (`bool`, *optional*): + If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see + `past_key_values`). + output_attentions (`bool`, *optional*): + Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned + tensors for more detail. + output_hidden_states (`bool`, *optional*): + Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for + more detail. + return_dict (`bool`, *optional*): + Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. +""" + + +@add_start_docstrings( + "The bare Fuyu Model outputting raw hidden-states without any specific head on top.", + FUYU_START_DOCSTRING, +) +class FuyuForCausalLM(FuyuPreTrainedModel): + """ + Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`FuyuDecoderLayer`] + + Args: + config: FuyuConfig + """ + + def __init__(self, config: FuyuConfig): + super().__init__(config) + self.padding_idx = config.pad_token_id + self.vocab_size = config.vocab_size + self.language_model = AutoModelForCausalLM.from_config(config.text_config) + + self.vision_embed_tokens = nn.Linear( + config.patch_size * config.patch_size * config.num_channels, config.hidden_size + ) + + self.gradient_checkpointing = False + # Initialize weights and apply final processing + self.post_init() + + def get_input_embeddings(self): + return self.language_model.get_input_embeddings() + + def set_input_embeddings(self, value): + self.language_model.set_input_embeddings(value) + + def gather_continuous_embeddings( + self, + word_embeddings: torch.Tensor, + continuous_embeddings: List[torch.Tensor], + image_patch_input_indices: torch.Tensor, + ) -> torch.Tensor: + """This function places the continuous_embeddings into the word_embeddings at the locations + indicated by image_patch_input_indices. Different batch elements can have different numbers of continuous + embeddings. + + Args: + word_embeddings: Tensor of word embeddings. Shape: [b, s, h] + continuous_embeddings: + Tensor of continuous embeddings. The length of the list is the batch size. Each entry is + shape [num_image_embeddings, hidden], and num_image_embeddings needs to match the number of non-negative + indices in image_patch_input_indices for that batch element. + image_patch_input_indices: Tensor of indices of the image patches in the input_ids tensor. Shape: [b, s] + """ + if not (word_embeddings.shape[0] == len(continuous_embeddings)): + raise ValueError( + f"Batch sizes must match! Got {len(continuous_embeddings)=} and {word_embeddings.shape[0]=}" + ) + + output_embeddings = word_embeddings.clone() + for batch_idx in range(word_embeddings.shape[0]): + # First, find the positions of all the non-negative values in image_patch_input_indices, those are the + # positions in word_embeddings that we want to replace with content from continuous_embeddings. + dst_indices = torch.nonzero(image_patch_input_indices[batch_idx] >= 0, as_tuple=True)[0] + # Next look up those indices in image_patch_input_indices to find the indices in continuous_embeddings that we + # want to use to replace the values in word_embeddings. + src_indices = image_patch_input_indices[batch_idx][dst_indices] + # Check if we have more indices than embeddings. Note that we could have fewer indices if images got truncated. + if src_indices.shape[0] > continuous_embeddings[batch_idx].shape[0]: + raise ValueError( + f"Number of continuous embeddings {continuous_embeddings[batch_idx].shape=} does not match " + f"number of continuous token ids {src_indices.shape=} in batch element {batch_idx}." + ) + output_embeddings[batch_idx, dst_indices] = continuous_embeddings[batch_idx][src_indices] + return output_embeddings + + @add_start_docstrings_to_model_forward(FUYU_INPUTS_DOCSTRING) + def forward( + self, + input_ids: torch.LongTensor = None, + image_patches: torch.Tensor = None, # [batch_size, num_total_patches, patch_size_ x patch_size x num_channels ] + image_patches_indices: torch.Tensor = None, + attention_mask: Optional[torch.Tensor] = None, + position_ids: Optional[torch.LongTensor] = None, + past_key_values: Optional[List[torch.FloatTensor]] = None, + inputs_embeds: Optional[torch.FloatTensor] = None, + use_cache: Optional[bool] = None, + output_attentions: Optional[bool] = None, + output_hidden_states: Optional[bool] = None, + return_dict: Optional[bool] = None, + ) -> Union[Tuple, BaseModelOutputWithPast]: + output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions + output_hidden_states = ( + output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states + ) + use_cache = use_cache if use_cache is not None else self.config.use_cache + + return_dict = return_dict if return_dict is not None else self.config.use_return_dict + + # retrieve input_ids and inputs_embeds + if input_ids is not None and inputs_embeds is not None: + raise ValueError("You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time") + elif input_ids is not None: + batch_size, seq_length = input_ids.shape + elif inputs_embeds is not None: + batch_size, seq_length, _ = inputs_embeds.shape + else: + raise ValueError("You have to specify either decoder_input_ids or decoder_inputs_embeds") + + seq_length_with_past = seq_length + past_key_values_length = 0 + + if past_key_values is not None: + past_key_values_length = past_key_values[0][0].shape[2] + seq_length_with_past = seq_length_with_past + past_key_values_length + + if position_ids is None: + device = input_ids.device if input_ids is not None else inputs_embeds.device + position_ids = torch.arange( + past_key_values_length, seq_length + past_key_values_length, dtype=torch.long, device=device + ) + position_ids = position_ids.unsqueeze(0) + + if inputs_embeds is None: + inputs_embeds = self.language_model.get_input_embeddings()(input_ids) + if image_patches is not None and past_key_values is None: + patch_embeddings = self.vision_embed_tokens(image_patches.to(self.vision_embed_tokens.weight.dtype)) + inputs_embeds = self.gather_continuous_embeddings( + word_embeddings=inputs_embeds, + continuous_embeddings=patch_embeddings, + image_patch_input_indices=image_patches_indices, + ) + + outputs = self.language_model( + inputs_embeds=inputs_embeds, + attention_mask=attention_mask, + position_ids=position_ids, + past_key_values=past_key_values, + output_attentions=output_attentions, + use_cache=use_cache, + ) + if not return_dict: + return tuple(v for v in outputs if v is not None) + return outputs + + def prepare_inputs_for_generation( + self, + input_ids, + past_key_values=None, + attention_mask=None, + inputs_embeds=None, + image_patches=None, + image_patches_indices=None, + **kwargs, + ): + if past_key_values: + input_ids = input_ids[:, -1:] + + position_ids = kwargs.get("position_ids", None) + if attention_mask is not None and position_ids is None: + # create position_ids on the fly for batch generation + position_ids = attention_mask.long().cumsum(-1) - 1 + position_ids.masked_fill_(attention_mask == 0, 1) + if past_key_values: + position_ids = position_ids[:, -1].unsqueeze(-1) + + # if `inputs_embeds` are passed, we only want to use them in the 1st generation step + if inputs_embeds is not None and past_key_values is None: + model_inputs = {"inputs_embeds": inputs_embeds} + else: + model_inputs = {"input_ids": input_ids} + + if image_patches_indices is not None: + model_inputs["image_patches_indices"] = image_patches_indices + + model_inputs.update( + { + "position_ids": position_ids, + "past_key_values": past_key_values, + "use_cache": kwargs.get("use_cache"), + "attention_mask": attention_mask, + "image_patches_indices": image_patches_indices if past_key_values is None else None, + "image_patches": image_patches if past_key_values is None else None, + } + ) + return model_inputs diff --git a/src/transformers/models/fuyu/processing_fuyu.py b/src/transformers/models/fuyu/processing_fuyu.py new file mode 100644 index 0000000000..ea660b072d --- /dev/null +++ b/src/transformers/models/fuyu/processing_fuyu.py @@ -0,0 +1,562 @@ +import re +from typing import Any, Iterable, List, Optional, Tuple, Union + +import numpy as np + +from ...image_utils import ( + ChannelDimension, + get_image_size, + infer_channel_dimension_format, + is_scaled_image, + to_numpy_array, +) +from ...processing_utils import ProcessorMixin +from ...utils import is_torch_available, is_vision_available, logging + + +if is_torch_available() and is_vision_available(): + from .image_processing_fuyu import FuyuImageProcessor + + +logger = logging.get_logger(__name__) + +if is_vision_available(): + import PIL + +if is_torch_available(): + import torch + +BBOX_OPEN_STRING = "<0x00>" # +BBOX_CLOSE_STRING = "<0x01>" # +POINT_OPEN_STRING = "<0x02>" # +POINT_CLOSE_STRING = "<0x03>" # + +TEXT_REPR_BBOX_OPEN = "" +TEXT_REPR_BBOX_CLOSE = "" +TEXT_REPR_POINT_OPEN = "" +TEXT_REPR_POINT_CLOSE = "" + +TOKEN_BBOX_OPEN_STRING = BBOX_OPEN_STRING = "<0x00>" # +BBOX_CLOSE_STRING = "<0x01>" # +TOKEN_BBOX_CLOSE_STRING = TOKEN_POINT_OPEN_STRING = POINT_OPEN_STRING = "<0x02>" # +TOKEN_POINT_CLOSE_STRING = POINT_CLOSE_STRING = "<0x03>" # +BEGINNING_OF_ANSWER_STRING = "<0x04>" # + + +def full_unpacked_stream_to_tensor( + all_bi_tokens_to_place: List[int], + full_unpacked_stream: List["torch.Tensor"], + fill_value: int, + batch_size: int, + new_seq_len: int, + offset: int, +) -> "torch.Tensor": + """Takes an unpacked stream of tokens (i.e. a list of tensors, one for each item in the batch) and does + the required padding to create a single tensor for the batch of shape batch_size x new_seq_len. + """ + + assert len(all_bi_tokens_to_place) == batch_size + assert len(full_unpacked_stream) == batch_size + + # Create padded tensors for the full batch. + new_padded_tensor = torch.full( + [batch_size, new_seq_len], + fill_value=fill_value, + dtype=full_unpacked_stream[0].dtype, + device=full_unpacked_stream[0].device, + ) + + # Place each batch entry into the batch tensor. + for bi in range(batch_size): + tokens_to_place = all_bi_tokens_to_place[bi] + new_padded_tensor[bi, :tokens_to_place] = full_unpacked_stream[bi][offset : tokens_to_place + offset] + + return new_padded_tensor + + +def construct_full_unpacked_stream( + num_real_text_tokens: Union[List[List[int]], "torch.Tensor"], + input_stream: "torch.Tensor", + image_tokens: List[List["torch.Tensor"]], + batch_size: int, + num_sub_sequences: int, +) -> List["torch.Tensor"]: + """Takes an input_stream tensor of shape B x S x ?. For each subsequence, adds any required + padding to account for images and then unpacks the subsequences to create a single sequence per item in the batch. + Returns a list of tensors, one for each item in the batch.""" + + all_bi_stream = [] + + for bi in range(batch_size): + all_si_stream = [] + + # First, construct full token stream (including image placeholder tokens) and loss mask for each subsequence + # and append to lists. We use lists rather than tensors because each subsequence is variable-sized. + for si in range(num_sub_sequences): + image_adjustment = image_tokens[bi][si] + si_stream = torch.cat([image_adjustment, input_stream[bi, si]], dim=0) + num_real_tokens = image_adjustment.shape[0] + num_real_text_tokens[bi][si] + + all_si_stream.append(si_stream[:num_real_tokens]) + # Combine all subsequences for this batch entry. Still using a list because each batch entry is variable-sized. + all_bi_stream.append(torch.cat(all_si_stream, dim=0)) + + return all_bi_stream + + +def _replace_string_repr_with_token_tags(prompt: str) -> str: + prompt = prompt.replace(TEXT_REPR_POINT_OPEN, TOKEN_POINT_OPEN_STRING) + prompt = prompt.replace(TEXT_REPR_POINT_CLOSE, TOKEN_POINT_CLOSE_STRING) + prompt = prompt.replace(TEXT_REPR_BBOX_OPEN, TOKEN_BBOX_OPEN_STRING) + prompt = prompt.replace(TEXT_REPR_BBOX_CLOSE, TOKEN_BBOX_CLOSE_STRING) + return prompt + + +def _segment_prompt_into_text_token_conversions(prompt: str) -> List: + """ + Given a string prompt, converts the prompt into a list of TextTokenConversions. + """ + # Wherever, we notice the [TOKEN_OPEN_STRING, TOKEN_CLOSE_STRING], we split the prompt + prompt_text_list: List = [] + regex_pattern = re.compile( + f"({TOKEN_BBOX_OPEN_STRING}|{TOKEN_BBOX_CLOSE_STRING}|{TOKEN_POINT_OPEN_STRING}|{TOKEN_POINT_CLOSE_STRING})" + ) + # Split by the regex pattern + prompt_split = regex_pattern.split(prompt) + for i, elem in enumerate(prompt_split): + if len(elem) == 0 or elem in [ + TOKEN_BBOX_OPEN_STRING, + TOKEN_BBOX_CLOSE_STRING, + TOKEN_POINT_OPEN_STRING, + TOKEN_POINT_CLOSE_STRING, + ]: + continue + prompt_text_list.append( + (elem, i > 1 and prompt_split[i - 1] in [TOKEN_BBOX_OPEN_STRING, TOKEN_POINT_OPEN_STRING]) + ) + return prompt_text_list + + +def _transform_coordinates_and_tokenize(prompt: str, transformed_image, tokenizer) -> List[int]: + """ + This function transforms the prompt in the following fashion: + - and to their respective token mappings + - extract the coordinates from the tag + - transform the coordinates into the transformed image space + - return the prompt tokens with the transformed coordinates and new tags + + Bounding boxes and points MUST be in the following format: y1, x1, y2, x2 x, y The spaces + and punctuation added above are NOT optional. + """ + # Make a namedtuple that stores "text" and "is_bbox" + + # We want to do the following: Tokenize the code normally -> when we see a point or box, tokenize using the tokenize_within_tag function + # When point or box close tag, continue tokenizing normally + # First, we replace the point and box tags with their respective tokens + prompt = _replace_string_repr_with_token_tags(prompt) + # Tokenize the prompt + # Convert prompt into a list split + prompt_text_list = _segment_prompt_into_text_token_conversions(prompt) + transformed_prompt_tokens: List[int] = [] + for elem in prompt_text_list: + if elem[1]: + # This is a location, we need to tokenize it + within_tag_tokenized = _transform_within_tags(elem[0], transformed_image, tokenizer) + # Surround the text with the open and close tags + transformed_prompt_tokens.extend(within_tag_tokenized) + else: + transformed_prompt_tokens.extend(tokenizer(elem[0], add_special_tokens=False).input_ids) + return transformed_prompt_tokens + + +def _transform_within_tags(text: str, transformed_image, tokenizer) -> List[int]: + """ + Given a bounding box of the fashion 1, 2, 3, 4 | 1, 2 This function is responsible for + converting 1, 2, 3, 4 into tokens of 1 2 3 4 without any commas. + """ + # Convert the text into a list of strings. + num_int_strs = text.split(",") + if len(num_int_strs) == 2: + # If there are any open or close tags, remove them. + token_space_open_string = tokenizer.vocab[TOKEN_POINT_OPEN_STRING] + token_space_close_string = tokenizer.vocab[TOKEN_POINT_CLOSE_STRING] + else: + token_space_open_string = tokenizer.vocab[TOKEN_BBOX_OPEN_STRING] + token_space_close_string = tokenizer.vocab[TOKEN_BBOX_CLOSE_STRING] + + # Remove all spaces from num_ints + num_ints = [float(num.strip()) for num in num_int_strs] + # scale to transformed image siz + if len(num_ints) == 2: + num_ints_translated = scale_point_to_transformed_image( + x=num_ints[0], y=num_ints[1], transformed_image=transformed_image + ) + elif len(num_ints) == 4: + num_ints_translated = scale_bbox_to_transformed_image( + top=num_ints[0], + left=num_ints[1], + bottom=num_ints[2], + right=num_ints[3], + transformed_image=transformed_image, + ) + else: + raise ValueError(f"Invalid number of ints: {len(num_ints)}") + # Tokenize the text, skipping the + tokens = [tokenizer.vocab[str(num)] for num in num_ints_translated] + return [token_space_open_string] + tokens + [token_space_close_string] + + +def _tokenize_prompts_with_image_and_batch( + tokenizer, + prompts: List[List[str]], + transformed_images: Optional[List[List["torch.Tensor"]]], + max_tokens_to_generate: int, + max_position_embeddings: int, + add_BOS: bool, # Same issue with types as above + add_beginning_of_answer_token: bool, +) -> Tuple["torch.Tensor", "torch.Tensor"]: + """ + Given a set of prompts and number of tokens to generate: + - tokenize prompts + - set the sequence length to be the max of length of prompts plus the number of tokens we would like to generate + - pad all the sequences to this length so we can convert them into a 3D tensor. + """ + + # If not tool use, tranform the coordinates while tokenizing + if transformed_images is not None: + transformed_prompt_tokens = [] + for prompt_seq, transformed_image_seq in zip(prompts, transformed_images): + transformed_prompt_tokens.append( + [ + _transform_coordinates_and_tokenize(prompt, transformed_image, tokenizer) + for prompt, transformed_image in zip(prompt_seq, transformed_image_seq) + ] + ) + else: + transformed_prompt_tokens = [[tokenizer.tokenize(prompt) for prompt in prompt_seq] for prompt_seq in prompts] + + prompts_tokens = transformed_prompt_tokens + + if add_BOS: + bos_token = tokenizer.vocab[""] + else: + bos_token = tokenizer.vocab["|ENDOFTEXT|"] + prompts_tokens = [[[bos_token] + x for x in prompt_seq] for prompt_seq in prompts_tokens] + if add_beginning_of_answer_token: + boa = tokenizer.vocab[BEGINNING_OF_ANSWER_STRING] + # Only add bbox open token to the last subsequence since that is what will be completed + for token_seq in prompts_tokens: + token_seq[-1].append(boa) + + # Now we have a list of list of tokens which each list has a different + # size. We want to extend this list to: + # - incorporate the tokens that need to be generated + # - make all the sequences equal length. + # Get the prompts length. + + prompts_length = [[len(x) for x in prompts_tokens_seq] for prompts_tokens_seq in prompts_tokens] + # Get the max prompts length. + max_prompt_len: int = np.max(prompts_length) + # Number of tokens in the each sample of the batch. + samples_length = min(max_prompt_len + max_tokens_to_generate, max_position_embeddings) + if max_prompt_len + max_tokens_to_generate > max_position_embeddings: + print( + f"Max subsequence prompt length of {max_prompt_len} + max tokens to generate {max_tokens_to_generate}", + f"exceeds context length of {max_position_embeddings}. Will generate as many tokens as possible.", + ) + # Now update the list of list to be of the same size: samples_length. + for prompt_tokens_seq, prompts_length_seq in zip(prompts_tokens, prompts_length): + for prompt_tokens, prompt_length in zip(prompt_tokens_seq, prompts_length_seq): + if len(prompt_tokens) > samples_length: + raise ValueError("Length of subsequence prompt exceeds sequence length.") + padding_size = samples_length - prompt_length + prompt_tokens.extend([tokenizer.vocab["|ENDOFTEXT|"]] * padding_size) + + # Now we are in a structured format, we can convert to tensors. + prompts_tokens_tensor = torch.tensor(prompts_tokens, dtype=torch.int64) + prompts_length_tensor = torch.tensor(prompts_length, dtype=torch.int64) + + return prompts_tokens_tensor, prompts_length_tensor + + +def original_to_transformed_h_coords(self, original_coords): + # apply crop + cropped_coords = ( + self._clamp_coords(original_coords, min_value=self.crop_top, max_value=self.crop_bottom) - self.crop_top + ) + # apply scale + scaled_coords = self._scale_coords(cropped_coords, scale=self.scaled_h / self.original_h) + # apply pad + return scaled_coords + self.padding_top + + +def original_to_transformed_w_coords(self, original_coords): + # apply crop + cropped_coords = ( + self._clamp_coords(original_coords, min_value=self.crop_left, max_value=self.crop_right) - self.crop_left + ) + # apply scale + scaled_coords = self._scale_coords(cropped_coords, scale=self.scaled_w / self.original_w) + # apply pad + return scaled_coords + self.padding_left + + +def scale_point_to_transformed_image(x: float, y: float) -> List[int]: + x_scaled = original_to_transformed_w_coords(np.array([x / 2]))[0] + y_scaled = original_to_transformed_h_coords(np.array([y / 2]))[0] + return [x_scaled, y_scaled] + + +def scale_bbox_to_transformed_image(top: float, left: float, bottom: float, right: float) -> List[int]: + top_scaled = original_to_transformed_w_coords(np.array([top / 2]))[0] + left_scaled = original_to_transformed_h_coords(np.array([left / 2]))[0] + bottom_scaled = original_to_transformed_w_coords(np.array([bottom / 2]))[0] + right_scaled = original_to_transformed_h_coords(np.array([right / 2]))[0] + return [top_scaled, left_scaled, bottom_scaled, right_scaled] + + +# Copied from transformers.models.detr.image_processing_detr.max_across_indices +def max_across_indices(values: Iterable[Any]) -> List[Any]: + """ + Return the maximum value across all indices of an iterable of values. + """ + return [max(values_i) for values_i in zip(*values)] + + +# Copied from transformers.models.detr.image_processing_detr.get_max_height_width +def get_max_height_width( + images: List[np.ndarray], input_data_format: Optional[Union[str, ChannelDimension]] = None +) -> List[int]: + """ + Get the maximum height and width across all images in a batch. + """ + if input_data_format is None: + input_data_format = infer_channel_dimension_format(images[0]) + + if input_data_format == ChannelDimension.FIRST: + _, max_height, max_width = max_across_indices([img.shape for img in images]) + elif input_data_format == ChannelDimension.LAST: + max_height, max_width, _ = max_across_indices([img.shape for img in images]) + else: + raise ValueError(f"Invalid channel dimension format: {input_data_format}") + return (max_height, max_width) + + +# Copied from transformers.models.detr.image_processing_detr.make_pixel_mask +def make_pixel_mask( + image: np.ndarray, output_size: Tuple[int, int], input_data_format: Optional[Union[str, ChannelDimension]] = None +) -> np.ndarray: + """ + Make a pixel mask for the image, where 1 indicates a valid pixel and 0 indicates padding. + + Args: + image (`np.ndarray`): + Image to make the pixel mask for. + output_size (`Tuple[int, int]`): + Output size of the mask. + """ + input_height, input_width = get_image_size(image, channel_dim=input_data_format) + mask = np.zeros(output_size, dtype=np.int64) + mask[:input_height, :input_width] = 1 + return mask + + +class FuyuProcessor(ProcessorMixin): + r""" + Constructs a Fuyu processor which wraps a Fuyu image processor and a Llama tokenizer into a single processor. + + [`FuyuProcessor`] offers all the functionalities of [`FuyuImageProcessor`] and [`LlamaTokenizerFast`]. See the + [`~FuyuProcessor.__call__`] and [`~FuyuProcessor.decode`] for more information. + + Args: + image_processor ([`FuyuImageProcessor`]): + The image processor is a required input. + tokenizer ([`LlamaTokenizerFast`]): + The tokenizer is a required input. + """ + attributes = ["image_processor", "tokenizer"] + image_processor_class = "FuyuImageProcessor" + tokenizer_class = "AutoTokenizer" + + def __init__(self, image_processor, tokenizer): + super().__init__(image_processor=image_processor, tokenizer=tokenizer) + self.image_processor = image_processor + self.tokenizer = tokenizer + self.max_tokens_to_generate = 10 + self.max_position_embeddings = 16384 # TODO Can't derive this from model files: where to set it? + self.image_processor = FuyuImageProcessor() + + def _process_images(self, images): + """Utility function to preprocess the images and extract necessary information about original formats.""" + batch_images = [] + image_unpadded_heights = [] + image_unpadded_widths = [] + + for image in images: + image = to_numpy_array(image) + if not is_scaled_image(image): + image = image / 255.0 + channel_dimension = infer_channel_dimension_format(image, 3) + if channel_dimension == ChannelDimension.FIRST: + width_index = 2 + height_index = 1 + elif channel_dimension == ChannelDimension.LAST: + width_index = 1 + height_index = 0 + + image_unpadded_widths.append([image.shape[width_index]]) + image_unpadded_heights.append([image.shape[height_index]]) + + # Reproduct adept padding sampler + padded_image = self.image_processor.apply_transformation(image) + + tensor_img = torch.Tensor(padded_image).permute(2, 0, 1) + batch_images.append([tensor_img]) + + return batch_images, torch.Tensor(image_unpadded_heights), torch.Tensor(image_unpadded_widths) + + def __call__(self, text=None, images=None, return_tensors=None, **kwargs): + """ + Main method to prepare for the model one or several sequences(s) and image(s). This method forwards the `text` + and `kwargs` arguments to LlamaTokenizerFast's [`~LlamaTokenizerFast.__call__`] if `text` is not `None` to + encode the text. To prepare the image(s), this method forwards the `images` and `kwrags` arguments to + FuyuImageProcessor's [`~FuyuImageProcessor.__call__`] if `images` is not `None`. Please refer to the doctsring + of the above two methods for more information. + + Args: + text (`str`, `List[str]`): + The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings + (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set + `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). + images (`PIL.Image.Image`, `List[PIL.Image.Image]`): + The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch + tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a + number of channels, H and W are image height and width. + + return_tensors (`str` or [`~utils.TensorType`], *optional*): + If set, will return tensors of a particular framework. Acceptable values are: + + - `'tf'`: Return TensorFlow `tf.constant` objects. + - `'pt'`: Return PyTorch `torch.Tensor` objects. + - `'np'`: Return NumPy `np.ndarray` objects. + - `'jax'`: Return JAX `jnp.ndarray` objects. + + Returns: + [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: + + - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. + - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when + `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not + `None`). + - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. + """ + if text is None and images is None: + raise ValueError("You have to specify either text or images. Both cannot be none.") + if text is not None and images is not None: + if isinstance(text, str): + prompts = [[text]] + elif isinstance(text, list): + prompts = [[text_seq] for text_seq in text] + batch_images = [] + if isinstance(images, PIL.Image.Image): + images = [images] + if isinstance(images, list): + batch_images, image_unpadded_heights, image_unpadded_widths = self._process_images(images) + # image_unpadded_heights = image_unpadded_heights.unsqueeze(0) + # image_unpadded_widths = image_unpadded_widths.unsqueeze(0) + else: + raise ValueError("images must be a list of ndarrays or PIL Images to be processed.") + + # Note: the original adept code has a handling of image_unpadded_h and w, but it doesn't seem to hold + # when there are several different size subsequences per batch. The current implementation reflects + # that limitation and should be documented. + # + self.subsequence_length = 1 # Each batch contains only one sequence. + self.batch_size = len(batch_images) + # FIXME max_tokens_to_generate is embedded into this processor's call. + prompt_tokens, prompts_length = _tokenize_prompts_with_image_and_batch( + tokenizer=self.tokenizer, + prompts=prompts, + transformed_images=batch_images, + max_tokens_to_generate=self.max_tokens_to_generate, + max_position_embeddings=self.max_position_embeddings, + add_BOS=True, + add_beginning_of_answer_token=True, + ) + # same so far + + # This is 1 if there is an image per subsequence, else 0. [batch, 1, presence] + # the remainder of current image processing logic assumes subsequence_size = 1. + # Here it is OK as the model cannot handle > 1 subsequences + # the image could be absent however and image presence should be inferred from user batch input + # hence this code assumes the images are present. Use an assert? + + image_present = torch.ones(self.batch_size, 1, 1) + + image_placeholder_id = self.tokenizer("|SPEAKER|", add_special_tokens=False)["input_ids"][1] + image_newline_id = self.tokenizer("|NEWLINE|", add_special_tokens=False)["input_ids"][1] + tensor_batch_images = torch.stack([img[0] for img in batch_images]).unsqueeze(1) + model_image_input = self.image_processor.process_images_for_model_input( + image_input=tensor_batch_images, + image_present=image_present, + image_unpadded_h=image_unpadded_heights, + image_unpadded_w=image_unpadded_widths, + image_patch_dim_h=30, + image_patch_dim_w=30, + image_placeholder_id=image_placeholder_id, + image_newline_id=image_newline_id, + variable_sized=True, + ) + + image_padded_unpacked_tokens = construct_full_unpacked_stream( + num_real_text_tokens=prompts_length, + input_stream=prompt_tokens, + image_tokens=model_image_input["image_input_ids"], + batch_size=self.batch_size, + num_sub_sequences=self.subsequence_length, + ) + # Construct inputs for image patch indices. + unpacked_image_patch_indices_per_batch = construct_full_unpacked_stream( + num_real_text_tokens=prompts_length, + input_stream=torch.full_like(prompt_tokens, -1), + image_tokens=model_image_input["image_patch_indices_per_batch"], + batch_size=self.batch_size, + num_sub_sequences=self.subsequence_length, + ) + max_prompt_length = max(x.shape[-1] for x in image_padded_unpacked_tokens) + max_seq_len_batch = min(max_prompt_length + self.max_tokens_to_generate, self.max_position_embeddings) + all_bi_tokens_to_place = [] + for bi in range(self.batch_size): + tokens_to_place = min(max_seq_len_batch, max(0, image_padded_unpacked_tokens[bi].shape[0])) + all_bi_tokens_to_place.append(tokens_to_place) + + # Use same packing logic for the image patch indices. + image_patch_input_indices = full_unpacked_stream_to_tensor( + all_bi_tokens_to_place=all_bi_tokens_to_place, + full_unpacked_stream=unpacked_image_patch_indices_per_batch, + fill_value=-1, + batch_size=self.batch_size, + new_seq_len=max_seq_len_batch, + offset=0, + ) + + image_patches_tensor = torch.stack([img[0] for img in model_image_input["image_patches"]]).unsqueeze(1) + return { + "input_ids": image_padded_unpacked_tokens[0].unsqueeze(0), + "image_patches": image_patches_tensor[0][0].unsqueeze(0), + "image_patches_indices": image_patch_input_indices, + } + + def batch_decode(self, *args, **kwargs): + """ + This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please + refer to the docstring of this method for more information. + """ + return self.tokenizer.batch_decode(*args, **kwargs) + + def decode(self, *args, **kwargs): + """ + This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to + the docstring of this method for more information. + """ + return self.tokenizer.decode(*args, **kwargs) diff --git a/src/transformers/utils/dummy_pt_objects.py b/src/transformers/utils/dummy_pt_objects.py index 2991bca449..2ab6279237 100644 --- a/src/transformers/utils/dummy_pt_objects.py +++ b/src/transformers/utils/dummy_pt_objects.py @@ -3614,6 +3614,20 @@ def load_tf_weights_in_funnel(*args, **kwargs): requires_backends(load_tf_weights_in_funnel, ["torch"]) +class FuyuForCausalLM(metaclass=DummyObject): + _backends = ["torch"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch"]) + + +class FuyuPreTrainedModel(metaclass=DummyObject): + _backends = ["torch"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch"]) + + GIT_PRETRAINED_MODEL_ARCHIVE_LIST = None diff --git a/src/transformers/utils/dummy_vision_objects.py b/src/transformers/utils/dummy_vision_objects.py index c0c39b57d0..0af6ef347d 100644 --- a/src/transformers/utils/dummy_vision_objects.py +++ b/src/transformers/utils/dummy_vision_objects.py @@ -219,6 +219,13 @@ class FlavaProcessor(metaclass=DummyObject): requires_backends(self, ["vision"]) +class FuyuImageProcessor(metaclass=DummyObject): + _backends = ["vision"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["vision"]) + + class GLPNFeatureExtractor(metaclass=DummyObject): _backends = ["vision"] diff --git a/tests/models/fuyu/__init__.py b/tests/models/fuyu/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/models/fuyu/test_image_processing_fuyu.py b/tests/models/fuyu/test_image_processing_fuyu.py new file mode 100644 index 0000000000..ee02052c7c --- /dev/null +++ b/tests/models/fuyu/test_image_processing_fuyu.py @@ -0,0 +1,65 @@ +import unittest + +import numpy as np + +from transformers import is_torch_available, is_vision_available +from transformers.testing_utils import ( + require_torch, + require_torchvision, + require_vision, +) + + +if is_torch_available() and is_vision_available(): + import torch + + from transformers import FuyuImageProcessor + +if is_vision_available(): + from PIL import Image + + +@require_torch +@require_vision +@require_torchvision +class TestFuyuImageProcessor(unittest.TestCase): + def setUp(self): + self.processor = FuyuImageProcessor(target_height=160, target_width=320, padding_value=1.0) + self.batch_size = 3 + self.channels = 3 + self.height = 300 + self.width = 300 + + self.image_input = torch.rand(self.batch_size, self.channels, self.height, self.width) + + self.image_patch_dim_h = 30 + self.image_patch_dim_w = 30 + self.sample_image = np.zeros((450, 210, 3), dtype=np.uint8) + self.sample_image_pil = Image.fromarray(self.sample_image) + + def test_patches(self): + expected_num_patches = self.processor.get_num_patches( + img_h=self.height, img_w=self.width, patch_dim_h=self.image_patch_dim_h, patch_dim_w=self.image_patch_dim_w + ) + + patches_final = self.processor.patchify_image( + image=self.image_input, patch_dim_h=self.image_patch_dim_h, patch_dim_w=self.image_patch_dim_w + ) + assert ( + patches_final.shape[1] == expected_num_patches + ), f"Expected {expected_num_patches} patches, got {patches_final.shape[1]}." + + def test_scale_to_target_aspect_ratio(self): + scaled_image = self.processor._scale_to_target_aspect_ratio(self.sample_image) + self.assertEqual(scaled_image.shape[0], 74) + self.assertEqual(scaled_image.shape[1], 160) + + def test_apply_transformation_numpy(self): + transformed_image = self.processor.apply_transformation(self.sample_image) + self.assertEqual(transformed_image.shape[0], 160) + self.assertEqual(transformed_image.shape[1], 320) + + def test_apply_transformation_pil(self): + transformed_image = self.processor.apply_transformation(self.sample_image_pil) + self.assertEqual(transformed_image.shape[0], 160) + self.assertEqual(transformed_image.shape[1], 320) diff --git a/tests/models/fuyu/test_modeling_fuyu.py b/tests/models/fuyu/test_modeling_fuyu.py new file mode 100644 index 0000000000..58d671bd57 --- /dev/null +++ b/tests/models/fuyu/test_modeling_fuyu.py @@ -0,0 +1,362 @@ +import io +import unittest + +import requests + +from transformers import AutoTokenizer, FuyuConfig, is_torch_available, is_vision_available +from transformers.testing_utils import require_torch, require_torch_gpu, slow, torch_device + +from ...test_modeling_common import ids_tensor, random_attention_mask + + +if is_vision_available(): + from PIL import Image + + +if is_torch_available() and is_vision_available(): + from transformers import FuyuImageProcessor, FuyuProcessor + + +if is_torch_available(): + import torch + + from transformers import FuyuForCausalLM + + +# Copied from transformers.tests.llama.test_modelling_llama.LlamaModelTest with Llama->Fuyu +class FuyuModelTester: + def __init__( + self, + parent, + batch_size=13, + seq_length=7, + image_size=300, + patch_size=30, + num_channels=3, + is_training=True, + use_input_mask=True, + use_token_type_ids=False, + use_labels=True, + vocab_size=99, + hidden_size=32, + num_hidden_layers=2, + num_attention_heads=4, + intermediate_size=37, + hidden_act="gelu", + hidden_dropout_prob=0.1, + attention_probs_dropout_prob=0.1, + max_position_embeddings=512, + type_vocab_size=16, + type_sequence_label_size=2, + initializer_range=0.02, + num_labels=3, + num_choices=4, + pad_token_id=0, + scope=None, + ): + self.parent = parent + self.batch_size = batch_size + self.seq_length = seq_length + self.image_size = image_size + self.patch_size = patch_size + self.num_channels = num_channels + self.is_training = is_training + self.use_input_mask = use_input_mask + self.use_token_type_ids = use_token_type_ids + self.use_labels = use_labels + self.vocab_size = vocab_size + self.hidden_size = hidden_size + self.num_hidden_layers = num_hidden_layers + self.num_attention_heads = num_attention_heads + self.intermediate_size = intermediate_size + self.hidden_act = hidden_act + self.hidden_dropout_prob = hidden_dropout_prob + self.attention_probs_dropout_prob = attention_probs_dropout_prob + self.max_position_embeddings = max_position_embeddings + self.type_vocab_size = type_vocab_size + self.type_sequence_label_size = type_sequence_label_size + self.initializer_range = initializer_range + self.num_labels = num_labels + self.num_choices = num_choices + self.pad_token_id = pad_token_id + self.scope = scope + + def prepare_config_and_inputs(self): + input_ids = ids_tensor([self.batch_size, self.seq_length], self.vocab_size) + + input_mask = None + if self.use_input_mask: + input_mask = random_attention_mask([self.batch_size, self.seq_length]) + + token_type_ids = None + if self.use_token_type_ids: + token_type_ids = ids_tensor([self.batch_size, self.seq_length], self.type_vocab_size) + + sequence_labels = None + token_labels = None + choice_labels = None + if self.use_labels: + sequence_labels = ids_tensor([self.batch_size], self.type_sequence_label_size) + token_labels = ids_tensor([self.batch_size, self.seq_length], self.num_labels) + choice_labels = ids_tensor([self.batch_size], self.num_choices) + + config = self.get_config() + + return config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels, choice_labels + + def get_config(self): + return FuyuConfig( + vocab_size=self.vocab_size, + hidden_size=self.hidden_size, + num_hidden_layers=self.num_hidden_layers, + num_attention_heads=self.num_attention_heads, + intermediate_size=self.intermediate_size, + hidden_act=self.hidden_act, + hidden_dropout_prob=self.hidden_dropout_prob, + attention_probs_dropout_prob=self.attention_probs_dropout_prob, + max_position_embeddings=self.max_position_embeddings, + type_vocab_size=self.type_vocab_size, + is_decoder=False, + initializer_range=self.initializer_range, + pad_token_id=self.pad_token_id, + ) + + def create_and_check_model( + self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels, choice_labels + ): + model = FuyuForCausalLM(config=config) + model.to(torch_device) + model.eval() + result = model(input_ids, attention_mask=input_mask) + result = model(input_ids) + self.parent.assertEqual(result.last_hidden_state.shape, (self.batch_size, self.seq_length, self.hidden_size)) + + def create_and_check_model_as_decoder( + self, + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + encoder_hidden_states, + encoder_attention_mask, + ): + config.add_cross_attention = True + model = FuyuForCausalLM(config) + model.to(torch_device) + model.eval() + result = model( + input_ids, + attention_mask=input_mask, + encoder_hidden_states=encoder_hidden_states, + encoder_attention_mask=encoder_attention_mask, + ) + result = model( + input_ids, + attention_mask=input_mask, + encoder_hidden_states=encoder_hidden_states, + ) + result = model(input_ids, attention_mask=input_mask) + self.parent.assertEqual(result.last_hidden_state.shape, (self.batch_size, self.seq_length, self.hidden_size)) + + def create_and_check_for_causal_lm( + self, + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + encoder_hidden_states, + encoder_attention_mask, + ): + model = FuyuForCausalLM(config=config) + model.to(torch_device) + model.eval() + result = model(input_ids, attention_mask=input_mask, labels=token_labels) + self.parent.assertEqual(result.logits.shape, (self.batch_size, self.seq_length, self.vocab_size)) + + def create_and_check_decoder_model_past_large_inputs( + self, + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + encoder_hidden_states, + encoder_attention_mask, + ): + config.is_decoder = True + config.add_cross_attention = True + model = FuyuForCausalLM(config=config) + model.to(torch_device) + model.eval() + + # first forward pass + outputs = model( + input_ids, + attention_mask=input_mask, + encoder_hidden_states=encoder_hidden_states, + encoder_attention_mask=encoder_attention_mask, + use_cache=True, + ) + past_key_values = outputs.past_key_values + + # create hypothetical multiple next token and extent to next_input_ids + next_tokens = ids_tensor((self.batch_size, 3), config.vocab_size) + next_mask = ids_tensor((self.batch_size, 3), vocab_size=2) + + # append to next input_ids and + next_input_ids = torch.cat([input_ids, next_tokens], dim=-1) + next_attention_mask = torch.cat([input_mask, next_mask], dim=-1) + + output_from_no_past = model( + next_input_ids, + attention_mask=next_attention_mask, + encoder_hidden_states=encoder_hidden_states, + encoder_attention_mask=encoder_attention_mask, + output_hidden_states=True, + )["hidden_states"][0] + output_from_past = model( + next_tokens, + attention_mask=next_attention_mask, + encoder_hidden_states=encoder_hidden_states, + encoder_attention_mask=encoder_attention_mask, + past_key_values=past_key_values, + output_hidden_states=True, + )["hidden_states"][0] + + # select random slice + random_slice_idx = ids_tensor((1,), output_from_past.shape[-1]).item() + output_from_no_past_slice = output_from_no_past[:, -3:, random_slice_idx].detach() + output_from_past_slice = output_from_past[:, :, random_slice_idx].detach() + + self.parent.assertTrue(output_from_past_slice.shape[1] == next_tokens.shape[1]) + + # test that outputs are equal for slice + self.parent.assertTrue(torch.allclose(output_from_past_slice, output_from_no_past_slice, atol=1e-3)) + + def prepare_config_and_inputs_for_common(self): + config_and_inputs = self.prepare_config_and_inputs() + ( + config, + input_ids, + token_type_ids, + input_mask, + sequence_labels, + token_labels, + choice_labels, + ) = config_and_inputs + inputs_dict = {"input_ids": input_ids, "attention_mask": input_mask} + return config, inputs_dict + + +@require_torch +@require_torch_gpu +@slow +class FuyuIntegrationTest(unittest.TestCase): # , ModelTesterMixin) + """ + Currently, all these tests depend on a value of max_tokens_to_generate of 10. + """ + + all_model_classes = ("FuyuForCausalLM") if is_torch_available() else () + + def setUp(self): + self.pretrained_model_name = "huggingface/new_model_release_weights" + tokenizer = AutoTokenizer.from_pretrained(self.pretrained_model_name) + image_processor = FuyuImageProcessor() + + self.processor = FuyuProcessor(image_processor=image_processor, tokenizer=tokenizer) + self.model = FuyuForCausalLM.from_pretrained(self.pretrained_model_name) + self.bus_image_url = ( + "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/bus.png" + ) + self.bus_image_pil = Image.open(io.BytesIO(requests.get(self.bus_image_url).content)) + + @slow + @require_torch_gpu + def test_model_8b_chat_greedy_generation_bus_captioning(self): + EXPECTED_TEXT_COMPLETION = """A bus parked on the side of a road.|ENDOFTEXT|""" + text_prompt_coco_captioning = "Generate a coco-style caption.\n" + + model_inputs_bus_captioning = self.processor(text=text_prompt_coco_captioning, images=self.bus_image_pil) + generated_tokens = self.model.generate(**model_inputs_bus_captioning, max_new_tokens=10) + text = self.processor.tokenizer.batch_decode(generated_tokens) + end_sequence = text[0].split("\x04")[1] + clean_sequence = ( + end_sequence[: end_sequence.find("|ENDOFTEXT|") + len("|ENDOFTEXT|")] + if "|ENDOFTEXT|" in end_sequence + else end_sequence + ) + self.assertEqual(EXPECTED_TEXT_COMPLETION, clean_sequence[1:]) + + +""" + @slow + @require_torch_gpu + def test_model_8b_chat_greedy_generation_bus_color(self): + EXPECTED_TEXT_COMPLETION = "The bus is blue.\n|ENDOFTEXT|" + text_prompt_bus_color = "What color is the bus?\n" + model_inputs_bus_color = self.processor(text=text_prompt_bus_color, images=self.bus_image_pil) + + generated_tokens = self.model.generate(**model_inputs_bus_color, max_new_tokens=10) + text = self.processor.tokenizer.batch_decode(generated_tokens) + end_sequence = text[0].split("\x04")[1] + clean_sequence = ( + end_sequence[: end_sequence.find("|ENDOFTEXT|") + len("|ENDOFTEXT|")] + if "|ENDOFTEXT|" in end_sequence + else end_sequence + ) + self.assertEqual(EXPECTED_TEXT_COMPLETION, clean_sequence) + + @slow + @require_torch_gpu + def test_model_8b_chat_greedy_generation_chart_vqa(self): + # fmt: off + EXPECTED_TEXT_TOKENS = ["The","life expectancy","at","birth","of male","s in","","20","18","is","","80",".","7",".","\n","|ENDOFTEXT|",] + # fmt: on + expected_text_completion = " ".join(EXPECTED_TEXT_TOKENS) # TODO make sure the end string matches + + text_prompt_chart_vqa = "What is the highest life expectancy at birth of male?\n" + + chart_image_url = ( + "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/chart.png" + ) + chart_image_pil = Image.open(io.BytesIO(requests.get(chart_image_url).content)) + + model_inputs_chart_vqa = self.processor(text=text_prompt_chart_vqa, images=chart_image_pil) + generated_tokens = self.model.generate(**model_inputs_chart_vqa, max_new_tokens=10) + text = self.processor.tokenizer.batch_decode(generated_tokens) + end_sequence = text[0].split("\x04")[1] + clean_sequence = ( + end_sequence[: end_sequence.find("|ENDOFTEXT|") + len("|ENDOFTEXT|")] + if "|ENDOFTEXT|" in end_sequence + else end_sequence + ) + self.assertEqual(expected_text_completion, clean_sequence) + + @slow + @require_torch_gpu + def test_model_8b_chat_greedy_generation_bounding_box(self): + EXPECTED_TEXT_COMPLETION = "\x00194213202244\x01|ENDOFTEXT|" + text_prompt_bbox = "When presented with a box, perform OCR to extract text contained within it. If provided with text, generate the corresponding bounding box.\\nWilliams" # noqa: E231 + + bbox_image_url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/bbox_sample_image.png" + bbox_image_pil = Image.open(io.BytesIO(requests.get(bbox_image_url).content)) + + model_inputs_bbox = self.processor(text=text_prompt_bbox, images=bbox_image_pil) + generated_tokens = self.model.generate(**model_inputs_bbox, max_new_tokens=10) + text = self.processor.tokenizer.batch_decode(generated_tokens) + end_sequence = text[0].split("\x04")[1] + clean_sequence = ( + end_sequence[: end_sequence.find("|ENDOFTEXT|") + len("|ENDOFTEXT|")] + if "|ENDOFTEXT|" in end_sequence + else end_sequence + ) + self.assertEqual(EXPECTED_TEXT_COMPLETION, clean_sequence) +""" diff --git a/tests/models/fuyu/test_processing_fuyu.py b/tests/models/fuyu/test_processing_fuyu.py new file mode 100644 index 0000000000..1c75b2b0ae --- /dev/null +++ b/tests/models/fuyu/test_processing_fuyu.py @@ -0,0 +1,126 @@ +import io +import unittest + +import requests + +from transformers import AutoTokenizer, is_torch_available, is_vision_available +from transformers.testing_utils import require_torch, require_torch_gpu, slow + + +if is_vision_available(): + from PIL import Image + +if is_vision_available() and is_torch_available(): + from transformers import FuyuImageProcessor, FuyuProcessor + +if is_torch_available(): + import torch + + from transformers.models.fuyu.processing_fuyu import construct_full_unpacked_stream, full_unpacked_stream_to_tensor + + +@require_torch +@require_torch_gpu +@slow +class FuyuProcessingTest(unittest.TestCase): # TODO Which mixins do we add here? + """ """ + + def setUp(self): + pretrained_model_name = "huggingface/pre_release_model" + tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name) + image_processor = FuyuImageProcessor() + + processor = FuyuProcessor(image_processor=image_processor, tokenizer=tokenizer) + text_prompt = "Generate a coco-style caption.\\n" + bus_image_url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/bus.png" + bus_image_pil = Image.open(io.BytesIO(requests.get(bus_image_url).content)) + + self.one_image_bus_model_inputs = processor(text=text_prompt, images=bus_image_pil) + + def test_fuyu_processing(self): + """ + Test to ensure that the standard processing on a gold example matches adept's code. + """ + # fmt: off + EXPECTED_IMAGE_PATCH_INPUTS = torch.Tensor([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, -1, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, -1, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, -1, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, -1, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, -1, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, -1, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, -1, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, -1, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, -1, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, -1, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, -1, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, -1, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, -1, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,]]).to(torch.int64) + EXPECTED_PADDED_UNPACKED_TOKEN_INPUTS = torch.Tensor([[71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71011, 71019, 1, 128340, 71374, 71389, 120412, 71377, 71835, 71374, 73615, 71375, 71399, 71435, 71122,]]).to(torch.int64) + # fmt: on + torch.testing.assert_close( + self.one_image_bus_model_inputs["image_patches_indices"], EXPECTED_IMAGE_PATCH_INPUTS + ) + torch.testing.assert_close(self.one_image_bus_model_inputs["input_ids"], EXPECTED_PADDED_UNPACKED_TOKEN_INPUTS) + + +@require_torch +class TestImageTextProcessingUtils(unittest.TestCase): + def setUp(self): + self.batch_size = 2 + self.new_seq_len = 8 + self.num_sub_sequences = 1 + + self.all_bi_tokens_to_place = [4, 6] + self.full_unpacked_stream = [torch.tensor([1, 2, 3, 4]), torch.tensor([5, 6, 7, 8, 9, 10])] + self.fill_value = 0 + + self.num_real_text_tokens = [[3, 2], [2, 4]] + # Here the input stream is padded to avoid inconsistencies (current model release matches) + self.input_stream = torch.tensor([[[1, 2, 3], [4, 5, 0]], [[6, 7, 0], [8, 9, 10]]]) + self.image_tokens = [ + [torch.tensor([1, 2]), torch.tensor([3])], + [torch.tensor([4, 5, 6]), torch.tensor([7, 8])], + ] + + def test_full_unpacked_stream_to_tensor(self): + result = full_unpacked_stream_to_tensor( + self.all_bi_tokens_to_place, + self.full_unpacked_stream, + self.fill_value, + self.batch_size, + self.new_seq_len, + offset=0, + ) + EXPECTED_TENSOR = torch.tensor([[1, 2, 3, 4, 0, 0, 0, 0], [5, 6, 7, 8, 9, 10, 0, 0]]) + self.assertTrue(torch.equal(result, EXPECTED_TENSOR)) + + def test_construct_full_unpacked_stream(self): + result = construct_full_unpacked_stream( + self.num_real_text_tokens, self.input_stream, self.image_tokens, self.batch_size, self.num_sub_sequences + ) + EXPECTED_UNPACKED_STREAM = [torch.tensor([1, 2, 1, 2, 3]), torch.tensor([4, 5, 6, 6, 7])] + for i in range(len(result)): + self.assertTrue(torch.equal(result[i], EXPECTED_UNPACKED_STREAM[i])) + + +@require_torch +class TestProcessImagesForModelInput(unittest.TestCase): + def setUp(self): + """ + Adding a mix of present and absent images. + """ + self.image_processor = FuyuImageProcessor() + + self.image_input = torch.randn([1, 1, 3, 64, 64]) + self.image_present = torch.tensor([[1]]) + self.image_unpadded_h = torch.tensor([[45]]) # Adjusted for subsequence of 1 + self.image_unpadded_w = torch.tensor([[50]]) # Adjusted for subsequence of 1 + self.image_patch_dim_h = 16 + self.image_patch_dim_w = 16 + self.image_placeholder_id = 999 + self.image_newline_id = 888 + self.variable_sized = True + + def test_process_images_for_model_input_fixed_sized(self): + self.variable_sized = False + result = self.image_processor.process_images_for_model_input( + image_input=self.image_input, + image_present=self.image_present, + image_unpadded_h=self.image_unpadded_h, + image_unpadded_w=self.image_unpadded_w, + image_patch_dim_h=self.image_patch_dim_h, + image_patch_dim_w=self.image_patch_dim_w, + image_placeholder_id=self.image_placeholder_id, + image_newline_id=self.image_newline_id, + variable_sized=self.variable_sized, + ) + print(result["images"][0][0]) + self.assertEqual(result["images"][0][0].shape, torch.Size([3, 64, 64])) diff --git a/utils/check_config_attributes.py b/utils/check_config_attributes.py index 0f0c5b41e4..b77028b294 100644 --- a/utils/check_config_attributes.py +++ b/utils/check_config_attributes.py @@ -36,6 +36,7 @@ SPECIAL_CASES_TO_ALLOW = { "EncodecConfig": ["overlap"], # used as `self.bert_model = BertModel(config, ...)` "DPRConfig": True, + "FuyuConfig": True, # not used in modeling files, but it's an important information "FSMTConfig": ["langs"], # used internally in the configuration class file diff --git a/utils/check_repo.py b/utils/check_repo.py index 85cf36eeac..1e5fc5ce21 100644 --- a/utils/check_repo.py +++ b/utils/check_repo.py @@ -79,6 +79,7 @@ PRIVATE_MODELS = [ # Being in this list is an exception and should **not** be the rule. IGNORE_NON_TESTED = PRIVATE_MODELS.copy() + [ # models to ignore for not tested + "FuyuForCausalLM", # Not tested fort now "InstructBlipQFormerModel", # Building part of bigger (tested) model. "UMT5EncoderModel", # Building part of bigger (tested) model. "Blip2QFormerModel", # Building part of bigger (tested) model. diff --git a/utils/not_doctested.txt b/utils/not_doctested.txt index 4ff70b0cbe..c2cedb66b2 100644 --- a/utils/not_doctested.txt +++ b/utils/not_doctested.txt @@ -566,6 +566,7 @@ src/transformers/models/funnel/configuration_funnel.py src/transformers/models/funnel/convert_funnel_original_tf_checkpoint_to_pytorch.py src/transformers/models/funnel/modeling_funnel.py src/transformers/models/funnel/modeling_tf_funnel.py +src/transformers/models/fuyu/convert_fuyu_model_weights_to_hf.py src/transformers/models/git/configuration_git.py src/transformers/models/git/convert_git_to_pytorch.py src/transformers/models/glpn/configuration_glpn.py