NielsRogge 9c6f7485a6 Add GIT (GenerativeImage2Text) (#20295) ...

* First draft

* Make model instantiation work

* Fix copied from statement

* More fixes

* Add correct output head

* Improve configuration

* Add conversion script

* Improve conversion script

* Remove token_type_ids

* Fix conversion of projection layers

* Convert all weights

* Use cats image

* Make logits match

* Generate caption on cats image

* Add GITProcessor

* Update conversion script

* Add support for more checkpoints

* Fix conversion script

* Add initial tests

* Remove cross-attention

* More improvements

* Remove is_decoder

* Improve model tests

* Improve tests

* Improve model outputs

* Fix model outputs equivalence

* Fix more tests

* Remove unused code

* Use generate to generate text, no use of cache for now

* Use generate more appropriately

* Fix config tests

* Fix style

* Add support for use_cache

Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>

* Fix style

* Fix GIT vision encoder

* Update README

* Fix integration test

* Set bos and eos token ids

* Improve docs

* Improve code

* Add support for provided attention_mask

* Add copied from statement

* Fix gradient checkpointing test

* Set model_input_names

* Investigate model_input_names

* Remove script

* Fix model inputs

* Fix docstring

* Rename GIT to Git

* Support more models

* Add support for textvqa model

* Add video support

* Extend conversion script for video

* Add support for large variant

* Add support for more models

* Fix config archive map

* Update integration test

* Fix README

* Fix CLIP mean and std

* Update processor

* Fix use_cache for video, thanks @gante

* Remove print statements

* Remove assertion

* Add processor tests

* Fix model_input_names

* Use Auto API for processor

* Fix processor tests

* Fix integration test

* Fix pipeline test

* Make tests faster

* Update conversion script

* Update conversion script

* Convert more checkpoints

* Update conversion script

* Fix typo

* Update docstrings

* Improve code snippets

* Fix doc tests

* Add more code examplesé

* Fix doc tests

* Add integration tests

* Fix unused variable

* revert

* Add GIT to Japanese README

Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>
Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

2023-01-03 14:17:18 +01:00

..

internal

MinNewTokensLengthLogitsProcessor for .generate method #20814 (#20892)

2023-01-03 06:29:02 -05:00

main_classes

Generate: use GenerationConfig as the basis for .generate() parametrization (#20388)

2022-12-15 18:27:20 +00:00

model_doc

Add GIT (GenerativeImage2Text) (#20295)

2023-01-03 14:17:18 +01:00

tasks

Spanish translation of asr.mdx and add_new_pipeline.mdx (#20569)

2022-12-12 09:23:23 -05:00

_config.py

Enable doc in Spanish (#16518)

2022-04-04 10:25:46 -04:00

_toctree.yml

Add GIT (GenerativeImage2Text) (#20295)

2023-01-03 14:17:18 +01:00

accelerate.mdx

✨ update to use interlibrary links instead of Markdown (#18500)

2022-08-08 10:53:52 -05:00

add_new_model.mdx

fix docs typos in "add_new_model" (#20900)

2022-12-27 02:49:15 -05:00

add_new_pipeline.mdx

Spanish translation of asr.mdx and add_new_pipeline.mdx (#20569)

2022-12-12 09:23:23 -05:00

add_tensorflow_model.mdx

docs: Resolve many typos in the English docs (#20088)

2022-11-07 09:19:04 -05:00

autoclass_tutorial.mdx

Update doc examples feature extractor -> image processor (#20501)

2022-11-30 14:50:55 +00:00

benchmarks.mdx

Enable doc in Spanish (#16518)

2022-04-04 10:25:46 -04:00

bertology.mdx

Enable doc in Spanish (#16518)

2022-04-04 10:25:46 -04:00

big_models.mdx

docs: Resolve many typos in the English docs (#20088)

2022-11-07 09:19:04 -05:00

community.mdx

Enable doc in Spanish (#16518)

2022-04-04 10:25:46 -04:00

contributing.md

Enable doc in Spanish (#16518)

2022-04-04 10:25:46 -04:00

converting_tensorflow_models.mdx

Docs - Guide to add a new TensorFlow model (#19256)

2022-09-30 20:30:38 +01:00

create_a_model.mdx

Update doc examples feature extractor -> image processor (#20501)

2022-11-30 14:50:55 +00:00

custom_models.mdx

Replace awkward timm link with the expected one (#20109)

2022-11-07 13:57:39 -05:00

debugging.mdx

Spanish translation of the file debugging.mdx (#20566)

2022-12-12 10:38:56 -05:00

fast_tokenizers.mdx

Enable doc in Spanish (#16518)

2022-04-04 10:25:46 -04:00

glossary.mdx

Update doc examples feature extractor -> image processor (#20501)

2022-11-30 14:50:55 +00:00

hpo_train.mdx

update doc for perf_train_cpu_many (#19506)

2022-10-11 22:54:19 -04:00

index.mdx

Add GIT (GenerativeImage2Text) (#20295)

2023-01-03 14:17:18 +01:00

installation.mdx

Move cache folder to huggingface/hub for consistency with hf_hub (#18492)

2022-08-05 13:14:00 -04:00

migration.mdx

Enable doc in Spanish (#16518)

2022-04-04 10:25:46 -04:00

model_sharing.mdx

Just re-reading the whole doc every couple of months 😬 (#18489)

2022-08-06 09:38:55 +02:00

model_summary.mdx

Embed circle packing chart for model summary (#20791)

2022-12-20 10:26:52 -08:00

multilingual.mdx

Enable doc in Spanish (#16518)

2022-04-04 10:25:46 -04:00

notebooks.md

Enable doc in Spanish (#16518)

2022-04-04 10:25:46 -04:00

pad_truncation.mdx

Enable doc in Spanish (#16518)

2022-04-04 10:25:46 -04:00

perf_hardware.mdx

[WIP] [doc] performance/scalability revamp (#15723)

2022-05-16 13:36:41 +02:00

perf_infer_cpu.mdx

add doc for (#20525)

2022-12-01 16:52:13 +01:00

perf_infer_gpu_many.mdx

add doc for (#20525)

2022-12-01 16:52:13 +01:00

perf_infer_gpu_one.mdx

add doc for (#20525)

2022-12-01 16:52:13 +01:00

perf_infer_special.mdx

Improve performance docs (#17750)

2022-06-23 14:51:54 +02:00

perf_train_cpu_many.mdx

update cpu related doc (#20444)

2022-11-28 08:54:35 -05:00

perf_train_cpu.mdx

update cpu related doc (#20444)

2022-11-28 08:54:35 -05:00

perf_train_gpu_many.mdx

Fix Typo in Docs for GPU (#20509)

2022-11-30 10:41:18 -05:00

perf_train_gpu_one.mdx

Migrate torchdynamo to torch.compile (#20634)

2022-12-08 11:18:52 -05:00

perf_train_special.mdx

Fix Typo in Docs for GPU (#20509)

2022-11-30 10:41:18 -05:00

perf_train_tpu.mdx

Fix Typo in Docs for GPU (#20509)

2022-11-30 10:41:18 -05:00

performance.mdx

Fix Typo in Docs for GPU (#20509)

2022-11-30 10:41:18 -05:00

perplexity.mdx

Fix incorrect size of input for 1st strided window length in Perplexity of fixed-length models (#18906)

2022-09-06 15:20:12 -04:00

philosophy.mdx

Update doc examples feature extractor -> image processor (#20501)

2022-11-30 14:50:55 +00:00

pipeline_tutorial.mdx

Fixing the pipeline tutorial test (#20746)

2022-12-13 19:08:30 +01:00

pipeline_webserver.mdx

Rework the pipeline tutorial (#20437)

2022-12-06 10:47:31 +01:00

pr_checks.mdx

📝 update documentation build section (#18548)

2022-08-09 18:22:55 -05:00

preprocessing.mdx

Fix code sample in preprocess (#20561)

2022-12-05 11:49:43 -08:00

quicktour.mdx

typo fix (#20891)

2022-12-26 02:06:23 -05:00

run_scripts.mdx

Just re-reading the whole doc every couple of months 😬 (#18489)

2022-08-06 09:38:55 +02:00

sagemaker.mdx

Enable doc in Spanish (#16518)

2022-04-04 10:25:46 -04:00

serialization.mdx

Add Onnx Config for PoolFormer (#20868)

2022-12-23 01:30:57 -05:00

task_summary.mdx

Update doc examples feature extractor -> image processor (#20501)

2022-11-30 14:50:55 +00:00

testing.mdx

fixed spelling error in testing.mdx (#20220)

2022-11-15 09:40:06 -05:00

tokenizer_summary.mdx

Update tokenizer_summary.mdx (#20135)

2022-11-15 01:18:13 +01:00

torchscript.mdx

Breakup export guide (#19271)

2022-10-03 13:18:29 -07:00

training.mdx

Convert tokenizer outputs for Keras in doc example (#20732)

2022-12-12 16:14:04 +00:00

troubleshooting.mdx

Update doc examples feature extractor -> image processor (#20501)

2022-11-30 14:50:55 +00:00