From cd5f7c17908a66d71c9ca6871c7479f9f1a75f39 Mon Sep 17 00:00:00 2001 From: Billy Cao Date: Wed, 19 Jun 2024 18:11:44 +0800 Subject: [PATCH] Add docs on zeroshot image classification prompt templates (#31343) * Add docs on pipeline templates * Fix example and comments Update usage tips * Update docs/source/en/tasks/zero_shot_image_classification.md Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update docs/source/en/model_doc/siglip.md Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Trigger CI --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> --- docs/source/en/model_doc/siglip.md | 8 ++++++-- docs/source/en/tasks/zero_shot_image_classification.md | 2 ++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/source/en/model_doc/siglip.md b/docs/source/en/model_doc/siglip.md index c6db0441e7..6ee237942a 100644 --- a/docs/source/en/model_doc/siglip.md +++ b/docs/source/en/model_doc/siglip.md @@ -29,6 +29,7 @@ The abstract from the paper is the following: - Usage of SigLIP is similar to [CLIP](clip). The main difference is the training loss, which does not require a global view of all the pairwise similarities of images and texts within a batch. One needs to apply the sigmoid activation function to the logits, rather than the softmax. - Training is not yet supported. If you want to fine-tune SigLIP or train from scratch, refer to the loss function from [OpenCLIP](https://github.com/mlfoundations/open_clip/blob/73ad04ae7fb93ede1c02dc9040a828634cb1edf1/src/open_clip/loss.py#L307), which leverages various `torch.distributed` utilities. - When using the standalone [`SiglipTokenizer`] or [`SiglipProcessor`], make sure to pass `padding="max_length"` as that's how the model was trained. +- To get the same results as the pipeline, a prompt template of "This is a photo of {label}." should be used. drawing @@ -59,7 +60,8 @@ The pipeline allows to use the model in a few lines of code: >>> image = Image.open(requests.get(url, stream=True).raw) >>> # inference ->>> outputs = image_classifier(image, candidate_labels=["2 cats", "a plane", "a remote"]) +>>> candidate_labels = ["2 cats", "a plane", "a remote"] +>>> outputs = image_classifier(image, candidate_labels=candidate_labels) >>> outputs = [{"score": round(output["score"], 4), "label": output["label"] } for output in outputs] >>> print(outputs) [{'score': 0.1979, 'label': '2 cats'}, {'score': 0.0, 'label': 'a remote'}, {'score': 0.0, 'label': 'a plane'}] @@ -81,7 +83,9 @@ If you want to do the pre- and postprocessing yourself, here's how to do that: >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) ->>> texts = ["a photo of 2 cats", "a photo of 2 dogs"] +>>> candidate_labels = ["2 cats", "2 dogs"] +# follows the pipeline prompt template to get same results +>>> candidate_labels = [f'This is a photo of {label}.' for label in candidate_labels] >>> # important: we pass `padding=max_length` since the model was trained with this >>> inputs = processor(text=texts, images=image, padding="max_length", return_tensors="pt") diff --git a/docs/source/en/tasks/zero_shot_image_classification.md b/docs/source/en/tasks/zero_shot_image_classification.md index 9f6e49a4bb..d923ca44b4 100644 --- a/docs/source/en/tasks/zero_shot_image_classification.md +++ b/docs/source/en/tasks/zero_shot_image_classification.md @@ -119,6 +119,8 @@ image for the model by resizing and normalizing it, and a tokenizer that takes c ```py >>> candidate_labels = ["tree", "car", "bike", "cat"] +# follows the pipeline prompt template to get same results +>>> candidate_labels = [f'This is a photo of {label}.' for label in candidate_labels] >>> inputs = processor(images=image, text=candidate_labels, return_tensors="pt", padding=True) ```