HuggingFace_transformer/docs/source/en/model_doc/superglue.md

SuperGlue

SuperGlue is a neural network that matches two sets of local features by jointly finding correspondences and rejecting non-matchable points. Assignments are estimated by solving a differentiable optimal transport problem, whose costs are predicted by a graph neural network. SuperGlue introduces a flexible context aggregation mechanism based on attention, enabling it to reason about the underlying 3D scene and feature assignments jointly. Paired with the SuperPoint model, it can be used to match two images and estimate the pose between them. This model is useful for tasks such as image matching, homography estimation, etc.

You can find all the original SuperGlue checkpoints under the Magic Leap Community organization.

Tip

This model was contributed by stevenbucaille.

Click on the SuperGlue models in the right sidebar for more examples of how to apply SuperGlue to different computer vision tasks.

The example below demonstrates how to match keypoints between two images with the [AutoModel] class.

from transformers import AutoImageProcessor, AutoModel
import torch
from PIL import Image
import requests

url_image1 = "https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/refs/heads/master/assets/phototourism_sample_images/united_states_capitol_98169888_3347710852.jpg"
image1 = Image.open(requests.get(url_image1, stream=True).raw)
url_image2 = "https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/refs/heads/master/assets/phototourism_sample_images/united_states_capitol_26757027_6717084061.jpg"
image2 = Image.open(requests.get(url_image2, stream=True).raw)

images = [image1, image2]

processor = AutoImageProcessor.from_pretrained("magic-leap-community/superglue_outdoor")
model = AutoModel.from_pretrained("magic-leap-community/superglue_outdoor")

inputs = processor(images, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

# Post-process to get keypoints and matches
image_sizes = [[(image.height, image.width) for image in images]]
processed_outputs = processor.post_process_keypoint_matching(outputs, image_sizes, threshold=0.2)

Notes

• SuperGlue performs feature matching between two images simultaneously, requiring pairs of images as input.

  from transformers import AutoImageProcessor, AutoModel
  import torch
  from PIL import Image
  import requests
  
  processor = AutoImageProcessor.from_pretrained("magic-leap-community/superglue_outdoor")
  model = AutoModel.from_pretrained("magic-leap-community/superglue_outdoor")
  
  # SuperGlue requires pairs of images
  images = [image1, image2]
  inputs = processor(images, return_tensors="pt")
  outputs = model(**inputs)
  
  # Extract matching information
  keypoints0 = outputs.keypoints0  # Keypoints in first image
  keypoints1 = outputs.keypoints1  # Keypoints in second image
  matches = outputs.matches        # Matching indices
  matching_scores = outputs.matching_scores  # Confidence scores
  

• The model outputs matching indices, keypoints, and confidence scores for each match.

• For better visualization and analysis, use the [SuperGlueImageProcessor.post_process_keypoint_matching] method to get matches in a more readable format.

  # Process outputs for visualization
  image_sizes = [[(image.height, image.width) for image in images]]
  processed_outputs = processor.post_process_keypoint_matching(outputs, image_sizes, threshold=0.2)
  
  for i, output in enumerate(processed_outputs):
      print(f"For the image pair {i}")
      for keypoint0, keypoint1, matching_score in zip(
              output["keypoints0"], output["keypoints1"], output["matching_scores"]
      ):
          print(f"Keypoint at {keypoint0.numpy()} matches with keypoint at {keypoint1.numpy()} with score {matching_score}")
  

• The example below demonstrates how to visualize matches between two images.

  import matplotlib.pyplot as plt
  import numpy as np
  
  # Create side by side image
  merged_image = np.zeros((max(image1.height, image2.height), image1.width + image2.width, 3))
  merged_image[: image1.height, : image1.width] = np.array(image1) / 255.0
  merged_image[: image2.height, image1.width :] = np.array(image2) / 255.0
  plt.imshow(merged_image)
  plt.axis("off")
  
  # Retrieve the keypoints and matches
  output = processed_outputs[0]
  keypoints0 = output["keypoints0"]
  keypoints1 = output["keypoints1"]
  matching_scores = output["matching_scores"]
  
  # Plot the matches
  for keypoint0, keypoint1, matching_score in zip(keypoints0, keypoints1, matching_scores):
      plt.plot(
          [keypoint0[0], keypoint1[0] + image1.width],
          [keypoint0[1], keypoint1[1]],
          color=plt.get_cmap("RdYlGn")(matching_score.item()),
          alpha=0.9,
          linewidth=0.5,
      )
      plt.scatter(keypoint0[0], keypoint0[1], c="black", s=2)
      plt.scatter(keypoint1[0] + image1.width, keypoint1[1], c="black", s=2)
  
  plt.savefig("matched_image.png", dpi=300, bbox_inches='tight')
  

Resources

• Refer to the original SuperGlue repository for more examples and implementation details.

SuperGlueConfig

autodoc SuperGlueConfig

SuperGlueImageProcessor

autodoc SuperGlueImageProcessor

• preprocess
• post_process_keypoint_matching

## SuperGlueForKeypointMatching

autodoc SuperGlueForKeypointMatching

• forward