OWLv2 is an open-vocabulary object detector from Google (Minderer et al., Scaling Open-Vocabulary Object Detection, NeurIPS 2023). It is the second version of OWL-ViT (Vision Transformer for Open-World Localization). You give it an image and a list of text queries such as a red cube, and it returns a bounding box and a score for each match.
Closed-set vs open-vocabulary detection
| Type | Classes | Example | New object at run time |
|---|---|---|---|
| Closed-set | fixed when the model is trained | YOLO trained on the 80 COCO classes | needs new labels and retraining |
| Open-vocabulary | given as free text at inference | OWL-ViT, OWLv2 | type a new query |
The open-vocabulary detector does not learn a fixed output layer with one neuron per class. Instead, it compares image features with text embeddings, so any phrase that the text encoder can embed becomes a possible class.
Architecture
OWLv2 starts from a CLIP-style model, which has an image encoder and a text encoder that were trained together on image-text pairs. Detection is added with small heads on top of the image encoder.
The steps for one image:
- the image is padded to a square, resized, and split into patches (16 by 16 pixels for the B/16 model)
- a Vision Transformer (ViT) encodes the patches into output tokens
- CLIP normally pools all tokens into one image embedding, but OWLv2 removes this pooling and keeps every token, so each token is a candidate object
- each token goes to 3 heads: a box head, a class head, and an objectness head
- each text query goes through the text encoder and gives one query embedding
- the class embedding of each token is compared with each query embedding to give a score
| Head | Output per token | Purpose |
|---|---|---|
| Box head | 4 numbers | where the object is |
| Class head | 1 class embedding | what the object is, by comparison with the queries |
| Objectness head | 1 score | how likely the token holds any object, with no query needed |
For the B/16 model at input, the number of tokens, and so the number of candidate boxes, is:
Scoring a token against a query
Let be the normalised class embedding of token and be the normalised embedding of query . The model also predicts a shift and a scale for each token. The logit and the probability are:
- the dot product measures how close the token and the query are in the shared embedding space
- a sigmoid is used, not a softmax, because one box can match several queries or none of them
- a box is kept when is above a threshold that you choose
Image-conditioned detection
The query does not have to be text. You can give an example image of the object, embed it with the same model, and use that embedding as . This is called one-shot or image-conditioned detection, and it is useful when an object is hard to describe in words.
What changed from OWL-ViT to OWLv2
The main change is the training data. Human-labelled detection datasets are small, so OWLv2 uses self-training (the paper calls the recipe OWL-ST) to make far more training boxes:
- Annotate: an existing OWL-ViT L/14 model labels the WebLI dataset (about 10 billion image-text pairs) with pseudo-boxes, using N-grams from each image caption as the queries
- Self-train: a new detector trains on these pseudo-boxes
- Fine-tune (optional): the detector trains on human-labelled data such as LVIS or Objects365
Training on billions of images is expensive, so the paper adds 3 changes to make it faster:
- token dropping: about half of the image patches, the ones with the lowest pixel variance (for example plain background), are removed before the encoder
- instance selection: the objectness head ranks the tokens, and only the top tokens go to the class head and the loss
- mosaics: several images are tiled into one training image, so each step sees more objects at smaller scales
The result on LVIS rare classes (classes with very few labelled examples) went from 31.2% to 44.6% AP for the L/14 model.
Code
import torch
from PIL import Image
from transformers import Owlv2Processor, Owlv2ForObjectDetection
checkpoint = "google/owlv2-base-patch16-ensemble"
processor = Owlv2Processor.from_pretrained(checkpoint)
model = Owlv2ForObjectDetection.from_pretrained(checkpoint)
image = Image.open("scene.jpg").convert("RGB")
text_labels = [["a red cube", "a blue bowl", "a robot gripper"]]
inputs = processor(text=text_labels, images=image, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
# OWLv2 pads the image to a square, so use the padded size here
side = max(image.size)
target_sizes = torch.tensor([[side, side]])
results = processor.post_process_grounded_object_detection(
outputs,
threshold=0.2,
target_sizes=target_sizes,
text_labels=text_labels,
)
result = results[0]
for box, score, label in zip(result["boxes"], result["scores"], result["text_labels"]):
x0, y0, x1, y1 = [round(v, 1) for v in box.tolist()]
print(f"{label}: {score:.2f} at ({x0}, {y0}, {x1}, {y1})")| Item | Meaning |
|---|---|
owlv2-base-patch16-ensemble | the B/16 model, fine-tuned on detection data, with an ensemble of the self-trained weights |
text_labels | a nested list, with one inner list of queries for each image in the batch |
outputs.logits | shape (batch, 3600, num_queries), one logit per token and query |
outputs.pred_boxes | shape (batch, 3600, 4), boxes in normalised form |
threshold | the minimum to keep a box |
target_sizes | the size used to scale boxes back to pixels |
- the padding is added at the bottom and right, so boxes scaled to the padded square line up with the pixels of the original image
- OWLv2 scores are often lower than OWL-ViT scores, so a threshold near 0.1 to 0.3 is a common start
- older versions of
transformersusepost_process_object_detectioninstead, which returns label indices and not label text
Role in a modular pipeline
OWLv2 gives the pipeline an explicit object location, which the robot can act on with classical tools:
LLM names the object --> OWLv2 box --> box centre pixel + depth --> 3D point --> pick primitive
This is the opposite design to OpenVLA, which never outputs a bounding box and maps the image straight to action tokens. With OWLv2, you can inspect the box when a grasp fails and see if the detector or the motion stage caused the error.