DINOv3 meets Sketches: A General Sketch-Photo Encoder

Sketches are considered a very relevant means of communication for humans. In fact, these abstract images are our first approach to real objects during our early childhood and allow us to unfold complex processes or transmit complex ideas, facilitating understanding. Therefore, it should also be considered an essential element in computer vision. Beyond sketch-based image retrieval, we can leverage sketches to guide other visual tasks, including segmentation or localization. To this end, it is critical to account for a highly effective sketch-based encoder trained in a bimodal fashion with sketches and photos. However, having sketch-photo pairs is unfeasible, so self or weakly supervised learning is required. In addition, given DINOv3's outstanding performance, it is crucial to leverage it to make it applicable to the sketch-based domain while retaining its learned knowledge. Therefore, in this work, we allow DINOv3 to meet sketches by proposing a new sketch-based encoder, skDINO, that demonstrates a general and outstanding effectiveness across different visual tasks. We evaluate our new encoder in zero-shot (ZS-SBIR) and zero-shot fine-grained retrieval (ZS-FG-SBIR). For ZS-SBIR, our proposal improves current models by 24.3%, 30.3%, and 15.7% in mAP@200 on Flickr15K, eCommerce, and Sketchy datasets, respectively. For ZS-FG-SBIR, our model reaches an improvement of 15.8% in accuracy in the Sketchy dataset. Furthermore, we leverage our encoder for sketch-based segmentation, where our model outperforms the state of the art, achieving a new mIoU of 0.7139, an increase of 18.8% over the best known model.

skDINO follows a Vision Transformer backbone, reusing the architecture of DINOv3 and extending it to handle both sketches and photos within a single encoder. Instead of learning two separate networks, we introduce learnable prompts that modulate the encoder for each modality: one set for sketches and another for photos. Most of the DINOv3 parameters remain frozen to preserve its general visual knowledge, while prompts, normalization layers, and a projection head are adapted to the sketch-photo task. We also explore a variant that adapts the attention blocks with LoRA instead (skDINO-LoRA).

Training uses a triplet loss based on cosine distance over sketch–photo pairs in a shared embedding space. Since collecting large amounts of real sketch–photo pairs is impractical, we first generate pseudo-sketches by applying a semantic contour detector (PiDiNet) to 75,701 photos from Flickr25K and eCommerce. These contours act as sketch surrogates and are paired with the corresponding photos to build positive and negative pairs; the model trained only this way is skDINO-SS (self-supervised). In a second stage, we follow a curriculum learning strategy: after 50 epochs on pseudo-sketches, we train 50 more epochs on 49,176 real sketch–photo pairs from Sketchy to better capture human abstraction, giving our best model, skDINO-WS (weakly supervised).

This training setup allows skDINO to align sketches and photos in a common latent space while keeping the pre-trained encoder stable. The resulting model can be used off-the-shelf for several tasks: global embeddings for retrieval, patch embeddings for correspondence and localization, and as a frozen backbone for sketch-guided segmentation with a lightweight decoder.

We freeze skDINO and attach a small mask-based decoder for sketch-guided segmentation. The sketch and the target photo are encoded by skDINO, and a bidirectional cross-attention module fuses their embeddings. The resulting image features are upsampled and combined with a sketch-derived mask embedding to predict segmentation masks and occurrence scores. Compared with SketchFusion, skDINO reaches an mIoU of 0.7139 and a pixel accuracy of 0.9396, improvements of 18.8% and 25.4%, respectively.