This project focuses on developing self-supervised models that exploit the multimodal nature of e-commerce to improve search engines, recommendation systems and customer segmentation. The multimodal nature of e-commerce is represented not only by catalogue products but also by user queries. We can even find multimodal information related to products or trends in social media posts. From an architectural point of view, the proposed models will evaluate the use of self-attention and cross-attention modules to improve multimodal data representations.

The sketch and photo encoders from this line of work can be tried in the Search Motor demo.