CMU researchers provide a content-based search engine for Modelverse, a model-sharing platform that contains a variety of deep generative models.

The intent of a content-based form search is introduced, which attempts to identify the most relevant forms to generate deep images that satisfy the user’s input query. As shown in the graphic below, a user can get a model based on their ability to group images that fit an image query (for example, a landscape shot), a text query (for example, African animals), a graphic query (for example, a graphic standing cat), or an analogy with the provided query form. But why does a typical search depend on useful content? Deep generative models are developed as a basis for content creation software and applications. They are no longer just the results of scientific studies.

The search method (first row) allows searching in four distinct ways – text, images, graphics and existing forms – from left to right. The top two models are shown in the second and third rows. The color of the icon for each model indicates the type of model. In all approaches, the technology discovers applicable models with comparable semantic concepts. | Source:

Each model represents a microcosm of carefully selected subjects, which may include realistic depictions of people and landscapes, images of ancient pottery, cartoonish animation, and the aesthetics of a single artist. Recently, various technologies have made it possible to creatively alter and customize existing models, whether through human-in-the-loop interfaces or tuning GANs and text-to-image models. Each generative model can refer to the significant involvement of the model creator in a particular concept. It is becoming increasingly impossible for the user to be familiar with every great generative model, even if it is necessary to choose the best one for their own needs.

The ability to rapidly synthesize an unlimited set of images, interpolations, or inherent variable manipulations is provided by each generative model. However, the researchers discovered that choosing the ideal model from a wide range can lead to much better results than those obtained from choosing an inappropriate model. Form search allows users to locate the form that most closely matches their unique requirements, as much as information and image retrieval allows users to find appropriate information within large collections of traditional materials. The challenge of researching a content-based model is difficult; Even direct inquiry whether a single model can generate a given image can be computationally demanding.

Unfortunately, many deep generative models do not provide an efficient or accurate method for density estimation and do not locally enable multimodal similarity (eg, text and image) to be measured. A naive Monte Carlo technique can compare an input query to tens or millions of samples from each generative model and repeatedly determine which model pieces match the input query. The search for the model will be prolonged using such a sampling-based technique. They first provide a general probability formula for the model search problem to address the above problems, followed by a Monte Carlo baseline. To save time and space, they “compress” the distribution of the model into pre-computed first and second-order moments from weddings with deep features of the original samples.

Then, they create closed-form solutions to retrieve the form using an input image, text, graphic, or form query. Calculations can be made in real time from the final version. In 133 deep generative models, including GANs (eg, StyleGAN family models), diffusion models (eg, DDPM), and auto-regression models, they evaluate their methods and perform ablation tests (eg, VQGAN). Their approach provides significantly faster search (within 0.08 ms, 5x acceleration) while maintaining good accuracy compared to the Monte-Carlo baseline. Finally, they demonstrate how to use GAN inversion and few-shot model tuning as typical search applications.

Their solution, to the best of their knowledge, is the first content-based search algorithm for machine learning models. The research method is published in Modelverse, an online platform for academics, students, and artists to efficiently use and share generative models, which can be found at

This Article is written as a research summary article by Marktechpost Staff based on the research paper 'Content-Based Search for Deep Generative Models'. All Credit For This Research Goes To Researchers on This Project. Check out the paper, github link, Modelverse and project.

Please Don't Forget To Join Our ML Subreddit and Youtube Channel

Content writing consultant trainee at Marktechpost.

Leave a Comment