This was announced by Nvidia researchers on Friday Magic3D, an AI model that can generate 3D models from text descriptions. After entering a prompt like “A blue poison dart frog is sitting on a water lily”, Magic3D generates a 3D mesh model with colored texture in about 40 minutes. With modifications, the resulting model can be used in video games or CGI art scenes.
In his academic paperNvidia Frames Magic3D in response to DreamFusion, a text-to-3D model that Google researchers announced in September. Similar to how DreamFusion uses a text-to-image model to generate a 2D image, which is then optimized into a volumetric image NeRF (neural radiation field) data, Magic3D uses a two-step process that takes a rough model generated at low resolution and optimizes it for higher resolution. According to the paper’s authors, the resulting Magic3D method can generate 3D objects two times faster than DreamFusion.
Magic3D can also perform command prompt based editing of 3D meshes. Given a low-resolution 3D model and a basic prompt, it is possible to change the text to change the resulting model. Also, the authors of Magic3D demonstrate the retention of the same motif across generations (a concept often referred to as coherence) and the application of the style of a 2D image (e.g. a Cubist painting) to a 3D model.
Nvidia has not released any Magic3D code along with its scientific work.
The ability to generate 3D from text feels like a natural progression in today’s diffusion models, which use neural networks to synthesize novel content after intensive training with a set of data. In 2022 alone, we have seen the emergence of powerful text-to-image models like DALL-E and Stable Diffusion, as well as rudimentary text-to-video generators from Google and Meta. Google also unveiled the aforementioned DreamFusion text-to-3D model two months ago, and people have been doing it ever since similar techniques adapted to work as an open source model based on stable diffusion.
As for Magic3D, the researchers behind it hope that it will allow anyone to create 3D models without special training. Once refined, the resulting technology could accelerate video game (and VR) development, and perhaps eventually find applications in film and television special effects. Towards the end of their article, they write: “We hope that with Magic3D we can democratize 3D synthesis and open up everyone’s creativity in creating 3D content.”
This article was previously published on Source link