Text-Guided Texturing by Synchronized Multi-View Diffusion

Abstract

This paper introduces a novel approach to synthesize texture to dress up a 3D object, given a text prompt. Based on the pre-trained text-to-image (T2I) diffusion model, existing methods usually employ a project-and-inpaint approach, in which a view of the given object is first generated and warped to another view for inpainting. But it tends to generate inconsistent texture due to the asynchronous diffusion of multiple views. We believe that such asynchronous diffusion and insufficient information sharing among views are the root causes of the inconsistent artifacts. In this paper, we propose a synchronized multi-view diffusion approach that allows the diffusion processes from different views to reach a consensus on the generated content early in the process, and hence ensures the texture consistency. To synchronize the diffusion, we share the denoised content among different views in each denoising step, specifically by blending the latent content in the texture domain from overlapping views. Our method demonstrates superior performance in generating consistent, seamless and highly detailed textures, comparing to state-of-the-art methods.