Future Landscape Visualization by Generating Images Using a Diffusion Model and Instance Segmentation

Abstract

When designing a new landscape, such as when demolishing buildings and building new ones, visual methods are effective in sharing a common image. It is possible to visualize future landscapes by making sketches and models, but this requires a great deal of skill and effort on the part of the creator. One method for visualizing future landscapes without the need for specialized skills or labor is image generation using deep learning, and a method has been proposed of using deep learning to generate landscape images after demolishing current buildings. However, there are two problems: the inability to remove arbitrary buildings and the inability to generate a landscape after reconstruction. Therefore, this study proposes a future landscape visualization method that integrates instance segmentation and a diffusion model. The proposed method can generate both post-removal images of existing buildings and post-reconstruction images based on text input, without the need for specialized technology or labor. Verification results confirmed that the post-removal image was more than 90% accurate when the building was removed and replaced with the sky. And the post-reconstruction image matched the text content with a best accuracy of more than 90%. This research will contribute to the realization of urban planning in which all project stakeholders, both professionals and the public, can directly participate by visualizing their own design proposals for future landscapes.