Should I Render or Should AI Generate? Crafting Synthetic Semantic Segmentation Datasets With Controlled Generation

Abstract

This work explores the integration of generative AI models for automatically generating synthetic image-labeled data. Our approach leverages controllable diffusion models to generate synthetic variations of semantically labeled images. Synthetic datasets for semantic segmentation struggle to represent real-world subtleties, such as different weather conditions or fine details, typically relying on costly simulations and rendering. However, diffusion models can generate diverse images using input text prompts and guidance images, such as semantic masks. Our work introduces and tests a novel methodology for generating labeled synthetic images, with an initial focus on semantic segmentation, a demanding computer vision task. We showcase our approach in two distinct image segmentation domains, outperforming traditional computer graphics simulations in efficiently creating diverse datasets and training downstream models. We leverage generative models for crafting synthetically labeled images, posing the question: “Should I render or should AI generate?” Our results endorse a paradigm shift toward controlled generation models.