AutoGEEval++: A multi-level and multi-geospatial-modality automated evaluation framework for large language models in geospatial code generation on Google Earth Engine

Abstract

Geospatial code generation is crucial in integrating AI with geo-scientific analysis, but standardized evaluation tools are lacking. This study presents AutoGEEval++, an enhanced framework for evaluating large language models (LLMs) that generate geospatial code on the Google Earth Engine (GEE) platform. Built on the GEE Python API, AutoGEEval++ includes a benchmark dataset—AutoGEEval++-Bench—comprising 6,365 test cases across 26 GEE data types and three task categories: unit test, combination test, and theme test. The framework offers a fully automated evaluation pipeline, from code generation to execution-based validation, using multi-dimensional metrics such as accuracy, resource consumption, runtime efficiency, and error types. It also supports boundary testing and error pattern analysis. We assess 24 leading LLMs (as of June 2025) spanning general-purpose, reasoning-enhanced, code-centric, and geoscience-specific models. Experimental results highlight distinct performance, stability, and error patterns, demonstrating the framework’s scalability for vertical-domain code generation. This study establishes the first standardized evaluation protocol and resource suite for GEE-based LLM code generation, providing a unified benchmark and a methodology for evaluating the transition from natural language to domain-specific code, advancing geospatial AI research.