Comparing advice on climate policy between academic experts and ChatGPT

Abstract

We compare the results from a recent global expert survey on climate policy with answers to the same survey by the online artificial-intelligence chatbot ChatGPT. Such a study is timely and relevant as many people around the world are likely to use ChatGPT and similar language models to inquire about climate solutions, which in turn might influence public opinion. The comparison provides insights about performance criteria, policy instruments, and use of information from distinct academic disciplines. With a few exceptions, responses by ChatGPT are informative and of high quality. We find that ChatGPT answers questions with less bias than experts from various scientific disciplines. The latter may also be a disadvantage as it seems to weight all the information available equally without accounting well for relevance, which arguably may require human rather than artificial intelligence. On the other hand, experts from distinct disciplines show difference in average responses, with some even expressing opinions inconsistent with objective evidence, meaning there is no consistent and unbiased expert opinion on climate policy. As a new way of synthesizing large amounts of academic and grey literature, ChatGPT can serve policymaking. However, since the procedure that it follows for collecting and summarizing information remains a black box, it is best regarded as a complement rather than a substitute to traditional literature reviews and expert surveys.