Examining the Impacts of Airbnb Review Policy Change on Listing Reviews

Abstract

In July 2014, Airbnb, one of the biggest firms in the sharing economy, decided to change the way that guests and hosts reviewed each other on the platform. Prior to this change, guests/hosts could post reviews about their experiences asynchronously—the guest/host would be able to see the other party’s review whenever it was posted. In contrast, the new review policy rolled out a simultaneous review system, making reviews viewable only after both the guest/host post their own reviews. This study empirically evaluates the impacts of this new review policy on the informativeness of guest reviews, measured by both informational content (semantic diversity and objectivity) and personal opinions (sentiment and sentiment heterogeneity). Using regression discontinuity design and a variety of techniques in the text analytics domain including a novel adaptation of BERT, we demonstrate that Airbnb’s review policy change enhanced the informational content of guest reviews in terms of semantic diversity and objectivity. We also show that review sentiment was reduced but became more diverse. Subgroup analysis revealed that low-quality listings were subject to more changes than high quality listings. We further explore the short-term and long-term effects of the review policy change and demonstrate that the simultaneous review system has had a long-lasting impact on the informativeness of guest reviews.