Are Comparisons of Consumer Satisfaction With Providers Biased by Nonresponse or Case-Mix Differences?

Abstract

OBJECTIVE: This study examined how consumer satisfaction ratings differ between mental health care providers to determine whether comparison of ratings is biased by differences in survey response rates or consumer characteristics.METHODS: Consumer satisfaction surveys mailed by a mixed-model prepaid health plan were examined. Survey data were linked to computerized records regarding consumers' demographic (age, sex, and type of insurance coverage) and clinical (primary diagnosis and initial versus return visit) characteristics. Statistical models examined probabilities of returning the survey (N=8,025 returned surveys) and of giving an excellent satisfaction rating. Variability was separated into within-provider effects and between-provider effects.RESULTS: The overall response rate was 33.8%, and 49.9% of responders reported excellent satisfaction. Neither response rate nor satisfaction rating was related to primary diagnosis. Within the practices of individual providers, response rate and receiving an excellent rating were significantly associated with female sex, older age, longer enrollment in the health plan, and making a return visit. Analyses of between-provider effects, however, found that only a higher proportion of return visitors was significantly associated with higher response rates and higher satisfaction ratings.CONCLUSIONS: There was little evidence that differences in response rate or in consumers served biased comparison of satisfaction ratings between mental health providers. Bias might be greater in a setting with more heterogeneous consumers or providers. Returning consumers gave higher ratings than first-time visitors, and analyses of satisfaction ratings may need to account for this difference. Extremely high or low ratings should be interpreted cautiously, especially for providers with a small number of surveys.