And now for something completely different: the congruence of the Altmetric Attention Score’s structure between different article groups

Abstract

Altmetric Attention Score (AAS) is an increasingly popular composite altmetric measure, which is being criticized for an inappropriate and arbitrary aggregation of different altmetric sources into a single measure. We examined this issue empirically, by testing unidimensionality and the component structure congruence of the five ‘key’ AAS components: News, Blogs, Twitter, Facebook, and Google+. As a reference point, these tests were also done on different citation data: WoS, Scopus, and Google Scholar. All tests were done for groups of articles with: (1) high citations, but lower AAS (HCGs), and (2) high AAS, but lower citations (HAGs). Changes in component structures over time (from 2016 to 2017) were also considered. Citation data consistently formed congruent unidimensional structures for all groups and over time. Altmetric data formed congruent unidimensional structures only for the HCGs, with much inconsistency for the HAGs (including change over time). The relationship between Twitter and News counts was shown to be curvilinear. It was not possible to obtain a satisfactory congruent and reliable linear unidimensional altmetric structure between the groups for any variable combination, even after Mendeley and CiteULike altmetric counts were included. Correlations of altmetric aggregates and citations were fairly inconsistent between the groups. We advise against the usage of composite altmetric measures (including the AAS) for any group comparison purposes, until the measurement invariance issues are dealt with. The underlying pattern of associations between individual altmetrics is likely too complex and inconsistent across conditions to justify them being simply aggregated into a single score.