Computing reliability with multilevel data

What is it about?

Reporting of reliability information of measures on the studied sample is a basic requirement in psychological research. This paper identifies limitations of an existing method for computing reliability with multilevel data---data that have a hierarchical structure such as students nested within schools and survey participants nested within neighborhoods, and proposes alternative indices that address these limitations.

Featured Image

Why is it important?

This paper shows that the previously proposed between-level composite reliability can provide overly optimistic reliability coefficient because it ignores one major source of error, namely the sampling error of cluster means. To obtain more accurate reliability information for multilevel data, this article proposes alternative reliability indices that correctly account for the different sources of measurement error. The computation of the between-level indices also takes into account whether the construct being measured is an aggregate of individual characteristics (e.g., mean student achievement of a school) or an inherent group-level characteristic (e.g., school climate). I illustrate the proposed indices using a large-scale national data set with four items measuring students' attitudes toward mathematics. Software code in the R and the Mplus software was provided so that applied researchers can use to compute the proposed indices for their data and the corresponding confidence intervals.

Perspectives