Multilevel Reliability · semstudio.tools

What to do

Fit the two-level factor model in Mplus with STDYX in the OUTPUT command and keep the .out file.
Drop it on the box below.
Say what the construct is taken to be. The choice is not a detail of estimation: a construct that exists within clusters, one that is shared by the members of a cluster and one that exists at both levels are three different things, and each has its own coefficient.
Read the coefficients with the note beside each one, which says which score it is the reliability of.

Drop the Mplus .out file of the two-level model here, or press Enter to choose one.

The file is read in your browser and is not sent anywhere.

What the construct is Within-cluster: it belongs to the individual, and the clustering is a nuisance. The reliability wanted is that of the individual score, purged of the cluster level. Shared: it belongs to the cluster and the individuals are its raters, as with a classroom climate rated by its pupils. The reliability wanted is that of the cluster mean. Configural: it exists at both levels with the same meaning, and both the individual score and the cluster mean are of interest.

What the file gives

The tables appear here once an output file has been read.

Level-specific fit

A two-level model is tested by one chi-square that combines misfit from both levels. Because the within level is based on many more effective observations, it can dominate the statistic, allowing substantial between-level misfit to go unnoticed. Ryu and West (2009) separate the two by fitting the model with one level saturated at a time. This page writes the four inputs that method needs and assembles the table from the four outputs.

Press Write the four inputs. The page writes them from the model of the file already loaded, or from the description you type below when no file is loaded.
Run all four in Mplus.
Drop the four .out files on the box below, in any order. The page recognizes each of them by its model.

Factors and their items One factor to a line, written as in Mplus. Used only when no file has been loaded above; with a file, the model in the file is used.

Cluster variable

Data file

Drop the four .out files here, or press Enter to choose them.

Each file is recognized by the title the page writes into it. The table appears as soon as the pair needed for a level is present.

Methods and formulas

What is nested in what

The variance of an item observed on individuals nested in clusters splits into a part that varies between clusters and a part that varies within them. The intraclass correlation is the share of the first,

ICC = \frac{σ_{b}^{2}}{σ_{b}^{2} + σ_{w}^{2}}, Deff = 1 + (\overset{―}{n} - 1) ICC

Here σ²_b and σ²_w are the between-cluster and within-cluster variances of the item, n̄ the average cluster size and Deff the design effect, the factor by which the clustering inflates the sampling variance of a mean compared with a simple random sample of the same size. Mplus prints the intraclass correlations of a two-level analysis; the page reads them when they are there and computes them from the variance components when they are not.

Reliability at each level

Geldhof, Preacher and Zyphur (2014) apply the ordinary coefficients to the parameters of each level of the model:

ω^{w} = \frac{{(\sum_{i} λ_{i}^{w})}^{2} ψ^{w}}{{(\sum_{i} λ_{i}^{w})}^{2} ψ^{w} + \sum_{i} θ_{i}^{w}}

Here λ_wi, ψ_w and θ_wi are the within-level loading, factor variance and residual variance; the between-level coefficient uses the corresponding between-level parameters. Maximal reliability H is computed from level-specific standardized loadings. Lai (2021) notes that between-level H can be upwardly biased when between-level reliability and ICCs are low and cluster sizes are small, so the page reports it with that limitation.

Which coefficient goes with which construct

Lai (2021) shows that the coefficients above are the reliability of no score anyone computes, and derives the reliability of the scores that are actually used, one for each meaning the construct can have. The formulas below are the ones his supplementary syntax computes.

For a within-cluster construct, the target score varies across individuals within clusters. Its reliability is estimated from the within-level measurement model while the between level is saturated, reducing dependence on a potentially misspecified between-level structure.

For a configural construct the raw score of an individual carries both levels, and its reliability is

ω^{2 L} = \frac{{(\sum_{i} λ_{i})}^{2} (1 + ψ^{b})}{{(\sum_{i} λ_{i})}^{2} (1 + ψ^{b}) + \sum_{i} θ_{i}^{b} + \sum_{i} θ_{i}^{w}}

Here λ_i is the loading of item i, held equal across the two levels as a configural construct requires, ψ^b is the between-level factor variance with the within-level one fixed at one, and θ^b_i and θ^w_i are the residual variances at the two levels.

The reliability of the cluster mean of that same score carries the sampling error of a mean over a finite number of members:

ω^{b} = \frac{{(\sum_{i} λ_{i})}^{2} ψ^{b}}{{(\sum_{i} λ_{i})}^{2} ψ^{b} + \sum_{i} θ_{i}^{b} + \frac{\sum_{i} θ_{i}^{w} + {(\sum_{i} λ_{i})}^{2}}{\overset{―}{n}}}

Here n̄ is the harmonic mean of the cluster sizes, which is the value Lai uses in his syntax and the one this page computes from the cluster sizes Mplus prints. The term divided by it is the variance of the cluster mean of the within-level part of the composite: with large clusters it vanishes and the coefficient approaches the one computed from the between-level parameters alone, and with small clusters it dominates.

For a shared construct there is no within-level factor, since what is measured belongs to the cluster and the differences among its members are disagreement. The within level is then left saturated and its whole composite variance enters as error:

ω^{b} = \frac{{(\sum_{i} λ_{i}^{b})}^{2} ψ^{b}}{{(\sum_{i} λ_{i}^{b})}^{2} ψ^{b} + \sum_{i} θ_{i}^{b} + \frac{1^{'} Σ^{w} 1}{\overset{―}{n}}}

Here Σ^w is the within-level covariance matrix of the items, left unstructured, and 1′Σ^w1 the sum of all its elements, that is the within-level variance of the unit-weighted total. The page computes it from the within-level variances and covariances the file prints, and says so when the within level of the model is not saturated, since the quantity is then the one the structure implies rather than the one the data show.

Level-specific fit

Ryu and West (2009) obtain the fit of one level by saturating the other. Fitting the model with the within level saturated gives a chi-square that reflects the between-level model alone, and the reverse gives one that reflects the within-level model. Their fit indices use those statistics with a baseline that is itself partially saturated, so that the saturated half of the model does not inflate the index:

{CFI}_{PS - B} = 1 - \frac{\max (χ_{PS - B}^{2} - {df}_{PS - B}, 0)}{\max (χ_{I - B}^{2} - {df}_{I - B}, 0)}

Here χ²_PS−B and df_PS−B are those of the model with the between level hypothesized and the within level saturated, and χ²_I−B and df_I−B those of the model with the between level an independence model and the within level saturated. The within-level index is the mirror image of this one.

{RMSEA}_{PS - B} = \sqrt{\frac{χ_{PS - B}^{2} - {df}_{PS - B}}{{df}_{PS - B} J}}, {RMSEA}_{PS - W} = \sqrt{\frac{χ_{PS - W}^{2} - {df}_{PS - W}}{{df}_{PS - W} (N - J)}}

Here J is the number of clusters and N the total number of observations: the between-level index is divided by J and the within-level index by N − J, the effective sample-size terms in the level-specific decomposition of Ryu and West (2009). Both are set to zero when the quantity under the root is negative.

Self-test

Append ?selftest=1 to check two-level parsing, cluster summaries, ICCs, harmonic mean, level-specific reliability coefficients, fit indices, and unsupported inputs.

References

Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246. https://doi.org/10.1037/0033-2909.107.2.238
Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138
Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable systems. In R. Cudeck, S. du Toit, & D. Sörbom (Eds.), Structural equation modeling: Present and future (pp. 195–216). Scientific Software International.
Lai, M. H. C. (2021). Composite reliability of multilevel data: It's about observed scores and construct meanings. Psychological Methods, 26(1), 90–102. https://doi.org/10.1037/met0000287
Muthén, B. O., & Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316. https://doi.org/10.2307/271070
Ryu, E., & West, S. G. (2009). Level-specific evaluation of model fit in multilevel structural equation modeling. Structural Equation Modeling, 16(4), 583–601. https://doi.org/10.1080/10705510903203466