Power Analysis · semstudio.tools

Power of the test of model fit

The test of model fit refers the statistic to a chi-square distribution with the degrees of freedom of the model. Under a false model the statistic follows a noncentral chi-square distribution whose noncentrality parameter is $λ = (N - 1) F_{0}$ , and the power of the test is the area of that distribution beyond the critical value.

What to compute A compromise analysis fixes the ratio between the two error probabilities instead of fixing the first at .05, which is what makes a test of fit interpretable when the sample is very large or very small (Moshagen & Erdfelder, 2016).

Effect size given as Every one of them is a function of F₀, and the page reports the conversion it used.

Its value

Degrees of freedom

Observed variables Read only when the effect size is GFI or AGFI.

Observations

α, test of fit

Power wanted

β/α Read only in a compromise analysis.

Close fit and not-close fit

The same machinery tests a hypothesis about the size of the misfit rather than its absence: the null hypothesis fixes a value of the RMSEA and the alternative another one, and the power is computed between the two noncentral distributions (MacCallum, Browne, & Sugawara, 1996).

RMSEA under the null

RMSEA under the alternative

Degrees of freedom

Observations

Power of a difference test between nested models

Two nested models differ by a set of restrictions. The power of the test of those restrictions is computed from the difference in the population discrepancy and the difference in the degrees of freedom, referred to a noncentral chi-square with that difference in degrees of freedom (Satorra & Saris, 1985).

F₀ of the restricted model

F₀ of the unrestricted model

Difference in degrees of freedom

Observations

α, difference test

When the two discrepancies are known through the RMSEA of each model, F₀ is ${df}_{} \times {RMSEA}^{2}$ with the degrees of freedom of that model, and the page below reports the conversion.

Design analysis of every parameter of a fitted model

Fit the model in Mplus and keep the .out file. Any standard output is enough: the page reads the estimates, their standard errors and the number of observations.
Drop the file on the box below.
Set the number of observations the design analysis is about, and how large the true effect is taken to be as a fraction of the estimate in this file. Half is a common choice when the file comes from a study whose estimate may itself be inflated.
Read the table: for each parameter, the power of its test, the probability that a significant estimate carries the wrong sign, and the factor by which a significant estimate would overstate the effect.

Drop the Mplus .out file here, or press Enter to choose one.

The file is read in your browser and is not sent anywhere.

Observations of the design

True effect, as a fraction of the estimate

α, design analysis

Power by simulation

A model whose power cannot be reduced to a single discrepancy — one with missing data, with categorical indicators, with a moderation, with clustered observations — is studied by simulation: the population values are set, many samples are drawn from them, the model is fitted to each, and the proportion of samples in which a parameter is significant is its power (Muthén & Muthén, 2002). Mplus does this with the MONTECARLO command, and this page reads what it prints.

Drop the .out file of a MONTECARLO run here, or press Enter to choose one. Several runs of the same model at different sample sizes can be dropped together.

The page reports, for every parameter, the population value, the average estimate and its bias, the bias of the standard error, the coverage of the 95 percent interval and the proportion of replications in which the parameter was significant, which is its power. It also reports how many replications were completed. With more than one run loaded it sets them side by side by sample size and says which is the smallest that meets the criteria of Muthén and Muthén (2002).

Methods and formulas

The noncentrality parameter

Let F₀ be the value of the maximum likelihood discrepancy function at the population covariance matrix, that is the misfit that remains when sampling error is removed. The test statistic is $T = (N - 1) \hat{F}$ , and under a model that is false in the population it follows a noncentral chi-square distribution with the degrees of freedom of the model and noncentrality parameter

λ = (N - 1) F_{0}

Here N is the number of observations, F₀ the population discrepancy and λ the noncentrality parameter. Power is the probability that T exceeds the critical value of the central chi-square distribution with the same degrees of freedom, computed under that noncentral distribution (MacCallum, Browne, & Sugawara, 1996).

Effect sizes and their conversion

Every effect size the page accepts is a function of F₀ and the degrees of freedom or the number of observed variables:

RMSEA = \sqrt{\frac{F_{0}}{df}}, Mc = e^{- 0.5 F_{0}}, GFI = \frac{p}{p + 2 F_{0}}

Here df is the number of degrees of freedom of the model and p the number of observed variables. The relation for the RMSEA is that of Steiger and Lind (1980) as set out by Browne and Cudeck (1992), the one for Mc is McDonald's (1989), and those for GFI and AGFI are the ones Jöreskog and Sörbom (1984) define; the adjusted index inverts to $F_{0} = \frac{p (1 - AGFI) df}{p (p + 1) - 2 df (1 - AGFI)}$ . The page prints the value of F₀ it worked with, whichever index was typed.

The three analyses

A post hoc analysis fixes α, the effect and the sample size and returns the power. An a priori analysis fixes α, the effect and the power and returns the sample size, found by searching over N until the power reaches the value asked for. A compromise analysis fixes the effect, the sample size and the ratio between the two error probabilities, and returns the critical value at which that ratio holds, with the α and the β that go with it. The last one answers the objection that a fixed α of .05 makes the test of fit reject every model in a large sample and accept every model in a small one: with the sample at hand it reports the decision rule under which the two mistakes are as likely as the researcher is willing to let them be (Moshagen & Erdfelder, 2016).

Close fit and not-close fit

MacCallum, Browne and Sugawara (1996) test a hypothesis about the size of the misfit. The null hypothesis states a value ε₀ of the RMSEA and the alternative a value ε_a; the critical value is taken from the noncentral chi-square with $λ_{0} = (N - 1) df ε_{0}^{2}$ and the power computed under the one with $λ_{a} = (N - 1) df ε_{a}^{2}$ . With ε₀ below ε_a the test is the test of close fit, and the power is the probability of rejecting a model whose misfit is ε_a; with ε₀ above ε_a it is the test of not-close fit, in which rejection supports the model.

Difference test between nested models

Satorra and Saris (1985) obtain the power of the likelihood ratio test of a set of restrictions from the difference between the population discrepancies of the two models: $λ = (N - 1) (F_{0 r} - F_{0 u})$ , referred to a chi-square distribution with the difference in degrees of freedom. The discrepancy of the unrestricted model is not required to be zero: the test concerns the loss the restrictions cause.

Design analysis

Power is the probability of rejecting a specified false null hypothesis. It does not describe the magnitude or direction of estimates conditional on significance. Gelman and Carlin (2014) distinguish two additional quantities: Type S error, the probability that a significant estimate has the wrong sign, and Type M error, the expected exaggeration of its magnitude.

power = 1 - Φ (z - \frac{D}{s}) + Φ (- z - \frac{D}{s}), S = \frac{Φ (- z - \frac{D}{s})}{power}

Here Φ is the standard normal distribution function, z the critical value at the chosen α, D the true effect the design analysis is about, s the standard error of its estimator, and S the probability that an estimate which reaches significance has the sign opposite to the true effect. The exaggeration ratio is the expected size of a significant estimate divided by the true effect, $M = E (| \hat{D} | ∣ significant) / | D |$ . Gelman and Carlin obtain it by simulation; this page computes the same expectation in closed form from the two tail integrals of the normal density, which removes the simulation error.

The standard error at the sample size of the design is obtained from the one in the file by the usual rate, $s_{new} = s_{file} \sqrt{\frac{N_{file}}{N_{new}}}$ , which holds for a maximum likelihood estimator of a correctly specified model and is stated on the page beside the table, because it is an approximation and not a property of the file.

Taking the estimate in the file as the true effect turns the design analysis into a post hoc power calculation, which is a function of the p value and adds nothing to it (Hoenig & Heisey, 2001). The fraction below the file is there for that reason: a design analysis is worth doing against an effect that comes from somewhere else, a meta-analysis, a smallest effect of interest, or the estimate of this study reduced by the amount its own significance is likely to have inflated it.

Simulation

For Monte Carlo studies, the page reports the criteria used by Muthén and Muthén (2002): parameter-estimate bias, standard-error bias, 95% coverage and empirical power. Their reference values are reported as source-specific guidelines rather than universal thresholds. The number of completed replications is shown because nonconvergence changes the set over which empirical performance is summarized.

What the page does not do

The analytic fit-power calculations are limited to a single group. Multiple-group power depends on how discrepancy and sample size are distributed across groups (Moshagen & Bader, 2024), and clustered or intensive-longitudinal designs additionally depend on cluster or occasion structure. These designs are therefore directed to simulation rather than reduced to one total sample size.

Self-test

Append ?selftest=1 to check chi-square and noncentral chi-square functions, effect-size conversions, a priori, post hoc and compromise analyses, close-fit tests, and Type S/Type M calculations.

References

Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21(2), 230–258. https://doi.org/10.1177/0049124192021002005
Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641–651. https://doi.org/10.1177/1745691614551642
Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1), 19–24. https://doi.org/10.1198/000313001300339897
Jöreskog, K. G., & Sörbom, D. (1984). LISREL VI: Analysis of linear structural relationships by maximum likelihood, instrumental variables, and least squares methods. Scientific Software.
MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130–149. https://doi.org/10.1037/1082-989X.1.2.130
McDonald, R. P. (1989). An index of goodness-of-fit based on noncentrality. Journal of Classification, 6(1), 97–103. https://doi.org/10.1007/BF01908590
Moshagen, M., & Bader, M. (2024). semPower: General power analysis for structural equation models. Behavior Research Methods, 56(4), 2901–2922. https://doi.org/10.3758/s13428-023-02254-7
Moshagen, M., & Erdfelder, E. (2016). A new strategy for testing structural equation models. Structural Equation Modeling, 23(1), 54–60. https://doi.org/10.1080/10705511.2014.950896
Muthén, L. K., & Muthén, B. O. (2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling, 9(4), 599–620. https://doi.org/10.1207/S15328007SEM0904_8
Satorra, A., & Saris, W. E. (1985). Power of the likelihood ratio test in covariance structure analysis. Psychometrika, 50(1), 83–90. https://doi.org/10.1007/BF02294150
Steiger, J. H., & Lind, J. C. (1980, May). Statistically based tests for the number of common factors [Conference presentation]. Annual meeting of the Psychometric Society, Iowa City, IA.