Chi-Square Difference Test

Reads two Mplus output files, checks the conditions that a nesting relation requires, and computes the difference test that the estimator of those two runs allows: the classical difference for ML, MLF, GLS and WLS, the Satorra–Bentler scaled difference for MLR, MLM and WLSM, and the DIFFTEST result for MLMV, WLSMV and ULSMV. Every number it uses is read from those files, and none of them can be typed in.

What to request from Mplus. Nothing beyond the standard output: the page reads MODEL FIT INFORMATION, available in the standard Mplus output. With MLMV, WLSMV or ULSMV add SAVEDATA: DIFFTEST = file.dat; to the unrestricted run and ANALYSIS: DIFFTEST = file.dat; to the restricted one, because for those estimators the difference of the two printed statistics is not chi-square distributed. A warning printed by Mplus is counted and reported next to the comparison, which is left in place; a run that did not converge blocks the test instead.

What to do
  1. Write two Mplus inputs, one for each of the two models you want to compare, and save each in its own .inp file. The two runs have to use the same data file, the same observed variables and the same estimator, and one of the two models has to be the other one with restrictions added to it.
  2. Add nothing to the OUTPUT command. The chi-square of a model is in the section MODEL FIT INFORMATION, which is part of the standard Mplus output.
  3. Only if the estimator of the two runs is MLMV, WLSMV or ULSMV: add the line SAVEDATA: DIFFTEST = deriv.dat; to the input of the unrestricted model (H1), add the line ANALYSIS: DIFFTEST = deriv.dat; to the input of the restricted model (H0), and run the unrestricted one first. For those three estimators the difference of the two printed statistics is not a chi-square, and only Mplus itself can produce the test.
  4. Run both inputs. Mplus writes for each one a plain text file with the same name and the extension .out, in the folder the input is in.
  5. Drag one .out file onto the first box below and the other onto the second, or press a box and choose the file there. Which file goes in which box makes no difference: the page reads the degrees of freedom of both and treats the model with more of them as the restricted one. The files are read inside your browser and semstudio.tools does not transmit the file contents; processing occurs in the browser.
  6. Read the list headed Compatible with a nesting relation. One line per condition says what was read from the two files: the estimator, the number of observations, the observed dependent variables, the degrees of freedom and the free parameters.
  7. Choose the significance level, and say which kind of parameter the restricted model holds fixed. Those two choices decide the level the p-value is compared with and which of the published cutoffs are marked as matching this comparison; they change nothing that was read from the files.
  8. Read the test in the table headed with the name of the test the estimator allows, and the sentence written under it for a manuscript. Every number in it comes from the two output files, and no field on this page can change one.
  9. Only if the page reports that the scaled test cannot be computed: run the unrestricted model once more with the estimates of the restricted model as starting values and no iterations, as described under Third output below, and drop that third output in the box at the end of the loading section.

Files are not compared when they are empty, not recognizable as Mplus output, truncated before the completed results, or produced with the abbreviated invariance syntax MODEL = CONFIGURAL METRIC SCALAR;, which contains several models. A comparison is also withheld when the estimator is unavailable or when the two files differ in estimator, sample size, or dependent variables. All numerical inputs are read from the files.

Load the two outputs

Drop one Mplus output file in each box, or press a box to choose a file.

Results
Waiting for two files Nothing has been loaded yet. Drop an Mplus output file on each of the two boxes above; the test appears here as soon as both have been read.

Methods and formulas

Nested models

A model M0 is nested in a model M1 when M0 follows from M1 by imposing restrictions on free parameters, such as fixing a parameter at a value or forcing two parameters to be equal, with the same data, the same observed variables and the same estimator. M0 therefore has fewer free parameters and more degrees of freedom than M1. Output files carry the necessary conditions, not the proof: they show whether the two runs share the sample, the dependent variables and the estimator, whether the difference in degrees of freedom matches the difference in free parameters, and whether the fit statistics fall in the expected order. Whether the restricted model is genuinely a restriction of the unrestricted one follows from the specification of the two models, which the page reports side by side.

Classical difference test: ML, MLF, GLS, WLS

These estimators produce a statistic that is asymptotically chi-square distributed, and the difference of two such statistics is again chi-square distributed:

T=χ02−χ12,Δdf=df0−df1

Here T is the difference statistic, χ²0 and df0 are the chi-square and the degrees of freedom of the restricted model, χ²1 and df1 are those of the unrestricted model, and Δdf is the number of degrees of freedom the test is referred to.

MLF differs from ML in the standard errors, which use the first-order derivative information matrix; the chi-square test of model fit it prints is the ML statistic, so the classical difference applies to MLF unchanged.

Scaled difference test: MLR, MLM, WLSM

These estimators print a statistic that has already been divided by a scaling correction factor c, so the scaled difference test of Satorra and Bentler (2001) multiplies the printed values back before differencing them:

cd=df0c0−df1c1df0−df1,T=χ02c0−χ12c1cd

Here cd is the difference scaling correction, df0 and df1 are the degrees of freedom of the restricted and of the unrestricted model, c0 and c1 are their scaling correction factors, χ²0 and χ²1 are the chi-square values printed for them, and T is the difference statistic.

T is referred to a chi-square distribution with df0−df1 degrees of freedom. The factors c0 and c1 are the values Mplus prints as Scaling Correction Factor under the chi-square of each model. Subtracting the two printed statistics without the correction does not give a chi-square statistic.

Log-likelihood version of the scaled test

When both outputs report an H0 log-likelihood and its scaling correction factor, the same test can be computed from the log-likelihoods:

cd=q0c0−q1c1q0−q1,T=−2(L0−L1)cd,Δdf=q1−q0

Here cd is the difference scaling correction, q0 and q1 are the numbers of free parameters of the restricted and of the unrestricted model, c0 and c1 are their H0 scaling correction factors, L0 and L1 are their H0 log-likelihoods, T is the difference statistic and Δdf is the number of degrees of freedom it is referred to.

When both formulations are available, both are reported. A material discrepancy is flagged because, for the same data and nesting relation, the two results are expected to agree within numerical and printing precision.

Strictly positive scaled test

The scaled difference test can produce a non-positive cd or a negative T, and neither outcome is a non-significant test: the test is not computable in that form. The correction cd of Satorra and Bentler (2001) is an approximation in which the correction factor of the unrestricted model is taken at the unrestricted estimates; Satorra and Bentler (2010) take it at the restricted estimates instead and obtain a correction that is positive by construction. It needs one additional Mplus run of the unrestricted model in which the restricted estimates supply the starting values and no iteration is taken, so that Mplus evaluates the unrestricted model at the restricted estimates and prints the scaling correction factor of that evaluation, written c10. Asparouhov and Muthén (2013) give the steps for Mplus, and Bryant and Satorra (2012) the general procedure. With that factor the correction becomes

cd′=df0c0−df1c10df0−df1,T=χ02c0−χ12c1cd′

Here cd′ is the strictly positive difference scaling correction, c10 is the scaling correction factor of the unrestricted model evaluated at the restricted estimates, df0, df1, c0, c1, χ²0 and χ²1 are the quantities of the previous formula, and T is the difference statistic. The formula is equation 4 of Asparouhov and Muthén (2013), written with the printed statistics multiplied back by their factors; c10 takes the place of c1, and df0c0 exceeds df1c10 in every sample, which is what keeps the correction positive.

Load that third output in the box at the end of the loading section and the page computes T with cd′ in place of cd. The box appears only when the estimator of the two runs is MLR, MLM or WLSM.

Mean- and variance-adjusted estimators: MLMV, WLSMV, ULSMV

These estimators adjust both the mean and the variance of the test statistic. The difference of two such statistics is not chi-square distributed, and the rescaling that makes it so (Asparouhov & Muthén, 2006, equations 7 and 10) needs the derivatives of both models, which the printed output does not contain. The test therefore has to be produced by Mplus itself with the DIFFTEST option, in two runs:

! Step 1: the unrestricted model (H1)
SAVEDATA:   DIFFTEST = deriv.dat;

! Step 2: the restricted model (H0), same data and variables
ANALYSIS:   DIFFTEST = deriv.dat;

The second output then contains a block headed Chi-Square Test for Difference Testing with a value, its degrees of freedom and a p-value. When the restricted model you load contains that block, the page reads it and reports it; when it does not, the page prints the procedure and computes nothing. Since Mplus 6 the degrees of freedom printed for MLMV, WLSMV and ULSMV are the usual ones, the number of moments minus the number of free parameters, and the degrees of freedom of the DIFFTEST result equal the difference in the number of free parameters (Asparouhov & Muthén, 2010). A DIFFTEST block whose degrees of freedom differ from the difference between the two printed values therefore comes from an older version or from a derivatives file saved by another model, and the page says so.

Files this page will not read

The parser does not use empty files, truncated runs, files that lack the usual Mplus output headings, or abbreviated invariance outputs containing several models. The message identifies the condition found. No difference test is computed from these files.

Estimators without a chi-square difference test

With ULS there is no chi-square test of model fit, so there is nothing to difference; the mean- and variance-adjusted ULSMV with DIFFTEST is the usual route to a test. BAYES estimates the model by MCMC, and fit is judged from the posterior predictive p-value: model comparison there rests on the posterior predictive p-value for absolute fit and on the DIC for relative fit.

Differences in fit indices

The chi-square difference test answers whether the restrictions are compatible with sampling error, and with a large sample a difference of no practical size can reach significance. For that reason the change in the approximate fit indices is usually reported next to the test. The page computes each difference as the value of the restricted model minus the value of the unrestricted model, for CFI, TLI, RMSEA, SRMR, AIC, BIC and the sample-size adjusted BIC. The cutoffs below for CFI, RMSEA and SRMR were all derived in simulations of multiple-group measurement invariance tests with normal continuous indicators; the page lists them for every nested comparison and marks a cutoff as matching the comparison only when the number of groups, the group sizes and the type of indicator agree with the design it was derived under (Svetina, Rutkowski, & Rutkowski, 2020, Table 1).

  • Cheung and Rensvold (2002): a change in CFI of at least −0.010, that is a drop no larger than 0.010, supports the restricted model. Two groups of 150 or 300 cases, two or three factors.
  • Chen (2007): for two groups of comparable size with more than 300 cases in each, a change in CFI of −0.010 together with a change in RMSEA of 0.015 and a change in SRMR of 0.030 when the constraints are on loadings, or 0.010 when they are on intercepts, thresholds or residual variances. For two unequal groups with 300 or fewer cases in each, the stricter values are −0.005 for CFI, 0.010 for RMSEA and 0.025 for SRMR with loadings or 0.005 with intercepts and residual variances. Chen simulated two groups of 150, 250 or 500 cases, and the sample-size condition is read here per group, following Svetina, Rutkowski and Rutkowski (2020, p. 114): the page compares the smallest group with 300, and reads two groups as of comparable size while the larger is less than one and a half times the smaller, a value this page adopts for reading rather than one Chen states. A comparison that falls between the two designs, two equal groups of 200 for instance, matches neither, and the table then lists both sets and marks neither as matching.
  • Rutkowski and Svetina (2014): 10 or 20 groups of 600 to 6,000 cases, one factor. A change in CFI of −0.020 with a change in RMSEA of 0.030 at the metric step, and −0.010 with 0.010 at the scalar step. The page lists them from five groups upward and marks them as matching from ten.
  • Raftery (1995), for BIC differences between two models fitted to the same data: 0 to 2 weak, 2 to 6 positive, 6 to 10 strong and above 10 very strong evidence for the model with the lower value.
  • Burnham and Anderson (2004), for AIC differences: a difference up to 2 leaves both models substantially supported, a difference of 4 to 7 gives the model with the higher AIC considerably less support, and a difference above 10 essentially none. They give no name to the intervals between these bands, and the page says so when a difference falls in one.

A change exactly equal to a cutoff is counted here as within it. These values come from simulation studies run under specific conditions of sample size, number of groups, number of indicators and type of misspecification, and they do not replace the evaluation of the model: a change within a cutoff is not evidence that the constraints hold, and a change beyond it is not by itself grounds for rejecting them. When the checks find something that a nesting relation excludes, the two sets of fit values are still shown but the cutoffs are not applied, because they were derived for nested comparisons. The information criteria are compared only when the two runs used the same number of observations, since they are on the scale of the log-likelihood of the sample.

Self-test

Append ?selftest=1 to run the in-browser checks. They cover chi-square tails, the classical, scaled, strictly positive and log-likelihood difference tests, fit-index changes and cutoff boundaries, parser behavior across estimators and file states, invalid numerical inputs, and common text encodings. The randomized checks include more than 1,000 generated comparisons.

References