Invariance Tables

1. Estimate each model of your invariance sequence in Mplus, in a separate input file with the extension .inp: the configural model, the metric model, the scalar model and whichever further models you fitted. All of them have to use the same data file, the same observed variables, the same grouping variable and the same estimator.
2. Add this line to every one of those inputs, exactly as it is written here: OUTPUT: TECH1 MODINDICES;
3. Run every input. Mplus writes one plain text file per input, with the same name and the extension .out, in the folder the input is in. One output per model.
4. Drag all of those .out files onto the box below at once, or press the box and select them all there. Drop them in any order you like: the page reads the constraints of each model and works out the order of the sequence itself. Files can also be added in more than one go; each new file joins the list. The files are read inside your browser and semstudio.tools does not transmit the file contents; processing occurs in the browser.
5. Read the list of files under the box. Each row names the file, the number of groups, the sample size, the estimator, the degrees of freedom, and the step of the ladder the constraints in TECH1 put that model on.
6. Read the ladder, then the fit table, then the table of comparisons between adjacent steps. Choose the significance level the p-values are compared with; that is the one choice the comparison section asks for, and it changes nothing that was read from the files.
7. To look at one item across the groups, or at one class of parameters in every item, choose the model, the item and the parameter under the two figures, then save the figure as PNG or as SVG at the width and the type size your journal asks for.

A model without TECHNICAL 1 is listed with its fit statistics but is not assigned to the invariance ladder or included in difference tests, because the equality constraints cannot be reconstructed from degrees of freedom or the file name. Difference tests are also withheld when the estimator is unavailable, estimation did not converge, or Mplus reported an error. Comparisons are restricted to adjacent ladder steps. Model step, estimator, fit statistics, and parameter estimates are read from the files.

The difference test answers whether the added constraints worsen fit beyond sampling error. With a large sample it reaches significance for a difference of no practical size, which is why the change in the approximate fit indices is reported next to it. The values below have been proposed as guidance, each under the conditions its authors studied. A change exactly equal to a cutoff counts here as within it.

• Cheung and Rensvold (2002): a change in CFI of at least −0.010, that is a drop no larger than 0.010, supports the constrained model. Their simulations used two groups and normal continuous indicators.
• Chen (2007): for two groups of comparable size with more than 300 cases in each, a change in CFI of −0.010 together with a change in RMSEA of 0.015 and a change in SRMR of 0.030 when the constraints are on loadings, or 0.010 when they are on intercepts, thresholds or residual variances. For two unequal groups with 300 or fewer cases in each, the stricter values are −0.005 for CFI, 0.010 for RMSEA and 0.025 for SRMR with loadings or 0.005 with intercepts and residual variances. Chen simulated two groups of 150, 250 or 500 cases, and the sample-size condition is read here per group, following Svetina, Rutkowski and Rutkowski (2020, p. 114), who state it as "larger than 300 in each group" on one side and "each sample size was smaller than 300" on the other: the page compares the smallest group with 300. Whether two groups count as of comparable size is decided here by the ratio of the larger to the smaller, with one and a half as the dividing value; that value is a reading convention of this page, since Chen contrasts equal groups with unequal ones without naming a ratio at which the one becomes the other. A sequence that falls between the two designs, two equal groups of 200 for instance, matches neither, and the comparison table then lists both sets and marks neither as matching.
• Rutkowski and Svetina (2014), for comparisons across many groups: a change in CFI of −0.020 with a change in RMSEA of 0.030 at the metric step, and −0.010 with 0.010 at the scalar step. Their design used 10 and 20 groups of 600 to 6,000 cases each, with one continuous factor, and the page marks their values as matching only a sequence with at least ten groups.
• Meade, Johnson and Braddy (2008) found ΔCFI to be affected little by sample size and by the number of indicators and factors, and proposed −0.002 as a criterion more sensitive than −0.010; it flags differences that the larger cutoffs let through. Their cutoffs for ΔMcDonald's NCI, which the page does not compute, do depend on the number of factors and indicators.
• Svetina, Rutkowski and Rutkowski (2020) set out the sequence for ordered categorical indicators, in which threshold invariance is tested first and the loadings are added to it at the next step, and collect the cutoffs proposed for that case. They state that the values are not interchangeable across designs and that the ones derived for continuous indicators should not be carried over to categorical ones without checking.

Change-in-fit cutoffs are simulation-based reference values, not universal thresholds. Their performance depends on sample size, group structure, indicator type, and the form and magnitude of non-invariance. Decisions should also consider the location and size of non-invariant parameters, their effect on the intended group comparisons, and the substantive measurement model. Values within a cutoff do not establish invariance, and values beyond a cutoff do not by themselves establish non-invariance (Putnick & Bornstein, 2016; Vandenberg & Lance, 2000; Millsap, 2011).

The ladder and where a model sits on it

Every model that the page reads is placed on exactly one step of a fixed ladder. With continuous indicators the steps, in order, are: configural, no parameter held equal across groups; metric, the loadings held equal; scalar, the loadings and the intercepts held equal; strict, also the residual variances; then two optional structural steps, the factor variances and covariances, and the factor means. With ordered categorical indicators the second step is threshold invariance and the third is thresholds and loadings, for the reason given below, and the page names the two steps accordingly. Nothing else is a step. A model that releases part of the constraints of its step, such as a scalar model with one intercept left free, sits on that same step and carries the flag partial. Partial invariance is a property of a model at a step, not a step of its own; treating it as a step is what lets two models be compared that stand in no nesting relation.

The step comes from the constraints Mplus actually imposed, read in TECH1. Mplus numbers the free parameters group by group and prints the numbers matrix by matrix; a parameter that carries the same number in every group is estimated once for all groups, which is what an equality constraint across groups means, and a parameter printed as 0 is not free. The page collects, for each class of parameters, how many cells are equal in every group, how many are free, and how many are equal in some groups but not all. A class is fully constrained when no cell of it is left loose, partial when some are, and free when none is constrained. The step is then the highest class that is constrained at all:

The class of parameters that decides the step, and the step it decides.

Factor means, factor variances and, for ordered categorical indicators, latent-response scale or residual parameters can serve identification rather than invariance. Under marker-variable identification factor variances are free across groups; under variance identification they are fixed at one. The page therefore reads none of them as an invariance constraint before the loadings and the intercepts or thresholds are themselves constrained. Reading them as constraints earlier would make the configural model look more constrained than the scalar one and the whole ladder would run backward.

No other evidence is used and none can be supplied. A file name, the title of the run and the number of degrees of freedom say nothing about which parameters were held equal across groups, and a model whose output has no TECH1 section is therefore listed without a step, with the reason, and takes part in no comparison. There is no control on this page that sets the step of a model.

Which models are compared

Difference tests are limited to adjacent ladder steps: full versus full, full versus partial at the next step, and full versus partial within the same step when the latter releases selected constraints. Nonadjacent steps are not compared because a degrees-of-freedom difference alone does not establish nesting.

Adjacency alone is not enough. Before any test the page checks the constraint sets themselves. Each constrained cell is written as a set of atoms: one atom for every pair of groups that share a parameter number, and one atom for every group in which the parameter is not free, for the classes where a fixed parameter is an invariance constraint rather than an identifying restriction. The comparison is produced only when the set of one model is a proper superset of the set of the other; the model with the larger set is the restricted one, the other is the unrestricted one, and the page names both together with the number of restrictions that separates them, class by class. When the sets are equal, or when neither contains the other, no test is produced and the page says which constraints are present in one model and missing in the other.

The count of restrictions is net of what the step gives back. A step of the sequence adds equality constraints and frees the parameters that identified the model below it: Mplus fixes the factor means at zero in every group while the intercepts or the thresholds are free and estimates k − 1 of them once those are held equal, and under variance identification it does the same with the factor variances at the metric step. The page counts the constraints added, subtracts the identifying parameters the restricted model frees, and reports both numbers. That net count is the difference in free parameters, so it is checked against the difference in degrees of freedom and a mismatch is reported.

For ordered categorical indicators, the ladder follows Wu and Estabrook (2016). A loadings-only equality model is equivalent to the baseline under their identification scheme and is therefore not treated as a separate testable step. Threshold invariance is tested first, followed by thresholds plus loadings. Threshold-only invariance is testable only when an item has at least four categories; with binary or three-category items, the thresholds can be matched by the identification transformation. The number of thresholds is read from TECH1. If no threshold-only model is supplied, the thresholds-plus-loadings model is compared directly with the configural model as defined by their Proposition 7.

Without TECH1, the constraint set cannot be reconstructed and the model is excluded from difference tests. A degrees-of-freedom difference is necessary but not sufficient for nesting; two models can differ in degrees of freedom without one being a restriction of the other.

Groups

Everything above is defined for k groups with k at least 2: a cell is constrained when it carries the same parameter number in every one of the k groups, and the count of restrictions on a cell is the number of parameters it removes, that is the number of free groups minus the number of distinct parameter numbers among them. A cell that is equal in some groups and free in others is reported separately and is not treated as a constraint of the class, because the class is then neither constrained nor free. The chi-square contribution of each group is read for all k groups, and the item plot draws one line per group.

The difference test, by estimator

The difference-test formula depends on the estimator. If the estimator cannot be read from the output, no test is computed. The estimator is not entered manually because an incorrect estimator would imply a different test.

ML, MLF, GLS and WLS produce a statistic that is asymptotically chi-square distributed, and the difference of two of them is again chi-square distributed:

Here T is the difference statistic, χ²₀ and df₀ are the chi-square and the degrees of freedom of the restricted model, χ²₁ and df₁ are those of the unrestricted model, and Δdf is the number of degrees of freedom the test is referred to.

MLR, MLM and WLSM print a statistic already divided by a scaling correction factor $c$, so the scaled difference test of Satorra and Bentler (2001) multiplies the printed values back before differencing them:

Here c_d is the difference scaling correction, df₀ and df₁ are the degrees of freedom of the restricted and of the unrestricted model, c₀ and c₁ are their scaling correction factors, χ²₀ and χ²₁ are the chi-square values printed for them, and T is the difference statistic.

T is referred to a chi-square distribution with ${df}_0-{df}_1$ degrees of freedom. When $c_d$ is zero or negative, or when T is negative, this formulation does not yield a valid chi-square difference statistic, so no p-value is reported. Satorra and Bentler (2010) give a version that is positive by construction. It needs one further Mplus run of the unrestricted model evaluated at the estimates of the restricted model, with those estimates as starting values and no iteration taken; its scaling correction factor $c_{10}$ replaces $c_1$, the factor of the unrestricted model, in the numerator of $c_d$, which gives $c_d^{\prime }=({df}_0{c}_0-{df}_1{c}_{10})/({df}_0-{df}_1)$. Asparouhov and Muthén (2013, equation 4) give the steps for Mplus: OUTPUT: SVALUES; in the restricted run, the MODEL command it prints pasted into the unrestricted input as starting values, and ANALYSIS: CONVERGENCE = 100000000; so that no iteration is taken. That run is a separate file and is not part of an invariance sequence, so the page reports the obstruction and points to the chi-square difference test page, which takes the third output.

MLMV, WLSMV and ULSMV adjust both the mean and the variance of the statistic, and the difference of two such statistics is not chi-square distributed. The rescaling that makes it so (Asparouhov & Muthén, 2006, equations 7 and 10) needs the derivatives of both models, which the printed output does not contain, so the test has to be produced by Mplus itself with the DIFFTEST option, in two runs:

! Step 1: the unrestricted model (H1)
SAVEDATA:   DIFFTEST = deriv.dat;

! Step 2: the restricted model (H0), same data and variables
ANALYSIS:   DIFFTEST = deriv.dat;

The second output then carries a block headed Chi-Square Test for Difference Testing. That block belongs to the one comparison whose derivatives file the run read, and since Mplus 6 its degrees of freedom are the difference in the number of free parameters (Asparouhov & Muthén, 2010). The page therefore reports it for the pair whose difference in degrees of freedom it matches, and for no other; when it is absent, or belongs to another pair, the page says so and prints the two runs to make. Neither ULS nor BAYES yields a chi-square test of model fit, so for those two estimators there is no difference test to compute.

The page recomputes every test from the fit statistics of the two models. When the sequence comes from the abbreviated syntax MODEL = CONFIGURAL METRIC SCALAR;, Mplus prints difference tests of its own inside that file; the page does not use them. It splits the file into its models, places each on the ladder from its own TECH1 and computes the tests itself, so that the pair being tested is stated and can be checked.

Differences in fit indices

Each difference is the value of the restricted model minus the value of the unrestricted model, for CFI, RMSEA and SRMR, so that a drop in CFI is negative and a rise in RMSEA is positive. The cutoffs proposed for those differences are collected in the guidance section above, with the conditions each was derived under; the comparison table marks a difference as within or beyond the criteria that match this comparison, by a label and a glyph and not by color alone.

Modification indices

The modification index of a constraint is the drop in the chi-square that releasing that one constraint would produce, tested on one degree of freedom, and the expected parameter change is the amount by which the parameter is expected to move once it is released, not the value it would take. The thresholds offered are the .05, .01 and .001 quantiles of a chi-square distribution with one degree of freedom, 3.841, 6.635 and 10.828. The indices are shown per model and per group and sorted by size. Under an equality constraint across groups the same index appears in every group, and with two groups the expected changes have opposite signs: it is one constraint, not k of them, and releasing it costs one degree of freedom in total. A high index says where to look, not what to conclude; constraints are released one at a time, the model is refitted, and the number of released constraints belongs in the report.

The two figures

The first figure draws one item across the groups. For a continuous indicator the expected value of item j in group g is a straight line in the factor:

Here y_j is item j, η is the factor it loads on, ν_jg is the intercept of that item in group g and λ_jg is its loading in that group.

Every line is drawn over one range, from the smallest of the group factor means minus 2.5 factor standard deviations of that group to the largest plus 2.5, so that the groups are read against the same axis. The band option adds ${\nu }_{jg}+{\lambda }_{jg}\eta \pm k\sqrt{{\theta }_{jg}}$ for k equal to 1 or 1.96, where θ_jg is the residual variance of the item in that group. The distribution option instead draws the normal density of the item at the factor mean of its own group, with mean ${\nu }_{jg}+{\lambda }_{jg}{\alpha }_g$ and variance θ_jg, where α_g is the factor mean in group g. For an ordered categorical indicator the vertical axis is the underlying response variable y*, ν is zero, the thresholds of the item are drawn as light reference lines, and the residual variance is not printed under the delta parameterization: it is recovered from the scale factor as

Here θ_jg is the residual variance of item j in group g on the metric of the underlying response variable, δ_jg is the scale factor Mplus prints for that item in that group (Muthén & Asparouhov, 2002), λ_jg is its loading and ψ_g is the variance of the factor in that group. The identity holds for an item that loads on one factor and on nothing else: with a cross-loading, or with a covariate in the model, the variance of the underlying response variable has further terms and the page does not recover θ from the scale factor.

The identity follows from $\mathrm{Var}(y^{\ast })=1/{{\delta }_{jg}}^2$ and $\mathrm{Var}(y^{\ast })={{\lambda }_{jg}}^2{\psi }_g+{\theta }_{jg}$. Under the theta parameterization the residual variance is printed and is used as it stands. Under the figure the page states, with the numbers, where the groups part: in slope, that is in the loading, in intercept or in threshold, and in residual standard deviation when the figure carries one. A difference smaller than .001 is not reported as a difference, and when nothing differs the page says that the lines coincide, since a figure of coincident lines shows only the one drawn last.

The second figure puts one class of parameters of one model on a single axis: the items on the vertical axis, and on the horizontal one the estimate in each group with the interval

Here b_g is the estimate of the parameter in group g as printed under MODEL RESULTS, SE(b_g) is the standard error printed beside it, and 1.959964 is the .975 quantile of the standard normal distribution.

That interval is the 95 percent Wald interval Mplus itself reports under CINTERVAL. Whether a parameter is held equal across groups is read from TECH1, by the rule used for the ladder: one number in every group. Such a row carries one estimate, drawn once as an open diamond and labeled as held equal, because k coincident points would read as k groups that happen to agree when they are one parameter. A parameter fixed by the model, the loading that sets the metric of the factor for instance, is drawn once as an open marker without an interval and labeled as fixed; a parameter fixed in some groups and free in the rest is labeled as fixed in some groups. When TECH1 is missing the page falls back to the same estimate and the same standard error in every group and says that it did so. Two intervals that do not overlap are a hint about where to look; the test of the constraint is the difference test between the two models.

The exported figures preserve the requested physical width and point-based type size. Axes use two spines, outward ticks and no grid; reference lines are lighter than data elements. Tick density and decimal precision are adjusted to avoid overlapping or duplicate labels. Groups differ by color, dash pattern and, when requested, marker shape to remain distinguishable in grayscale and under common color-vision deficiencies. Legends are placed outside the plotting area. SVG is exported as vector graphics; PNG is exported at the selected width and 300, 600 or 1200 dpi.

The chi-square tail

The p-value is the upper tail $\mathrm{P}(X\ge x)$ of a chi-square distribution, computed from the regularized incomplete gamma function by its series expansion below the crossover and by its continued fraction above it. The upper tail is computed directly rather than as $1-F(x)$, which loses every significant digit once p falls below about ${10}^{-16}$.

• Asparouhov, T., & Muthén, B. (2006, May 26). Robust chi square difference testing with mean and variance adjusted test statistics (Mplus Web Notes No. 10). Muthén & Muthén. https://www.statmodel.com/download/webnotes/webnote10.pdf
• Asparouhov, T., & Muthén, B. (2010, May 3). Simple second order chi-square correction. Muthén & Muthén. https://www.statmodel.com/download/WLSMV_new_chi21.pdf
• Asparouhov, T., & Muthén, B. (2013, April 17). Computing the strictly positive Satorra–Bentler chi-square test in Mplus (Mplus Web Notes No. 12). Muthén & Muthén. https://www.statmodel.com/examples/webnotes/SB5.pdf
• Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504. https://doi.org/10.1080/10705510701301834
• Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5
• Meade, A. W., Johnson, E. C., & Braddy, P. W. (2008). Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology, 93(3), 568–592. https://doi.org/10.1037/0021-9010.93.3.568
• Millsap, R. E. (2011). Statistical approaches to measurement invariance. Routledge. https://doi.org/10.4324/9780203821961
• Muthén, B., & Asparouhov, T. (2002, December 9). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus (Mplus Web Notes No. 4). Muthén & Muthén. https://www.statmodel.com/download/webnotes/CatMGLong.pdf
• Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004
• Rutkowski, L., & Svetina, D. (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74(1), 31–57. https://doi.org/10.1177/0013164413498257
• Satorra, A., & Bentler, P. M. (2001). A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66(4), 507–514. https://doi.org/10.1007/BF02296192
• Satorra, A., & Bentler, P. M. (2010). Ensuring positiveness of the scaled difference chi-square test statistic. Psychometrika, 75(2), 243–248. https://doi.org/10.1007/s11336-009-9135-y Corrigenda: Psychometrika, 90(1), 415. https://doi.org/10.1017/psy.2024.12
• Svetina, D., Rutkowski, L., & Rutkowski, D. (2020). Multiple-group invariance with categorical outcomes using updated guidelines: An illustration using Mplus and the lavaan/semTools packages. Structural Equation Modeling, 27(1), 111–130. https://doi.org/10.1080/10705511.2019.1602776
• Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70. https://doi.org/10.1177/109442810031002
• Wu, H., & Estabrook, R. (2016). Identification of confirmatory factor analysis models of different levels of invariance for ordered categorical outcomes. Psychometrika, 81(4), 1014–1045. https://doi.org/10.1007/s11336-016-9506-0