Investigating the performance of level-specific fit indices in multilevel confirmatory factor analysis with dichotomous indicators: A Monte Carlo study

We conducted a Monte Carlo study to examine the performance of level-specific χ² test statistics and fit regarding their capacity to determine model fit at specific levels in multilevel confirmatory factor analysis with dichotomous indicators. Five design factors—numbers of groups (NG), group size (GS), intra-class correlation (ICC), thresholds of dichotomous indicators (THR), and factor loadings (FL)—were considered in this study. According to our simulation results, we recommend that practitioners should be aware that the performance of between-level-specific (b-l-s) χ² and fit indices was mainly influenced by ICC and FL, followed by NG. At the same time, THR could slightly weigh in the performance of b-l-s fit indices in some conditions. Both b-l-s χ² and fit indices were more promising indicators to correctly indicate model fit when ICC or FL increased. A small to medium NG (50–100) might be sufficient for b-l-s χ² and fit indices only if both ICC and factor loadings were high, while in remaining conditions, an NG of 200 was needed. Moreover, practitioners could use within-level-specific (w-l-s) χ² and fit indices (except for RMSEA_W) along with traditional cut-off values to evaluate within-level models comprising dichotomous indicators. W-l-s χ² and fit indices were more promising to determine model fit when FL increased. THR had a slight impact and could weigh in the performance of χW2, RMSEA_W, CFI_W, and TLI_W. Unfortunately, RMSEA_W was heavily affected by FL and THR and could determine model fit only when FL was high and THR was symmetric.