The influence of violations of assumptions on multilevel parameter estimates and their standard errors
Introduction
Social research often involves problems that investigate the relationship between individual and society. The general concept is that individuals interact with their social contexts, meaning that individual persons are influenced by the social groups or contexts, and that the properties of those groups are in turn influenced by the individuals who make up that group. Generally, the individuals and the social groups are conceptualized as a hierarchical system, with individuals and groups defined at separate levels of this hierarchical system.
Standard multivariate models are not appropriate for the analysis of such hierarchical systems, even if the analysis includes only variables at the lowest (individual) level, because the standard assumption of independent and identically distributed observations is generally not valid. The consequences of using uni-level analysis methods on multilevel data are well known: the parameter estimates are unbiased but inefficient, and the standard errors are negatively biased, which results in spuriously ‘significant’ effects (cf. de Leeuw and Kreft, 1986; Snijders and Bosker, 1999; Hox 1998, Hox 2002). Multilevel analysis techniques have been developed for the linear regression model (Bryk and Raudenbush, 1992; Goldstein, 1995), and specialized software is now widely available (Raudenbush et al., 2000; Rasbash et al., 2000).
The assumptions underlying the multilevel regression model are similar to the assumptions in ordinary multiple regression analysis: linear relationships, homoscedasticity, and normal distribution of the residuals. In ordinary multiple regression, it is known that moderate violations of these assumptions do not lead to highly inaccurate parameter estimates or standard errors. Thus, provided that the sample size is not too small, standard multiple regression analysis can be regarded as a robust analysis method (cf. Tabachnick and Fidell, 1996). In the case of severe violations, a variety of statistical methods for correcting heteroscedasticity are available (Scott Long and Ervin, 2000). Multilevel regression analysis has the advantage that heteroscedasticity can also be modeled directly (cf. Goldstein, 1995, pp. 48–57).
The maximum likelihood estimation methods used commonly in multilevel analysis are asymptotic, which translates to the assumption that the sample size is large. This raises questions about the accuracy of the various estimation methods with relatively small sample sizes. This concerns especially the higher level(s), because the sample size at the highest level (the sample of groups) is always smaller than the sample size at the lowest level. A large simulation by Maas and Hox (2003) finds that the standard errors for the regression coefficients are slightly biased downwards if the number of groups is less than 50. With 30 groups, they report an operative alpha level of 6.4% while the nominal significance level is 5%. Similarly, simulations by Van der Leeden and Busing (1994) and Van der Leeden et al. (1997) suggest that when assumptions of normality and large samples are not met, the standard errors have a small downward bias.
Sometimes it is possible to obtain more nearly normal distributions by transforming the outcome variable. If this is undesirable or even impossible, another method to obtain better tests and confidence intervals is to correct the asymptotic standard errors. One correction method to produce robust standard errors is the so-called Huber/White or sandwich estimator (Huber, 1967; White, 1982), which is available in several of the available multilevel analysis programs (e.g., Raudenbush et al., 2000; Rasbash et al., 2000).
In this paper we look more precisely at the consequences of the violation of the assumption of normally distributed errors at the second level of the multilevel regression model. Specifically, we use simulation to answer the following two questions: (1) what group level sample size can be considered adequate for reliable assessment of sampling variability when the assumption of normally distributed residuals is not met, and (2) how well do the asymptotic and the sandwich estimators perform when the assumption of normally distributed residuals is not met.
Section snippets
The multilevel regression model
Assume that we have data from J groups, with a different number of respondents nj in each group. On the respondent level, we have the outcome variable Yij. We have one explanatory variable Xij on the respondent level, and one group level explanatory variable Zj. To model these data, we have a separate regression model in each group as follows:The variation of the regression coefficients βj is modeled by a group level regression model, as follows:and
Maximum likelihood estimation
The usual estimation method for the multilevel regression model is maximum likelihood (ML) estimation (cf. Eliason, 1993). One important assumption underlying this estimation method is normality of the error distributions. When the residual errors are not normally distributed, the parameter estimates produced by the ML method are still consistent and asymptotically unbiased. However, the asymptotic standard errors are incorrect. Significance tests and confidence intervals can thus not be
The simulation model and procedure
We use a simple two-level model, with one explanatory variable at the individual level and one explanatory variable at the group level, conforming to Eq. (4), which is repeated hereFour conditions are varied in the simulation: (1) Number of groups (NG: three conditions, NG=30, 50 and 100), (2) group size (GS: three conditions, GS=5, 30 and 50), (3) intraclass Correlation (ICC: three conditions, ICC=0.1, 0.2 and 0.3; note that the ICC varies with the X
Convergence and inadmissible solutions
The estimation procedure converged in all 3×27,000=81,000 simulated data sets. The estimation procedure in MLwiN can and sometimes does lead to negative variance estimates. Such solutions are inadmissible, and common procedure is to constrain such estimates to the boundary value of zero. However, all simulated data sets produced only admissible solutions.
Percentage relative bias
For across all 27 conditions the mean relative bias is calculated. Tested is whether this relative bias differs from one, with an α of 0.001.
Summary and discussion
Non-normal distributed residual errors on the second (group) level of a multilevel regression model appear to have little or no effect on the estimates of the fixed effects. The estimates of the regression coefficients are unbiased, and both the ML and the robust standard errors are accurate. There is no advantage here in using robust standard errors. This corresponds to the general belief that ML estimation methods are generally robust (cf. Eliason, 1993).
Non-normal distributed residual errors
Uncited reference
Bryk et al., 1996.
References (31)
- Afshartous, D., 1995. Determination of sample size for multilevel model design. Unpublished paper. Annual Meeting of...
- Browne, W.J., 1998. Applying MCMC methods to multilevel models. Unpublished Ph.D. Thesis, University of Bath, Bath,...
- et al.
Implementation and performance issues in the Bayesian and likelihood fitting of multilevel models
Comput. Statist.
(2000) - et al.
Hierarchical Linear Models
(1992) - et al.
HLM. Hierarchical Linear and Nonlinear Modeling with the HLM/2L and HLM/3L programs
(1996) - Busing, F., 1993. Distribution characteristics of variance estimates in two-level models. Unpublished manuscript....
- Carpenter, J., Goldstein, H., Rasbash, J., 1999. A non-parametric bootstrap for multilevel models. Multilevel Modelling...
Statistical Power Analysis for the Behavioral Sciences
(1988)Maximum Likelihood Estimation
(1993)- et al.
Statistical Distributions
(1993)
Components of variance and intraclass correlations for the design of community-based surveys and intervention studies
Amer. J. Epidemiol.
Multilevel modeling: when and why
Multilevel Analysis, Techniques and Applications
The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples
Struct. Equation Modeling
Cited by (277)
Exploring multilevel data with deep learning and XAI: The effect of personal-care advertising spending on subjective happiness
2024, International Business ReviewThe Effects of a Science and Social Studies Content Rich Shared Reading Intervention on the Vocabulary Learning of Preschool Dual Language Learners
2024, Early Childhood Research QuarterlyEvaluating the precision and reproducibility of non-invasive deformation measurements in an arterial phantom
2023, Measurement: Journal of the International Measurement ConfederationCross-cultural adaptation and validation of the Spanish version of the Anesthetists' Non-Technical Skills (ANTS) assessment tool
2023, Journal of Clinical Anesthesia