William R. Shadish
 

Annotated Bibliography of Methods for Meta-Analysis of Single Subject Designs

Allison, D. B., & Groman B. S. (1994). “Make things as simple as possible, but no

simpler.” A rejoinder to Scruggs and Mastropieri. Behavioral Research Therapy.

32, 885 – 890.

            This article is a response to researchers Scruggs and Mastropieri’s issues with criticisms made by the authors in a previous publication. Scruggs and Mastropieri had argued for their methods of single subject meta-analysis Percent of Nonoverlapping Data (PND) by stating that their method is good for small samples, easy to compute, goes beyond the assumptions of normality, and it correlates with visual judgments made by experts. Allison and Gorman show that the values that PND produce are closely correlated with sample size and are thus meaningless.

Allison, D. B., & Gorman, B. S. (1993). Calculating effect sizes for meta-analysis: The

case of the single case. Behavioral Research Therapy. 31, 621 – 631.

            The authors discuss previous methods for deducing the efficiency for interrupted time-series designs. They present arguments for the limitations of several models such as the PND, PZD, and Gorsuch method and champion the method developed by Center et al. They state that, “Center et al. present the most sophisticated methodology for the calculation of effect size in single case designs. They developed a regression approach in which the independent or combined effects of level, trend, or changes in slope can be calculated.”

Baron, A., & Derenne, A. (2000). Quantitative summaries of single-subject studies: What

do group comparisons tell us about individual performances? The Behavior

Analyst. 23, 101 – 106.

            In this article, Baron and Derenne are criticizing a quantitative summary completed by Kollins, Newland, and Critchfield in which they found unexpected results that they believe should be a springboard for new research. Baron and Derenne argue that while they agree that empirical research should result from the work of Kollins and his colleagues, it should not further the idea of quantitative summaries but call into question the procedure that brought about different findings. The authors conclude by stating their indifference to these summaries, as they aren’t worthy of discussion in the arena of outcomes.

Browne, W. J. (2003). MCMC Estimation in MLwiN. London: William J. Browne.

            This is the manual for MLwiN. It provides guidelines and more information on the MCMC functions available for making statistical computations with the computer program.

Bryk, A. S., & Raudenbush, S. W. (1988). Toward a more appropriate conceptualization

of research on school effects: A three-level hierarchical linear model. American

Journal of Education. 1, 65 – 108.

            The authors illustrate the difficulties in measuring change in educational arenas and the “unit of analysis” problem. They argue that the development of hierarchical linear models have allowed for the analysis of such information to begin to take place. The authors illustrate how these hierarchical linear models have been used to do so, and summarize their article by providing a model that they believe should be the basis for more research.

Busk, P. L., & Serlin, R. C. (2005). Meta-analysis for single-case research. In T. R.

Kratochwill & J. R. Levin (ed.), Single-Case Research Design and Analysis: New

directions for psychology and education. Hillsdale: Lawrence Erlbaum

Associates, Publishers.

            The authors discuss previous methods for meta-analysis and the assumptions that come with them. They also mention three different instances during which one would attempt to do a meta-analysis with single-case research. These include multiple baseline designs, ABAB designs, and ABC designs. Finally, they make their recommendations for meta-analysis of single-case research, which is to first decide what the purpose of the meta-analysis is. Should one be intent on reviewing the literature, Glass et al. (1981)’s methods are sufficient. Otherwise, if one is trying to test a specific hypothesis then one would use the Hedges and Olkin (1985) method.

Busse, R. T., Kratochwill, T. R., & Elliott, S. N. (1995). Meta-analysis for single-case

consultation outcomes: Applications to research and practice. Journal of School

Psychology. 33, 269 – 285.

            The authors of this article discuss the importance of systematic analysis for single-subject research. They then propose the use of effect size to do such analysis. In their argument they reference many other methods of meta-analysis such as ITSACCOR. The authors concluded their article by making several recommendations for using meta-analytic methods to evaluate outcomes of single-case consultation treatment (Busse et al., 1995).

Carpenter, J, Goldstein, H, & Rasbash, J. (1999, December). A non-parametric bootstrap

for multilevel models. Multilevel Modelling Newsletter, 11, 2 – 5.

            Carpenter et al. discuss the bootstrap (or synthetic) parameters for statistical models. They describe the two different forms of bootstrap parameters, case re-sampling and residual non-parametric bootstrap. The authors then give examples of usage for the residual non-parametric bootstrap for multilevel models and argue that this bootstrap provides a robust alternative to a fully parametric bootstrap, and can be used where standardized residual plots indicate departures from normality.

Center, B. A., Skiba, R. J., & Casey, A. (1985). A methodology for the quantitative

synthesis of intra-subject design research. Journal of Special Education. 19, 387 –

400.

            The authors of this article discuss the limitations that one encounters when attempting to include single-case experiments in meta-analysis. They argue that this is due to a lack of suitable statistical methodology. They then describe a regression model that can be used to generate effect sizes for both changes in slope and changes in level occurring as a result of treatment intervention is outlined (Center et al., 1985). The authors then compare meta-analyses using said method and the ANOVA model between phases. They used 35 single-case behavioral experiments based on criteria set forth in their 1986 publication. The effect size that resulted from the regression model was 1.66 and 1.88 when the combined effects of level and slope change were taken into account (Center et al., 1985). ANOVA produced an effect size of 3.20. The conclusion made is that the piecewise regression model is superior to the ANOVA model due to its ability to not trends and the changes in trends.

Critchfield, T. S., Newland, M. C., & Killins, S. H. (2000). The good, the bad, and

aggregate. The Behavior Analyst. 23, 107 – 115.

            The purpose of this article is to defend the process of generalizing data from different types of research known as literature review. Previous articles, such as that written by Baron & Derenne (2000), argue that these methods are not valid as they use data from dissimilar research. Critchfield et al. argue that generalities cannot be made without also varying how issues are studied. Several examples of such reviews are listed in hopes of defending their claim.

Crosbie, J. (1993). Interrupted time-series analysis with brief single-subject data.

Journal of Consulting and Clinical Psychology, 61, 966 – 974.

            In this article, Crosbie argues that ITSACORR is a beneficial new interrupted time-series that will make better estimates of autocorrelation. Usually difficult to accurately portray in short series studies, ITSACORR is presented as easy to use and practical for use with real data (Crosbie, 1993).

Gingerich, W. J. (1984). Meta-analysis of applied time-series data. Journal of Applied

Behavioral Science. 20, 71 – 79.

            The author discusses a brief history of single-subject research and how current trends in such research provide more applicable results where group research designs could not. Gingerich then describes the methods of meta-analysis used by Smith, Glass, and Miller (1980). These include developing a hypothesis, defining the population of studies, collecting data and developing a common metric (effect size), conducting data analysis, and preparing interpretations of the findings. The author then discusses the importance of validity for these tests. While the results of one single-subject design experiment can be brought under heavy scrutiny, Gingerich writes, the combined conclusions of many provide insight to both the statistical significance and the magnitude of client change following intervention (Gingerich, 1984).

Gorsuch, R. L. (1983). Three methods for analyzing limited time-series (N of 1) data.

Behavioral Assessment. 5, 141 – 154.

            Gorsuch addresses the need for meta-analysis and the role of autocorrelation as a fundamental issue regarding the assumptions its involved in when comparing data from different projects. He then discusses the method known as GLS (generalized least squares). Gorsuch argues that GLS is better in comparison to the Box and Jenkins method due to the smaller amount of parameters examined. The result is a model that has a lowered likelihood to be over influenced by chance findings. The author then provides three ways in which GLS reduces autocorrelation.

Kavale, K. A., Mathur, S. R., Forness, S. R., Quinn, M. M., & Rutherford, R. B. (2000).

Right reason in the integration of group and single-subject research in behavioral

disorders. Behavioral Disorders. 25, 142 – 157.

            Kavale et al. discuss the issues with synthesizing single-subject research including the obstacle of finding the equivalent of an effect size, which is used for the meta-analysis of group research. They then discuss the pluses and minuses of many current methods of meta-analysis of single-subject designs. These include ANOVA, baseline or baseline and treatment standard deviations, least squares procedures, alternative regression approaches, interrupted time-series analysis, nonparametric techniques, the Busk and Serlin method, and the Busse, Kratchowill, and Elliott. The authors conclude that the percentage of nonoverlapping data method (PND) is the best method for meta-analysis of single subject design.

Kromrey, J. D., & Foster-Johnson, L. (1996). Determining the efficacy of intervention:

The use of effect sizes for data analysis in single-subject research. The Journal of

Experimental Education. 65, 73 – 93.

            The authors first illustrate the common methods for analyzing within-subject data such as visual analysis and statistical analysis. They present negatives and positives for both. Then they discuss the use of effect-size formulas for different types of data. They are mean shift, changes in variance, changes in trend, and changes in level when the data show trends (Kormrey & Foster-Johnson, 1996).  They conclude by discussing the interpretation of effect sizes.

Li, Z., & Begg, C. B. (1994). Random effects models for combining results from

controlled and uncontrolled studies in a meta-analysis. Journal of the American

Statistical Association. 89, 1523 – 1527.

            The authors of this article discuss the combination of controlled and uncontrolled studies in meta-analysis. They provide a mathematical method for doing so which combines a Bayes estimator (in substitution for the between-study variance) and least squares. They argue that the estimator can be defended by tests performed when the actual between study variance is known.

Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. (2000). WinBUGS – A Bayesian

modeling framework: Concepts, structure, and extensibility. Statistics and

Computing. 10, 325 – 337.

            The authors of this article describe a computer program named WinBUGS, which is a fully extensible modular framework for constructing and analyzing Bayesian full probability models (Lunn et al., 2000). They discuss modern computing concepts such as object-orientation, modular programming, and run-time linking and their roll in WinBUGS. Then they describe DAGs and the Bayesian statistical methodology that is particularly suited to their analysis (Lunn et al., 2000). They also mention DAGs, how WinBUGS satisfies the fundamental requirements of general MCMC software, how models are specified, the basic concepts of object orientated programming, and how the software is organized into a hierarchical collection of subsystems (Lunn et al., 2000).

Nugent, W. R. (1996). Integrating single-case and group-comparison designs for

evaluation research. Journal of Applied Behavioral Science. 32, 209 – 226.

            The aim of this article is to explore means in which group and single-subject designs can be integrated so as to cover up the shortcomings of both designs. In this article, the author discusses the use of hierarchical linear models as a method of analyzing single-case design data (Nugent, 1996). Then he describes how researchers are presently combining the two different types of designs. Finally, the author breaks down the advantages and disadvantages of combining the two methods.

Salzberg, C. L., Strain, P. S. & Baer, D. M. (1987). Meta-analysis for single-subject

research: When does it clarify, when does it obscure? Remedial and Special

Education. 8, 43 – 48.

            A research article published by Scruggs, Mastropieri, and Casto argues for the use of the percentage of nonoverlapping data method (PND) for the synthesis of single-subject data. In this article, the authors present arguments in apposition to Scruggs, Mastropieri, and Casto. First they argue that most of the important information provided by single-subject designs is in the change over time. The PND attempts to represent those changes with a single number. They also debate the fact that PND may miss vital idiosyncrasies in behavior within and across studies (Salzberg et al., 1987). The authors state that the PND might misrepresent facts and outcomes, and draw inappropriate conclusions about the relative merits of broad categories of intervention (Salzberg et al., 1987).

Scruggs, T. E., & Mastropieri, M. A. (1994). The utility of the PND statistic: A reply to

Allison and Gorman. Behavioral Research Therapy. 32, 879 – 883.

            This article is a response to Allison and Gorman (1993). In that article, Allison and Gorman listed some problematic characteristics for using percent of non-overlapping data (PND), and suggested the use of a regression-based solution to the computation of effect sizes for single-subject research. The authors argue that while Allison and Gorman have definitely pointed out some of the limitations involved with PND, they have overlooked the strengths (Scruggs & Mastropieri, 1993).

Scruggs, T. E., &  Mastropieri, M. A. (1998). Summarizing single-subject research:

Issues and Applications. Behavior Modification. 22, 221 – 242.

            This article describes several procedures for meta-analysis such as the PND and discusses their strengths and weaknesses. They then describe some alternative procedures that have recently been described in the literature and review the results of meta-analysis of single-subject research that had been completed to date (Scruggs & Mastropieri, 1998).

Scruggs, T. E., Mastropieri, M. A., & Casto, G. (1987a). The Quantitative synthesis of

single-subject research: Methodology and validation. Remedial and Special

Education. 8, 24 – 33.

            In this article the authors discuss both the need for objective, systematic review procedures for single-subject research, and the previous approaches that have been previously developed (Scruggs et al., 1987a). The authors then go into great depth outlining the method that they themselves use in meta-analysis of single subject research; the percentage of nonoverlapping data (PND) methodology. They mention both the positives and the negatives, but conclude that it is a preferable method because it is widely versatile and easy to interpret meaningfully (Scruggs et al., 1987a).

Scruggs, T. E., Mastropieri, M. A., & Casto, G. (1987b). Reply to Owen White. Remedial

and Special Education. 8, 40 – 42.

This article is a response to Owen White who has written that the model previously presented by Scruggs & Mastropieri is not sensitive to trends. White had then gone on to propose a new method for calculating PND based on baseline trends (Scruggs, Mastropieri, & Casto, 1987b). They continue to argue that White’s new PND method is not practical since it requires a large number of observations usually not provided from single-subject research.

Scruggs, E. T., Mastropieri, M. A., & Casto, G. (1987c). Response to Salzberg, Strain,

and Baer. Remedial and Special Education. 8, 49 – 52.

            This article is a response to an article written by Slazberg, Strain, and Baer (1987) in which they argue that the method used by Scruggs, Mastropieri, & Casto is insensitive to the complexities of single-subject data. They also presented a method of their own to be used instead. Scruggs, Mastropieri, and Casto believe that methods such as the one presented by Salzberg et al. are more traditional and are not always useful due to the variations of standards used for single-subject research.

Shine, L. C., & Bower, S. M. (1971). A one-way analysis of variance for single-subject

designs. Educational and Psychological Measurement. 31, 105 – 113.

            In this paper, the authors are presenting a One-way case to analyze the variance of single-subject design research. The authors describe the feud taking place between group and single subject researchers, and suggests that there are means of making statistically significant calculations to help analyze single-subject research designs. The One-way case is very similar to a standard One-way ANOVA (Shine & Bower, 1971). It uses repeated measures except that instead of using a group of subjects on which is taken a single trial of repeated measures across the levels of the Experimental factor, a single subject is used on which is taken several trials of repeated measures across the levels of the experimental factor (Shine & Bower, 1971).

Smoot, S. L. (1989). Meta-analysis of single subject research in education: A comparison of four metrics. Unpublished Dissertation, Georgia State University, 1989.

            This dissertation sets out to study four different formulas [percent of nonoverlapping data (PND), Glass’s formula used with group designs (AGF), piecewise regression discontinuity (PW), and a variability ration (VR)] for the computation of effect size in the hopes of finding one that could be deemed most useful.  The research used 23 articles testing treatments for classroom behavior problems and another 13 articles on conduct disorders in preschoolers (Smoot, 1989). Data were extracted from these articles using a microcomputer system. The results of the study lead the researcher to believe that AGF may be the best way to calculate effect size due to the high correlation with both the PW and PND effect sizes, AGF’s ability to be characteristically very similar to the PW, the effect sizes were nearly normally distributed, it is computationally simplistic, and it calculates effect sizes similarly to the effect size measures used in between subject designs (Smoot, 1989).

Smoot, S., Curlette, W., & Deitz, S. (1990, April). Meta-analysis of single-subject research in special education: A common metric and a computerized method. Unpublished Manuscript available from ERIC (ED 322 151, TM 0114 876).

            In this article, the authors compare three types of meta-analytic methods to test which one is the best measure. Secondly, they set out to discover the best means for extracting visual data from a graph in numerical form. They compared a piecewise regression formula used by Center, Skiba, and Casey (1985-86), the percent of nonoverlapping data (PND) statistic, and an altered version of Glass’ formula for group experiments. The authors concluded that the altered Glass formula was most likely the best because of the correlation to the other two methods, it was similar to the piecewise regression formula which includes behavior change over time as a factor, the distribution of the ES generated were practically normally distributed, it is a simple calculation, and because its similar to that which is currently used with group designs (Smoot et al., 1990). The second part of the article used a scanner and a “mouse” device with a microcomputer (Smoot et al., 1990). The programs used were Publisher’s Paintbrush and a BASIC program written to estimate the values of the data points and to do the simper calculations of the effect sizes (Smoot et al., 1990). The validity and reliability of this method was .999.

Van den Noortgate, W., & Onghena, P. (2003a). Combining single-case experimental

data using hierarchical linear models. School Psychology Quarterly. 18, 325 –

346.

            The authors of this article discuss the meta-analysis of single-subject design as a meaningful and important means of analysis as it uses parameters of both the individual and the group. They then describe a hierarchical linear model that consists of one or more regression equations at each level in which the characteristics of the units from that level are used as predictors in describing the coefficients of the equations of the level just below (Van den Noortgate & Ohghena, 2003a). The authors then compare that method to that of Busk and Serlin (1992) and show both used in an example meta-analysis.

Van den Noortgate, W., & Onghena, P. (2003b). Hierarchical linear models for the

quantitative integtration of effect sizes in single-case research. Behavior Research

Methods, Instruments, & Computers. 35, 1 – 10.

            The authors discuss the use of hierarchical linear models and their use in the calculation of effect size measure in single-case research (Van den Noortgate & Ohghena, 2003b). They also discuss meta-analyses that analyze all linear trends in data, and argue that these MA don’t work since they don’t distinguish between effects on level and slope (Van den Noortgate & Ohghena, 2003b). Finally, they propose the use of a multivariate meta-analysis to negate the issues they presented.

Weinstein, K. S., Bray, M. A. & Kehle, T. J. (2004). [Review of Swanson et al. (1999).

Interventions for students with learning disabilities: A meta-analysis of treatment

outcomes.] Psychology in the Schools. 41, 273 – 274.

            This article is a review of the text Interventions for Students with Learning Disabilities: A Meta-Analysis of Treatment Outcomes written by Swanson, Hoskyn, and Lee (1999). The article gives a quick overview of the text and claims that the text is more comprehensive than previous meta-analytical research in the field, and that it is written for grad students and experts in the field interested in learning disabilities (Weinstein et al., 2004). The writers’ final critique is that the text is an excellent and needed comprehensive study of the effectiveness of interventions for students with learning disabilities (Weinstein et al., 2004).

White, D. M., Rusch, F. R., Kazdin, A. E., & Hartmann, D. P. (1989). Applications of

meta analysis in individual-subject research. Behavioral Assessment. 11, 281 –

296.

            The authors of this article discuss current methods of meta-analysis for single subject research. They mention the shortcomings of many methods including the percentage nonoverlapping data method (PND) and the parametric statistical procedure suggested by Scruggs, Mastropieri, and Casto. The authors then outline what they perceive to be the best method to carryout a meta-analysis. This includes locating primary studies, aggregating primary study findings (effect size), and evaluating the impact of the study’s quality on the study’s outcome (White et al., 1989). The writers then discuss the benefits of meta-analysis on single subject research and its limitations.

White, O. R. (1987). Some comments concerning “the quantitative synthesis of single-

subject research.” Remedial and Special Education. 8, 34 – 39.

            In this article, White discusses some issues with the PND (percentage of nonoverlapping data) method. He argues that the PND is too sensitive to abnormal baselines, can’t differentiate between treatments, and is too subject to trends in the data. White’s conclusion is that more research needs to be carried out before this method is widely used.

Zucker, D. R., Schmid, C. H., McIntosh, M. W., D’ Agostino, R. B., Selker, H. P., &

Lau, J. (1997). Combining single patient (N of 1) trials to estimate population

treatment effects and to evaluate individual patient responses to treatment.

Journal of Clinical Epidemiology. 50, 401 – 410.

            The authors discuss the emerging importance of research-based treatments and the resulting focus on single patient trials. They then present a hierarchical Bayesian random effects model as a method of combining these studies to make a general conclusion on the effects of medications. This model uses a paired unit randomization for n paired periods. Each patient has n resulting measure that represent the differences in their disease status scores for each of their paired treatment periods (Zucker, et al., 1997). From there they take a mean difference Y₁ that is assumed to follow a normal distribution centered about a patient’s true mean effect difference m₁ with variance s₁ (Zucker, et al., 1997).