ES: A Computer Program and Manual for Effect Size Calculation
by
William R. Shadish
Leslie Robinson
and
Congxiao Lu
The University of Memphis
ES (Effect Size) is a computer program for calculating the effect size measure called the standardized mean difference statistic (d), computed as
.
ES is particularly useful for effect size computation in meta-analysis, but may also be of some use in computing effect sizes for use in power analysis.
ES can compute d using more than 40 different kinds of data including the standard raw data (continuous or dichotomous outcomes) or means and standard deviations, but also from various forms of data from t-tests, F-tests (oneway, two-way repeated measures, or factorial ANOVA with up to five factors), change scores, probability levels, correlations, and multivariate statistics. The data sources on which these calculations can be made are:
Raw score means, standard deviations, and sample sizes (01)
0101: Raw score means, standard deviations, and sample sizes
Dichotomous outcome, 2 x 2 table (02)
0201: Dichotomous outcome, 2 x 2 table: cell frequencies
0202: Dichotomous outcome, 2 x 2 table: chi-square and total sample size
T-test statistics (03)
0301: Between-groups t-test on raw posttest scores
0302: Raw means and sample sizes on three or more groups, with a t-statistic comparing one group to a combination of the other groups
0303: Dependent t-test on each of two groups, raw pretest and posttest means and sample sizes for each group, pretest-posttest correlation for each group
0304: T-test for two matched groups, sample sizes, correlation between groups
Change score data (04)
0401: Between-groups t-test on change scores with intraclass correlation
0402: Change-score means and change-score standard deviations with intraclass correlation
0403: Two-group between-groups oneway F-statistic on change scores with intraclass correlation
0404: Three-group or more between-groups oneway F-statistic on change scores, intraclass correlation, change score means and mean square error are available
0405: Three-group or more between-groups oneway F-statistic on change scores, intraclass correlation, change score means and cell sample sizes are available
0406: Change score means and sample sizes on three or more groups, t-statistic comparing one group to a combination of the other groups
Oneway F-test statistics (05)
0501: Two-group between-groups F-statistic on raw posttest scores
0502: Three-group or higher between-groups oneway ANOVA on raw posttest scores: group means, sample sizes, and F-statistic
0503: Three-group or higher between-groups oneway ANOVA on raw posttest scores: treatment and comparison group means and mean square error
Probability levels and reports that results are significant or nonsignificant (06)
0601: From probability level and sample size for groups
0602: Coding results described only as significant if sample size for groups is known.
0603: Coding results described only as nonsignificant
Multifactor completely-between subjects ANOVA (07)
0701: Sum of squares and Degrees of Freedom for all sources, and Marginal Means for Treatment Conditions
0702: Mean Squares and Degrees of Freedom for all sources, and Marginal Means for Treatment Conditions
0703: Sum of squares and Degrees of Freedom for all sources, with Cell Means and Cell Sample Sizes
0704: Mean Squares and Degrees of Freedom for all sources, with Cell Means and Cell Sample Sizes
0705: Cell means, cell sample sizes, the F-statistic for the treatment factor, and the degrees of freedom for the error term
0706: F-statistics and degrees of freedom for all sources, sample size for treatment and comparison groups, where treatment factor has only two levels
Two-factor (groups x time) repeated measures ANOVA (08)
0801: Between-groups mean square error, within-groups mean square error, posttest means, and sample sizes
0802: F-ratio for groups, F-ratio for time, cell means and sample sizes
0803: F-ratio for groups, F-ratio for group x time interaction, cell means and sample sizes
0804: Between-groups mean square error, intraclass correlation, treatment and comparison group means and sample sizes
0805: Within-groups mean square error, intraclass correlation, treatment and comparison group means and sample sizes
0806: Two-Factor (Groups x Time) Repeated Measures ANOVA with only two time points: F-ratio for group x time interaction, intraclass correlation, treatment and comparison group change score means and sample sizes
0807: F-ratio for group x time interaction with one degree of freedom for the interaction, intraclass correlation, and treatment and comparison group sample sizes
0808: F-ratio for time, cell means, cell sample sizes, and intraclass correlation
0809: F-ratio for group x time interaction, cell means, cell sample sizes, and intraclass correlation
0810: F-ratio for groups, cell means, cell sample sizes, and intraclass correlation
Oneway two-group or more ANCOVA (09)
0901: Oneway two-group or more ANCOVA: Mean square error term, F for covariate, raw score means, and total sample size
0902: Oneway two-group or more ANCOVA: Mean square error term, F for covariate, adjusted means, and total sample size
0903: Oneway two-group ANCOVA: F for groups, sample size per group, partial correlation
0904: Oneway two-group or more ANCOVA: F for groups, sample size per group, partial correlation, adjusted means and cell sample sizes
r to d conversion (10)
1001: Converting r to d
1002: Converting d to r
Multivariate statistics (11)
1101: Two-group multivariate test with multiple outcomes and sample size for groups
Raw data, except dichotomous outcomes (12)
1201: Raw data (except dichotomous data)
Other (13)
1301: Other
The ES program does more than effect size calculation. It can also
 | save data used to compute effect sizes, |
| add new effect sizes to existing files so data entry can be done in multiple sessions, |
| find and correct previously saved data, |
| delete previously saved cases, and |
| create an export file of basic effect size data for use in other computer programs that can read text files. |
ES has a user-friendly, point-and-click interface for all these functions. It can be started from an icon that can be placed anywhere on the computer, such as on the Desktop or in a separate directory.
Clicking the ES icon opens the program’s Main Screen. The Main Screen contains the central working menu of the ES program (can be computed). Users select one data source from this list, and this selection then takes the user to subsequent screens that guide data entry and saving. The Main Screen also has buttons that allow users to find or delete previously saved data (by identification number), and to create a partial export file. The latter is output in text-only format so that it can be ready by most data analysis programs such as Microsoft Excel, SPSS, and SYSTAT. The export file contains identification numbers, sample sizes, effect size, and effect size calculation method number, which are the essential effect size data needed for most meta-analytic work.
To illustrate ES’s effect size calculation capabilities, consider an example where the user wants to compute d using data from a t-test. Selecting "t-test" from the main menu leads to a screen offering the choice of four possible calculation methods depending on the data available:
Choosing one of the four available t-test options then leads to a data entry screen. For instance, choosing the first option (Between-groups t-test on raw posttest scores) yields the following screen (with sample data filled in):
The screen has spaces to enter the data (sample sizes for the two groups, and the t-statistic), up to four identification numbers (e.g., to record a study identification number), an indication of which group did better (to determine the sign of the effect size), and up to 100 characters of notes to record such things as the location of the data in the study report or any peculariaties in the data. The screen also has buttons to allow saving the newly-entered data into a new file (save as) or a previously created file (save), or to load previously-saved data for review or modification (load data).
All the data entry screens in ES function similarly, although they may occasionally require responses to intermediate screens that ask for further clarification about the data available (e.g., about the number of factors in the ANOVA) prior to showing the data entry screen. Most screens ask for considerably more data to be entered than this example illustrates (e.g., for large factorial ANOVAs the user may have to enter cell means and sample sizes, the number of which can be considerable); but the process is always similar.
The ES program is accompanied by an extensive manual. The manual explains all the details of how to use ES, and it contains extensive descriptions, formulas, and examples for each of the effect size calculation methods.
Minimum requirements: Windows 95 or Windows NT 4.0 or higher, hard disk with 2MB of free space.
ES is available for purchase from the Assessment Systems Corporation; and a free demonstration version of ES is also available for downloading over the Web. Visit their Web site at http://www.assess.com, and visit the ES page on that site clicking on the Effect Size and Meta-Analysis link on that site. Both the full version for purchase and the free demo version are available at the latter site.