Ken Beneth says:

Hello Stephan,

Thank you for your post! You will need to conduct sample size analysis.

If you choose to use a power calculator, there are Web sites that offer statistical power calculators. Both of the following Web sites offer options for calculating sample sizes given a variety of problems.

• http://calculators.stat.ucla.edu/
If the UCLA power calculator is down please use:
• http://www.stat.uiowa.edu/%7Erlenth/Power/
• http://www.psycho.uni-duesseldorf.de/aap/projects/gpower/

Think also about your design: independent and dependent t-tests, correlation, and 3-5 group ANOVA designs

In order to calculate the necessary sample size for your study, you will need to determine values for three items. Whether you use tables or calculators on Web sites, you will need to know the same three pieces of information each of which is explained in detail under separate headings that follow.

• Statistical Power
• Alpha
• Effect size

Statistical Power
The first piece of information you need when conducting a sample size analysis is the statistical power you want your test to have. Statistical power, often referred to simply as power, is the probability that a given statistical test will detect a real treatment effect or real relationship between variables. You need a large enough sample to ensure a reasonable likelihood of detecting a difference (or relationship) if it really exists in the population. If there is not a reasonable chance of detecting a real treatment effect or relationship, then there probably is not a reason to do the study. High statistical power also helps improve the chances that findings are not due to chance alone.

The accepted value for power (the probability that a test will detect a real treatment effect or real relationship) is .80 (80%).

This means that given a specific sample size, you would expect to find a real, or true, effect 80% of the time. In other words, if a study was repeated 100 times, the null hypothesis would be correctly rejected 80 times if there was indeed a real effect/relationship. For example, if your statistical test has statistical power at .50 (50%), this means that your test will do no better at finding a real or true effect than tossing a coin. You are just as likely to guess the outcome by flipping a coin as you would spending the time to collect and analyze data.

Alpha
The second value that must be determined is the alpha (α) level, which is set by the researcher. Like power, this value is predetermined. The conventional values used are
α = .05 or α = .01. When you choose a larger value for alpha, you expand the rejection region. If you expand this area, you provide more opportunities to reject the null hypothesis (correctly). Thus, larger values of alpha (.05) result in more power than small values (such as .01).

It is standard practice to set the alpha level at .05, for most psychological research.

This means essentially that there is only a 5% chance that you will arrive at the wrong conclusion – or, there is a 95% chance that you will arrive at the right conclusion. (Recall Type I and Type II error from your statistics course.)

Effect Size
The final value that must be determined is the effect size. Unlike power and alpha, which rely on predetermined conventions, this value must be calculated by hand. Recall that an effect size gives an indication of how “large” an effect is or how “strong” a relationship is. If an intervention or the like has a large effect, it stands to reason that fewer people would need to be assessed to detect this effect. Conversely, a small effect will require more people to detect an effect. If you are interested in learning more about effect size, there are entire books written about the topic.

One way to calculate a value for an effect size is to use prior research. Recall from your statistics course that:

Effect size = Mean Difference/Standard Deviation.

For the simple two-group design, the effect size is given by the difference between the before and after treatment means divided by the average standard deviation. Cohen’s d is one measure of effect size. It is based on the t-statistic and is calculated as:

d = M1 – M2
SD
Where M = mean and SD = standard deviation

Cohen specified the following effect size conventions:
Small: d .80

Other measures of effect size include: Square of the correlation coefficients (r2); R2 (from a multiple regression); ω2 (measure of effect size for analysis of variance; provides similar values to R2).

Small effect: ω2 .14

Lipsey and Wilson (1993) provide effect sizes for a number of psychological, educational, and behavioral treatments. This document is included in this week’s readings.

You may need to review literature in your area to determine the magnitude of effect sizes. You do this by extracting means and standard deviations from articles and computing a rough estimate.

There are terms that usually confuse students during data analysis, and I will like to explain it .
What is meant by ‘small’, ‘medium’ and ‘large’?


In Cohen’s terminology, a small effect size is one in which there is a real effect — i.e., something is really happening in the world — but which you can only see through careful study. A ‘large’ effect size is an effect which is big enough, and/or consistent enough, that you may be able to see it ‘with the naked eye’. For example, just by looking at a room full of people, you’d probably be able to tell that on average, the men were taller than the women — this is what is meant by an effect which can be seen with the naked eye (actually, the d for the gender difference in height is about 1.4, which is really large, but it serves to illustrate the point). A large effect size is one which is very substantial.