[COE/EDP 502]  [EDP 552]  [EDP 554]  [Using SPSS]  [Q and A]  [Students' Sites]  [Resources]

 

Post Hoc Comparisons

 

There are two lessons included : SPSS Lesson and Vassa Stats Lesson.

The SPSS ONEWAY procedure produces a one-way analysis of variance. ONEWAY also allows you to specify contrasts, use multiple comparison tests, and to test for nonlinear trends.

A Priori Comparisons vs. Post Hoc Comparisons

It is important  to distinguish between a priori comparisons, which are chosen before the data are collected, and post hoc comparisons, which are tested after the researcher had collected the data.

An Example

To evaluate the effects of four different teaching methods, twenty-eight junior high students have been randomly selected and assigned to one of four teaching methods (A, B, C, and D). Their scores on the final examination were obtained.

Independent variable and Dependent variable

bullet

The type of teaching methods is the independent variable. It has four levels.

bullet

The dependent variable is scores on the final exam.

Appropriate Tests of Significance

Can you use the t test to compare differences among four groups?

No. The t test is limited to the comparison of two groups at a time. The F-test is the appropriate statistical method to be called for. In the example, the four groups are independent. We will choose the SPSS One-Way ANOVA procedure to analyze our data.

Assumptions

Analysis-of-variance procedures require the following assumptions:

bullet

Each of the groups is an independent random sample from a normal population.

bullet

In the population, the variances of the groups are equal.

Check the assumption of equal variances: To test the null hypothesis that the four groups come from populations with the same variance, you can use the Levene test.

Hypotheses

Are there significant differences among the four teaching methods?

a. State the null hypothesis

All groups have the same means in the population.

b. State the alternative hypothesis.

The population means are not all equal

c. Use a .05 significant level.

d. The Overall F Test

The statistical test for the null hypothesis that all groups have the same mean in the population is based on the F ratio. The F ratio is often called the overall or omnibus F test.

Multiple Comparison Procedures

A significant F value tells you only that the population means are not all equal. Use multiple comparison procedures to determine which means are significantly different from each other.

All Possible Comparisons

The Scheffe Procedure

The Scheffe procedure allows for a comparison of all possible paired comparisons (e.g., M1 vs. M2) and complex comparisons between combined means (e.g., (M1 + M2)/2 vs. (M3 + M4)/2 ).

The Tukey HSD Procedure

The Tukey HSD (honestly significant difference) procedure only allows for a comparison of the possible pairs of means. For example, if there are four groups, six possible paired comparisons (comparisons between individual means) can be performed.
 

  Group 1 Group 2 Group 3 Group 4
Group 1        
Group 2 M2 vs. M1      
Group 3 M3 vs. M1 M3 vs. M2    
Group 4 M4 vs. M1 M4 vs. M2 M4 vs. M3  

 

Two kinds of Type I Error

1. Per comparison type I error (alpha) refers to the probability of committing a type I error for an individual statistical test. 

2. Experimentwise error (or familywise error) refers to the probability of committing type I errors for a set of statistical tests in the same experiment. 

When you make many comparisons involving the same means, the probability that one of the comparisons will be statistically significant increases. Thus, experimentwise error may exceed a chosen significance level.

Multiple comparison procedures protect us from obtaining too many significant results. The more comparisons you make, the larger the difference between pairs of means is required to find them significant.

For example, the Tukey HSD (honestly significant difference) procedure allows for a comparison of the possible pairs of means and maintains the experimentwise error at the chosen level.
 

SPSS for Windows

First, explore the data.

A. Create a new data file: File \ New \ Data. 

B. Define the two variable names: method and final. 

C. Examining the data. Are there any unusual data values or patterns?

Use the Explore procedure to obtain a default boxplot for each group. Choose Analyze \ Descriptive Statistics \ Explore.  move the variable `final` to the dependent variable list and `method` to the factor list. Click on OK.

SPSS Printout

1. Sample Means and Confidence Intervals

         Sample Mean

The sample mean is the best guess for the unknown population mean.

  List the sample mean for each group.

         

(1) Group 2 has the largest mean. The mean values for Group 1 and group 3 are very close. Group 4 has the smallest mean.

(2) There are four groups. How many possible paired comparisons (comparisons between individual means) can be made?

[4(4-1)]/2 = 6 

Confidence Intervals

It is unlikely that the sample mean is exactly equal to the population mean. However, based on the sample mean and standard error of the mean, we can construct a 95% confidence interval that is likely to include the unknown population mean. 

List the confidence interval for each group mean.

        

Preliminary Examination.

Examine the confidence intervals for M1, M2 and M3.

Do the confidence intervals for M1, M2 and M3 overlap? Yes.

Will there be any significant differences among these three groups?

Examine the confidence intervals for M1, M2, M3 and M4.

Do the confidence intervals for M4 and every other group (M1, M2, and M3) overlap? No.

Will there be any significant difference between Group 4 and every other group? 

2. Boxplot

A boxplot plots the median, the 25th percentile, the 75th percentile, and values that far from the rest.

What to look for

a. The horizontal line inside the box identifies the group median. Examine the medians for the four methods. 

b. The larger the box, the greater the spread of the data. Examine the vertical length of the boxes ( a measure of the spread) .
 

What do you observe?  

SPSS for Windows

Second, conduct a one-way analysis of variance and select the Tukey HSD test to determine which pairs of group means are significantly different.

A. From the menus choose: Analyze \ Compare Means \ One-Way ANOVA.

Select the dependent variable (final) and the factor variable (method).

B. To produce post hoc multiple comparison tests, click on Post Hoc.

Select Tukey (Tukey`s honestly significant difference).

There are many post hoc comparison methods. Some years ago, Scheffe was de rigueur. Nowadays, it is Student--Newman-Keuls. The LSD of the SPSS is little more than a t-test. Scheffe is the most rigorous, LSD the least, the rest try to take the middle road.

Mathematics is part of the general system of formal logic. This system is a deductive system of abstract reasoning and if there is more than one solution, all cannot be right. One has to be right and the rest of them wrong, or all must be wrong. From this viewpoint. all  post-hoc tests are wrong. However, it is possible, that they approximate the solution to some degree.

C. To obtain descriptive statistics and the Levene statistic, click on Options.

Select Descriptive and Homogeneity-of-variance. Click Continue. Click OK.

 SPSS Printout

Check the assumption of equal variances

Do the four groups come from populations with the same variance?

To test the null hypothesis that groups come from populations with the same variance, you can use the Levene test.
 

The observed significance level is larger than .05. The null hypothesis is not rejected. The two variances are equal. 

Analysis of Variance   
 

a. For this example, F(3,24) = 6.663. The observed significance level was .002. There were significant differences among the four teaching methods. The differences among groups represented systematic effects.

b. About 45% of the variation in the scores was accounted for by teaching methods.

The sums of squares are additive. SSb + SSw = SSt 



 

Post hoc comparisons should be conducted only if a significant result is obtained in the overall analysis of variance. 


Post Hoc Comparisons

When one-way ANOVA rejects the null hypothesis of equal means, the multiple comparison procedure may be applied to answer the follow-up questions: Which groups have different means?.

The Tukey Test

In the Tukey’s test results, the formula indicates how large an observed difference must be for the multiple comparison procedure to call it significant.

1. Compute the value of HSD

Any absolute difference between means has to exceed the value of HSD to be statistically significant.

a. The listed range (3.9) is the critical value

b. The actual range is a product of the listed range and the square root of the within group mean square divided by the number of subjects in each group. In our example, there are 7 subjects in each group.

         HSD = (3.9) (1.1525) = 4.4947

2. Pairwise Differences

What to look for

Group names are listed in ascending order. Asterisks in the matrix of group names indicate which pairs of means are significantly different.

All six pairwise differences for four means are shown below:

         

Three of the differences, 5.2857 and 5.4285 and 6.7142, are larger than the HSD value (4.4947).

The asterisks in the first column indicate that the mean of Group D (coded as  group 4) is significantly different from every other group. The teaching method 4 is less efficient than the teaching methods 1, 2, and 3.

3. Homogeneous Subsets

Groups that appear in the same homogeneous subset are not significantly different from each other.

bullet

The teaching method 4 is in a subset of its own. It is significantly different from all of the other means.

bullet

The teaching methods 1, 2, and 3 are not different from each other. They are in the same subset.  

 

Web Resources

Online statistical analysis tools: VassaStats (Very easy to use)
 

1. Choose ANOVA from the left panel. Click One-Way ANOVA.

2. Enter 4 in the Number of samples in analysis textbox. Next, click the Independent Samples button.
 


3. Click the weighted button.

4. Enter Data. 

Instructions:

After clicking the cursor into the scrollable text area for each sample, enter the values for that sample in sequence, pressing the carriage return key (the Enter key) after each entry except the last value of each sample.

    5. Click the Calculate button.
    6. Output

    A. Measures of Central Tendency and Variability

   


    B. ANOVA Summary Table in Sum of Squares Form
 

1. There were significant differences among the four teaching methods, F(3,24) = 6.66, p < .05. The differences among groups represented systematic effects.

2. About 45% (185.86 / 409 = .4544) of the variation in the scores was accounted for by teaching methods.

The sums of squares are additive. SSb + SSw = SSt 



 

C. Post Hoc Tests

Post hoc comparisons should be conducted only if a significant result is obtained in the overall analysis of variance. 

1. Compute the value of HSD

Any absolute difference between means has to exceed the value of HSD to be statistically significant.

HSD = 4.5

2. The mean of Group D (coded as  group 4) is significantly different from every other group. The teaching method 4 is less efficient than the teaching methods 1, 2, and 3.

HyperStat Online Textbook

http://www.davidmlane.com/hyperstat/intro_ANOVA.html

 

 

[Home]