Analysis of Variance

The basic idea is to compare alternative estimates of the population variance for the purpose of deciding whether the groups have differnet means. The alternative estimates are predicated on the notion that the mean for certain groups within the population is the same. Under that working supposition there should be no difference in variance estimates constructed from the different sample groups corresponding to the grouping in the population.

One Factor Analysis of Variance

Suppose we have four heads of household, each with a different type of education regarding the effects of alcohol consumption. Their educational program can be regarded as the treatment. For each of them we record their weekly consumption of distilled spirits for each of three weeks. Is their weekly consumption the same? Formally,



We can formulate the problem in one of two alternative ways:


i=1,2,3 and j=1,2,3

where i denotes the particular head of household and j denotes the week. In this formulation three heads of household are measured against an arbitrary base person.

An alternative formulation is:


i=1,2,3,4 and j=1,2,3

Now each household is measured absolutely.

We have the following data:

	Week
Household	1	2	3
x1	13	8	9	10
x2	15	11	13	13
x3	8	12	7	9
x4	11	15	10	12

denotes the three week mean for the i^th head of household for m=4 households.
The grand mean is = 11.
The sample sizes are n₁=n₂=n₃=n₄=3, and n=.
With respect to the two alternative linear models, the entries in the body of the table are observations on the dependent variable.

If all four households have a common mean, then we could pool the data in the computation of the sample variance as an estimate of the population variance. That is, we could compute the total sum of squares as

SS(TOT)= (1)

But each household may have a different mean, so using (1) would be in error. Accounting for the differences between households, groups, we would use

SS(E)= (2)

In (2) we compute the sum of squares within a group or treatment, then add them up. This is referred to as the error sum of squares. It is large if there is a lot of variation within the group.

We could also see if there is variation between the groups:
SS(B)=

This would be large if there is variation between the groups. That is, there are large differences between the group means and the grand mean.

Things to note:

1. Each of the sums of squares is ² if the variable x is normal with finite mean and variance.

2. Regardless of whether or not the null is true, we know that , but SS(B) is an unbiased estimator of the population variance only if the group means are equal:

Proof:

E[SS(B)]=
        =
Adding and subtracting some terms, we can write this as E[SS(B)]=



=
=
=
This observation suggests the test statistic

m-1 is the number of unique restrictions necessary to impose equality of means across all groups. In the denominator there are n terms in the sum, but we have used the data m times to compute different group means.

For our example:
SS(TOT)=80, SS(E)=50, SS(B)=30

F=

H_o is not rejected.

This sort of analysis of variance is often summarized in a table. Almost every computer package has some variant of this general presentation.

To return to our presentation of the problem as a linear model:

Formulation 1:



With our data and a bit of linear algebra we can write this as:



Now we can use a regression package to estimate the parameters. The intercept, or mean for household 1 is . That for household 2 is . For three it is and for the last household it is . The output for this regression looks like

Note:

1. The t-statistics for the coefficients on D1, D2, D3 are not different from zero. That is, none of the group means differ from the mean of the first group.
2. The SS(E)= 50.
3. The F-statistic in the table is 1.6; the same as our previous answer.
   

Formulation 2:



With our data and a bit of linear algebra we can write this as:



And gotten the regression output:

Note:

1. The top part of the table has not been presented because the information does not change.
2. Now the t-statistics are different from zero. Each of the group means is different from zero, but we don't know if they differ from one another.
3. The F-statistic is again 1.6.
   

If we fit a linear model with the assumption that the means are equal across groups we get

Two Factor Analysis

Suppose that we have four ways of teaching and three different instructors. For the sake of argument, suppose that the standardized test score by class size and teacher is given by:

	Teaching method
Instructor	Lecture	Discussion	Computer	Video
1	16	18	21	21	19
2	14	15	18	17	16
3	15	15	18	16	16
	15	16	19	18

Now we believe that the mean test score is affected by instructional method and specific instructor. We could write this as a linear model:



where _i is the row effect and _j is the column effect. Each instructor and method pair is measured relative to the base instructor 1 and lecture method. With this in mind, we have , and in the table x_ij is the mean test score in equal size classes for i=1,2,…,a rows and j=1,2,…,b columns.

The regression results are

where B2: discussion, B3: computer, B4: video and the Aj represent the second and third instructors. The F-statistic indicates that the coefficients are not all zero. In fact, judging from the t-ratios, only the coefficient on B2 is statistically no different from zero.

In general, when there are 'a' rows and 'b' columns, we have

So we revise our ANOVA table as

To test the row effect

To test the column effect



For the example