A test for comparing proportions from two independent samples can be performed by using the CHISQ option in the FREQ, or – in SAS® Viya® – the FREQTAB procedure. (For information about comparing dependent proportions, see SAS Note 22562.) The two independent proportions in the two samples can be displayed in a 2×2 table. For example, suppose you ask 50 men and 50 women if they voted on election day. You find that 12 men voted (or 12/50 = 24%) and 18 women voted (or 18/50 = 36%) – a difference of 12%. You want to test whether there is a significant difference in the probabilities of men and women voting in the population from which you sampled.
The following DATA step creates a data set that contains this data. PROC FREQ displays the 2×2 table. The WEIGHT statement specifies the variable that contains the cell counts of the table. If you have raw data (one observation per person recording their gender and whether they voted) instead of cell counts, then the WEIGHT statement can be omitted. The CHISQ option requests chi-square tests and other statistics.
data vote; input gender $ vote $ count; datalines; male yes 12 male no 38 female yes 18 female no 32 ; proc freq; weight count; table gender*vote / chisq; run;
In the displayed results from PROC FREQ below, the Row Pct values in the VOTE="yes" column are the proportions of men and women who voted in your sample. In the statistics table, the first statistic labeled Chi-Square, is Pearson's chi-square statistic, which has a value of 1.714. Pearson's chi-square statistic tests the hypothesis of equal population proportions. Large values of the statistic indicate inequality of the population proportions. The small p-value (p = 0.1904) in the Prob column means that the probability of obtaining a chi-square value at least as large as 1.714 when there is really no difference in population proportions is 0.1904.
Based on this information, should you reject the hypothesis of equality? Should you accept it? Can a conclusion even be made about the equality of the population proportions? Consider that there are two types of errors that you can make based on this test. You make a type 1 error by rejecting equality when the population proportions really are truly equal. The chance of a type 1 error is denoted by α (alpha). It is also called the size or significance level of the test. You make a type 2 error by accepting equality when the population proportions are truly not equal. The chance of a type 2 error is denoted by β (beta) for a given amount of difference between the population proportions.
If you reject equality in the voting example, then your chance of making a type 1 error is α = 0.19. If this chance were smaller, say 0.05 or less, then most analysts would conclude that observing a value of Pearson's statistic as large as 1.714 indicates that the population proportions truly do differ. They would then reject the hypothesis of equality and accept the 0.05 chance of a type 1 error. However, in this case, the 0.19 chance indicates that such large statistic values can occur reasonably often when the population proportions are truly equal. Because you might have merely observed such a case with this data, you cannot conclude that the population proportions differ.
But should you conclude that men and women really voted in equal proportions? There was, after all, a difference of 12% in the observed proportions. Just as you needed to know the chance of a type 1 error when deciding whether to reject equality, you now need to know the probability of making a type 2 error if you accept the hypothesis of equality. The power of this test is the probability of rejecting equality when the population proportions differ by a given amount. The probability of a type 2 error, β, is the probability of accepting equality given this amount of difference and is equal to 1-power.
The POWER2x2 macro (SAS Note 25012) provides power and β of the Pearson chi-square statistic when computed for 2×2 tables such as this voting example. Some details of the power computations that are used by the macro are given in the Note below. For computing power for two-way tables larger than 2×2, see Example 3 in the documentation of the POWERRxC macro (SAS Note 25013).
Power depends on sample size, the significance level of the test, α, and the unknown population proportions. For each of these, you supply values at which you are interested in computing the power. It's a good idea to compute power for several settings close to what you expect the true proportions to be. For this example, assume that the population proportions really differ by the 12% observed. By setting the significance level of the test (chance of a type 1 error) at α = 0.05 (the default) and both sample sizes at 50, the POWER2x2 macro provides the power of the test that was performed above.
%power2x2(p1=.36, p2=.24, n1=50, n2=50, level=.05)
The macro displays the following results:
The power for this test can also be computed by using the POWER procedure. The TWOSAMPLEFREQ statement provides power calculations for tests that compare two proportions. The TEST=PCHI option computes results for the Pearson chi-square test. Specify the observed proportions (GROUPPROPORTIONS=), group sample sizes (NPERGROUP=), and the null hypothesis difference (NULLPDIFF=). By default, the significance level is 0.05. POWER = . requests computation of the power. Calculations for the likelihood ratio chi-square test and Fisher's exact test are also available. The following statements compute the power of the Pearson test for the voting example:
proc power;
twosamplefreq test=pchi
groupproportions=(.36 .24)
nullpdiff=0
npergroup=50
power =.;
run;
Both the macro and procedure estimate the power of the Pearson test to be 0.256 indicating that, if the population proportions really differ by 12%, then your chance of incorrectly accepting equality is almost 75%. Stated another way, your chance of detecting a 12% difference is only 25%. Of course, your chance of detecting larger differences will increase. But for the sample size, significance level, and difference in population proportions that are assumed above, the resulting risk is large and probably unacceptable. The results from these data leave you in a gray zone – you can neither accept nor reject the hypothesis of equality without incurring an unacceptably large risk of error.
The problem is that the data provide insufficient evidence to accept or reject equality. Had the same proportions resulted from samples of 1000 men and 1000 women, then there would not have been this ambiguity. Power is affected by sample size – as sample size increases, power increases and β decreases. You can avoid inconclusive results after the expense and effort of data collection by selecting a sample size that is expected to yield adequate type 1 (α) and 2 (β) error rates.
Suppose at the next election you plan to do another study of gender and voting and you would like to pick a sample size to avoid inconclusive results. Assuming equal-sized samples, you'd like to examine power and beta for total sample sizes that range from 10 to 1000. To compute the power for rejecting the null hypothesis of equal probabilities, you must specify the expected voting probabilities for men and women at which to calculate the power and the α level that you are willing to accept. Suppose you can tolerate a 0.05 probability of a type 1 error. From your previous study, a 30% overall voter turnout can be expected. The POWER2x2 macro enables you to find a sample size that will detect a 12% difference in population proportions with reasonably high probability. Again, because the population proportions are unknown, you might want to try several reasonable settings of the difference.
%power2x2(p1=.36, p2=.24, nmin=10, nmax=1000)
The following results are displayed:
Again, PROC POWER can produce the same results. Adding the PLOT statement provides a graph showing how power increases with sample size.
proc power;
twosamplefreq test=pchi
groupproportions=(.36 .24)
nullpdiff=0
ntotal=10, 50 to 1000 by 50
power =.;
plot yopts=(ref=0.9 crossref=yes);
run;
PROC POWER produces the following plot in addition to tables (not shown) with the same values as from the POWER2x2 macro.
Alternatively, you can have PROC POWER directly find the total sample size needed to achieve specified type 1 and type 2 error rates. Specify the desired power in POWER= and NTOTAL= . to solve for the notal sample size needed. The following statements request the total sample size needed to achieve α = 0.05 and β = 1 - power = 0.90.
proc power;
twosamplefreq test=pchi
groupproportions=(.36 .24)
nullpdiff=0
ntotal=.
power = .9;
run;
The results from PROC POWER follow.
Unfortunately, these results show that it will take a lot more data to avoid inconclusive results. If you want a 90% chance of detecting a difference of 12% between the population proportions, then you'll need to take a sample of at least 600 (300 men and 300 women). This would lower the chance of making a type 2 error to about 10%.
The sample sizes that are shown are estimates based on your guesses of the voting probabilities. As the probabilities become either very small or very large, the variance of the difference in proportions decreases, causing the power for a given sample size to increase. The sample size estimates are also affected by the difference that you want to detect and the type 1 error rate that you choose. If you decide that you want to detect a smaller difference in the population proportions, then more data will be required. If you decide that you are willing to accept only a 0.01 type 1 error probability, instead of 0.05, then you again will need more data. Experiment with different settings and notice the results. The graphs available with the PLOT statement in PROC POWER can help to visualize the effect of changing the above criteria.
Power and beta are computed by the macro, using the statements below. p1 and p2 are the group success probabilities under the alternative hypothesis. n1 and n2 are the samples sizes for the two groups. ph0 is the overall success probability, which assumes the null hypothesis of no difference. stdh0 is the standard error under the null hypothesis, and stdha is the standard error under the alternative hypothesis. The PROBNORM function returns the probability of smaller values from a standard normal variate. The PROBIT function is the inverse of PROBNORM and returns the value of a standard normal variate that is associated with the given probability.
diff=p1-p2;
ph0=(n1*p1+n2*p2)/(n1+n2);
stdh0=sqrt(ph0*(1-ph0)*(1/n1+1/n2));
stdha=sqrt(p1*(1-p1)/n1+p2*(1-p2)/n2);
power=1-probnorm(-probit(level/2)*stdh0/stdha-diff/stdha) +
probnorm( probit(level/2)*stdh0/stdha-diff/stdha);
beta=1-power;
Agresti, A. 2002, 2013. Categorical Data Analysis. New York: John Wiley & Sons, Inc.