Introduction

In analyzing the results of studies we often encounter the problem of comparing two binomial success probabilities p₁ and p₂, p₁ > 0 and p₂ > 0. Implicit in this comparison is the independent observations X₁ ~ B (n₁, p₁) and X₂ ~ B (n₂, p₂). The most common comparison is the hypothesis H_o: p₁ = p₂ versus H_a: p₁ ≠ p₂. Accompanying this test is the construction of a confidence interval for the difference between p₁ and p₂. The most commonly described textbook method is the Wald-z. Occasionally, a continuity corrected version is given (Wald-c). Nearly all introductory statistics textbooks include the Wald-z confidence interval and issue a warning - usually in a footnote - when not to use it.

The general problems associated with constructing a confidence interval for the difference between two independent proportions are similar to those associated with constructing a confidence interval for a single proportion - overshoot and poor coverage probability. Despite these shortcomings, the Wald-z and Wald-c methods continue to dominate in textbooks. The purpose of this paper is to review the coverage probability functions of the Wald methods and a set of alternative non-iterative methods for computing a confidence interval for the difference between two independent proportions.

Methods

The primary criteria for evaluating a confidence interval method is the coverage probability function. This coverage probability for the difference between two independent proportions, C(π₁,π₂|n₁,n₂,α), is found by fixing n₁, n₂, π₁, and π₂, then computing the confidence interval for each x_i= 0, . . ., n_i for i= 1, 2. The coverage probability is then defined by:

C(π₁,π₂|n₁,n₂,α) = Σ Pr(X₁=x₁|n₁,π₁)Pr(X₂=x₂|n₂,π₂)δ(π₁,π₂|x₁,x₂,n₁,n₂,α).

If (π₁-π₂) ∈ [LB(x₁,x₂,n₁,n₂,α), UB(x₁,x₂,n₁,n₂,α)], δ(π₁,π₂|x₁,x₂,n₁,n₂,α)=1 and 0 otherwise.

The Wald-z and Wald-c confidence interval lower upper bounds for the difference between two independent proportions are defined as (See Table 1 for a typical data structure):

Figure 1

Figure 2

Table 1: Methods for calculation of confidence intervals for the difference between independent proportions.

Figure 1 shows the 95% confidence interval coverage probability function for the Wald-z and Wald-c methods as a function of π₁, π₁ ∈ [0,1] for n₁ = n₂ = 20 and p₂ = 0.3. The sawtooth appearance of the coverage functions is due to the discontinuities for values of p₁ – p₂ corresponding to any lower or upper limits in the set of confidence intervals. Like the one sample cousin, the Wald-z coverage probability curve is subnominal and less than 0.95 overall. The Wald-c coverage probability always exceeds 0.95 overall with interval widths larger than Wald-z.

Figure 3

Figure 1: Coverage probabilities for nominal 95% Wald-z and Wald-c as a function of p when p=0.3 with n=n=20.

Figure 2 shows the 95% confidence interval coverage probability function for the Wald-z and Wald-c methods as a function of π₁, π₁ ∈ [0,1] for n₁ = 20, n₂ = 10 and p₂ = 0.2. The Wald-z coverage probability curve is subnominal for differences in proportions near 0 and 1 and less than 0.95 overall.

Figure 4

Figure 2: Coverage probabilities for nominal 95% Wald-z and Wald-c as a function of p when p=0.3 with n=20, n=10.

Beal, (Beal, 1987) evaluated asymptotic methods for computing confidence interval for the differences of two independent proportions. All involved identifying the interval within which (θ - θ') ² ≤ z ² V(ψ, θ'), where θ'= p₁ — p₂, and V(ψ, θ')=u{4ψ(1 —ψ)θ = π₁(1 —π₁)/m + π₂(1 — π₂)/n . The Haldane (H) and Jeffreys-Perks (JP) method provides non-degenerative confidence intervals for all values of p₁ and p₂ unlike Wald-z or Wald-c. H and JP generally performed better than the Wald-z and Wald-c and of the two, JP was preferred (Beal, 1987, Radhakrishna, et al, 1992). The Haldane and Jeffreys-Perks lower and upper limits are defined by:

Figure 5

Newcombe compared eleven methods for estimating the difference between independent proportions (Newcombe, 1998). Similar to the single proportion, the virtues of Wald-z and Wald-c methods are in their simplicity but overshoot and inappropriate intervals are still common. The Haldane and Jeffreys-Perks methods attempt to overcome the overshoot and inappropriate intervals while maintaining closed-form tractability. Newcombe concluded that both H and JP were improvements over the Wald-z and Wald-c methods, but both were still inadequate. Newcombe recommended a hybrid method based on Wilson's score method for a single proportion without continuity correction (NS). The LB and UB for the NS method are:

Figure 6

l₁, l₂, u₁, u₂ are the lower and upper bounds for the two proportions p₁ and p₂ using Wilson's score method.

Agresti and Coull's adjustment to the Wald method for a single proportion adds t/2 successes and t/2 failures (Agresti and Coull, 1998). Agresti and Caffo later suggest that by adding two successes and two failures (total) to the two-sample method would improve the simple Wald method (Agresti and Caffo, 2000). This is an adjustment that adds one pseudo observation of each type to each sample. For instance, for sample i, p _i = (r _i +1)/(n _i +2).

For p₁ - p₂, Carlin and Lewis (Carlin and Lewis, 1996) considered independent uniform prior distributions for p₁ and p₂. The posterior distribution of p_i is β with mean of (x_i + 1)/(n_i + 2) and variance p_i (1 - p_i)/(n_i + 3). Then, a crude normal approximation for the distribution of the difference of the posterior β variates leads to a closed form confidence interval.

Figure 7

Zou and Donner (Zou and Donner, 2004) proposed using Fisher's z transformation to the estimator Δ', where Δ' = p₁ - p₂. By letting σ' = √ (ac/m ³ + bd/n ³ ), the resulting interval for Δ' is given by:

Figure 8

where l and u are given by: ½ log[(1 + Δ')/(1- Δ')] ± z_α/2 σ'/(1 - Δ' ² )

Results

Figure 3 shows the 95% confidence interval coverage probability function for the Newcombe NS, Haldane, Jeffreys-Perks, and Agresti-Caffo, Carlin-Lewis and Fisher's-Z methods as a function of π₁, π₁ ∈ [0,1] for n₁ = n₂ = 20 and p₂ = 0.3. The NS, Agresti-Caffo and Fisher's-Z methods demonstrate coverage probabilities that are near nominal over π₁ ∈ [0, 1]. Table 2 details the corresponding minimum, maximum, mean, and standard deviation for the coverage probability of these confidence interval methods. The Newcombe NS, Fisher's-Z, Agresti-Caffo and Jeffreys-Perks methods are near nominal while the Wald-z is subnominal.

Figure 9

Figure 3: Coverage probabilities for nominal 95% Newcombe NS, Haldane, Jeffreys-Perks, and Agresti-Caffo as a function of p when p=0.3 with n=n=20.

Figure 10

Figure 11

Figure 12

Table 2: Mean Coverage probabilities for nominal 95% Wald-z and Wald-c as a function of p when p=0.3 with n=n=20.

Figure 4 shows the 95% confidence interval coverage probability function for the Newcombe NS, Haldane, Jeffreys-Perks, Agresti-Caffo, Carlin-Lewis and Fisher's Z methods as a function of π₁, π₁ ∈ [0,1] for n₁ = 20, n₂ = 10 and p₂ = 0.2. In the unequal sample size situation, Newcombe NS and Agresti-Caffo coverage probability functions are near nominal over π₁ ∈ [0, 1]. The other methods are subnominal. Table 3 details the corresponding minimum, maximum, mean, and standard deviation for the coverage probability of these confidence interval methods. The Newcombe NS and Agresti-Caffo are nominal. Fisher's-z and the Jeffreys-Perks are equivalent while the Wald-z and Carlin-Lewis methods are subnominal.

Figure 13

Figure 4: Coverage probabilities for nominal 95% Newcombe NS, Haldane, Jeffreys-Perks, and Agresti-Caffo as a function of p when p=0.3 with n=20, n=10.

Figure 14

Figure 15

Figure 16

Table 3: Mean coverage probabilities for nominal 95% Wald-z and Wald-c as a function of p when p=0.3 with n=20, n=10.

Discussion

The popularity of the Wald-z closed form confidence interval is based in part because of its overwhelming use in elementary statistics textbooks. The similarity of the computational formula to the confidence interval for the difference between two sample means is simplicity in action. Rarely do these elementary textbooks warn of the inadequacy of the Wald-z, Wald-c or even of a Wald-t. In the case of differences between two independent proportions the Wald-z confidence interval behaves so poorly with coverage probabilities below nominal values that it would seem that authors of these texts would at least consider other closed form methods which are better. Considering the coverage probability criterion, three alternative methods demonstrate superior coverage properties and both are easily programmable. Our recommendation is in the equal n or nearly equal n case, use the Newcombe NS, Agresti-Caffo or Fisher's-Z methods. In the unequal n situation, use either the Newcombe NS or Agresti-Caffo methods.