Introduction

The textbook binomial confidence interval that has near universal acceptance is the familiar Wald binomial method (Wald-z). This interval is easy to compute and just as easily interpreted. This confidence interval is typically presented along with a justification based on the central limit theorem. Most users believe that the larger the sample size, n, the better the normal approximation resulting in the actual coverage approaching the nominal level 1 - α. Textbook authors recognize that the actual coverage probability of Wald-z is poor for p near 0 or 1, and typically include a warning that Wald-z should only be used when n  min (p, 1 - p)  5 (or 10). This “rule” may vary from text to text, but each variation reflects the unease about the subnominal coverage when p lies close to either 0 or 1.

Inadequate coverage of the Wald-z interval can be erratic, even when p is not close to boundaries and when sample sizes are moderately large (Vollset, 1993; Santner, 1998; Agresti and Coull, 1998; and Newcombe, 1998). An interesting phenomenon occurs in the actual coverage probability, CP (| n, α), when n is fixed at 50 and p  (0, 1). In Figure 1, the oscillation of CP (| 50, 0.05) demonstrates values of p when n = 50 where the actual coverage probability is very close to or larger than the nominal level. There are also values of p where the corresponding coverage probability is considerably smaller than the nominal level. These smaller CP (| n, α) may occur at values of p that are close to successful values of p.

Figure 1

Figure 1. Oscillation phenomenon of the Wald-z CP (| , α) for n = 50, α = 0.05 and p = 0.080, . . . , 0.920.

Are there better alternative binomial confidence interval construction methods to Wald-z? Alternative methods include the “gold standard' Clopper-Pearson method (Clopper and Pearson, 1937), Wilson's Score, Wilson's Score with a continuity correction (Wilson, 1927), and adjusted Wald methods (Santner, 1998; Agresti and Coull, 1998; Borkowf, 2005). All were initially evaluated in terms of their overall coverage probability for   [0, 1] for a fixed n. We intended to show the defective nature of the accepted “rules” demonstrating that there are indeed “unlucky n's” by examining the behavior of the CP (| n, α) for np  5 for a set of alternative methods.

Numerical Methods

When constructing a confidence interval we would like the actual coverage probability to be close to the nominal confidence interval. However, because of the discrete nature of the binomial distribution, this is not always possible (Agresti 1998, Newcombe 1998). We use the coverage probability, CP (| n, ), as our standard measure of binomial confidence interval performance (Agresti and Coull, 1998; Newcombe, 1998; Borkowf, 2005). Given a random variable X, X = k, n, and , let ( | k, n, ) = 1 when   [LB(k, n, ), UB(k, n, )], and ( | k, n, ) = 0 otherwise. Then, CP ( | n, ) for a given  is:

CP (| n, ) =  P(X = k| n, ) (| k, n, )

The Wald-z binomial confidence interval computational formula, with or without a continuity correction, is found in virtually every statistics text. These two methods are labeled by Santner as the z and c-intervals (Santner, 1998). Santner also identifies a t-interval method that replaces α-quantile of the standard normal with t_n-1,α - quantile of the standard t-distribution with n-1 degrees of freedom and suggested an alternative, a q-interval. The q-interval is a straightforward application of the Central Limit Theorem, does not involve any continuity correction factor, and has superior actual coverage than either the Wald-t or Wald-c methods (Santner, 1998). The lower bounds (LB) and upper bounds (UB) for these methods are:

Wald (z-Interval) LB = p  z_/2 [pq/n]

UB = p + z_/2 [pq/n]

Wald (c-Interval) LB = p  (z_/2 [pq/n]+1/(2n))

UB = p + (z_/2 [pq/n]+1/(2n))

Wald (t-Interval) LB = p  t_{n-1, /2} [pq/n]

UB = p + t_{n-1, /2} [pq/n]

Wald (q-Interval) LB = {p (n/(n+z²)) + (z²/(n+z²))/2}  [ (pq/n)² (nz²/(n+z²))² + (z⁴/(n+z²)²)/4]

UB = {p (n/(n+z²)) + (z²/(n+z²))/2} + [ (pq/n)² (nz²/(n+z²))² + (z⁴/(n+z²)²)/4]

Where: p = r/n, q = 1-p, and r is the number of successes and n is the sample size.

The Clopper-Pearson (CLP) binomial confidence interval is the best-known “exact” method and is considered by most to be the “gold standard” (Clopper and Pearson, 1934). The CLP lower and upper limits are defined by:

Clopper-Pearson CLP LB=0 if x = 0, (/2)^1/n if x = n.

LB=[1+(nr+1)/(r  F_{2r, 2(nr+ 1), 1/2})]^-1

UB=1 - (/2)^1/n if x=0, 1 if x = n.

UB=[1 + (nr)/(r  F_{2(r+1), 2(nr),/2})]^-1

Two methods attributed to Wilson (Wilson, 1927) are the Score (S) and Score with continuity correction (SC) (Wilson, 1924). The LB and UB are defined by:

Score LB=(2np+z²z{z²+4npq})/2(n+z²)

UB=(2np+z²+z{z²+4npq})/2(n+z²)

Score - C LB=[2np+z²1z{z²21/n+4p(nq+1)}]/(2n+2z²)

UB=[2np+z²+1+z{z²+21/n+4p(nq-1)}]/(2n+2z²)

Agresti and Coull proposed a straightforward adjustment - the “add 4 to Wald” that simply adds two successes and two failures and then uses the Wald-z formula (Agresti and Coull, 1998). Alternatively, one could add z²/2 successes and z²/2 failures before computing the Wald-z confidence interval. The Agresti-Coull (AC) lower and upper bounds are:

Agresti - Coull (AC) LB=p'z[p'q'/n'],

UB=p'+z[p'q'/n'], where

p' = (2r+z²)/(2n+z²), and n' = n+z²

Borkowf augments the original data with a single imaginary failure to compute the lower confidence bound and a single imaginary success to compute the upper confidence bound - a single augmentation with an imaginary failure or success (SAIFS) method (Borkowf, 2005). The lower and upper confidence bounds for this method are:

Borkowf - z LB = p₁ - _1-/2 [p₁q₁/n] and

UB = p₂ + _1-/2 [p₂q₂/n], with

p₁ = (r + 0)/(n+1) and p₂ = (r+1)/(n+1),

_1-/2 = α-quantile of the standard normal distribution

Results

Figure 2 shows the oscillation of the coverage probability of the nominal 95% Wald-z, Wald-c, Wald-t, Wald-q, CP, Score, Score-c, Agresti-Coull, and Borkowf-z methods for p = 0.1 with n  [50, 150]. The Wald-z, Walt-c, Wald-t, Wald-q and to a lesser extent the Score methods show a large variability in their coverage probability for n  [50, 150]. For a few lucky sample sizes, the actual coverage probability approaches the nominal level. At the same time, a simple addition of one to the sample size could have a disastrous consequence in the actual coverage probability. While Score method is a recommended alternative to the Wald-z, there exist values of n where the actual coverage probability is noticeably less than the nominal level. For instance, at n=100 the actual coverage probability is considerably less than the nominal level, while at n=101 the actual coverage probability exceeds the nominal level. The Clopper-Pearson, Score-c, and Borkowf-z methods demonstrate the same oscillation effect, but the actual coverage probability is at least at the nominal level for all n in the interval [50, 150].

Figure 2

Figure 2. Oscillation phenomenon of the Wald-Z, Wald-C, Wald-t, Clopper-Pearson, Score, Score-c, Agresti-Coull, Borkowf-Z CP (| , α) for  = 0.1, α = 0.05 and = 50, . . . , 150.

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

The average coverage probability for this set of methods with the minimum and maximum CP (| n, α) for n  [50, 150] is given in first column of Table 1. With the exception of a few cases (of n), the coverage probability of the {Wald-z, Wald-c, and Wald-t} do not approach the nominal level as n increases. The average coverage probabilities for this set of methods are 0.927, 0.948, and 0.930, respectively. The set of methods, {CLP, Score-c, and Borkowf-z} maintain their conservative properties with average coverage probability of 0.967, 0.968, and 0.964, respectively. The Wald-q, Score and Agresti-Coull methods fall between the inadequate performing methods and the conservative methods (0.952, 0.952 and 0.959, respectively).

Figure 11

Table 1. Mean coverage probability (minimum, maximum) of the Wald-z, Wald-c, Wald-t, Wald-q, Clopper-Pearson, Score, Score-c, Agresti-Coull, and Borkowf-Z.

The proportion of times when CP (| n, α) is greater to or equal to the nominal level is given in column 1 of Table 2. The Wald-z and Wald-t are almost always less than the nominal level and the Wald-c is slightly better. The Clopper-Pearson and Score-c are above the nominal level while the Borkowf-z is at the nominal level 95% of the time. The Wald-q, Score and Agresti-Coull methods are at the nominal level 61%, 61% and 85% respectively.

Figure 12

Table 2. Proportion of times the Wald-z, Wald-c, Wald-t, Wald-q, Clopper-Pearson, Score, Score-c, Agresti-Coull, and Borkowf-Z meet the 95% nominal coverage probability

Figure 3 shows the coverage probability of the nominal 95% Wald-z, Wald-c, Wald-t, Wald-q, CP, Score, Score-c, Agresti-Coull, and Borkowf-z methods for  = 0.3 with n  [15, 150]. The average coverage probability is given in the third column of Table 1. The average coverage probability of Wald-z and Walt-t are subnominal while the rest of the methods investigated are nominal. The CP, Score-c, and Borkowf-z methods continue to maintain their conservative properties with average coverage probability of 0.964, 0.965, and 0.961, respectively. The Wald-c, Score, and Agresti-Coull perform better than when  = 0.1.

The oscillation phenonomen continues when  = 0.3 (Figure 3). There is an increase in the number of n's where the actual coverage probability of the Wald methods meets or exceeds the nominal level. And, there is a corresponding increase as many points in the sample size continuum where adding one to the sample size severely effects the actual coverage probability. The Clopper-Pearson and Score-c have similar oscillations, but maintain at least a nominal coverage probability. The Agresti-Coull and Score methods have actual coverage probabilities that have the similar properties as any of the Wald methods - some values of n are fine, but adding one to the sample size adversely affects the actual coverage probability.

Figure 13

Figure 3. Oscillation phenomenon of the Wald-Z, Wald-C, Wald-t, Clopper-Pearson, Score, Score-c, Agresti-Coull, Borkowf-Z CP (| , α) for  = 0.3, α = 0.05 and = 15, . . . , 150.

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

The proportion of times when CP (| n, α) is greater to or equal to the nominal level is given in column 3 of Table 2. The Wald-z and Wald-t are the least efficient in achieving a nominal coverage at 16.2% and 28.7% respectively. The Wald-q, Score, Agresti-Coull, and Wald-c are slightly better than chance at meeting the nominal level (55.9%, 55.9%, 61.0%, and 78.7%, respectively). The Clopper-Pearson (100%) and Score-c (100%) are always nominal while the Borkowf-z is almost always nominal (97.8%).

The average coverage probability, minimum and maximum CP (| n, α) for n  [50, 150] are given in last column of Table 1. The oscillation phenonomen continues even with an optimal value of . The mean coverage probability of the Wald-z and Wald-t improve to 0.941 and 0.947 respectively. The Wald-c, Score, and Agresti-Coull average coverage probability are essentially nominal at 0.955, 0.950, and 0.951, respectively. The Borkowf-z, Clopper-Pearson, and Score-c demonstrate their conservative properties at 0.961, 0.963, and 0.9664.

Table 2, Column 5 lists the proportion of times that these methods achieve a nominal 95% coverage. The Wald-z and Wald-t continue to demonstrate their inadequate coverage at 30.5% and 43.3%, respectively. Clopper-Pearson and Score-c methods are again conservative followed closely by the Borkowf-z method (92.2%). The Agresti-Coull and Score methods perform slightly better than chance (53.2% and 52.5%, respectively).

Conclusion

The Score-c and Clopper-Pearson confidence interval methods have coverage probabilities that are bounded below by the nominal confidence level. These two methods may be considered to be too conservative, but both guarantee the actual coverage probability to meet or exceed the nominal level. Agresti, Coull, and Borkowf have proposed methods that have reasonable coverage probability properties and lessen the reliance on “luck.”

In forming a 95% confidence interval, it would seem better to use a method that guarantees that the actual coverage probability is at least 0.95. The three Wald type confidence intervals methods should be put on the shelf {Wald-z, Wald-c, and Wald-t}. This set of methods, even when adhering to the general warning or 'rule of thumb' do not reach the nominal coverage probability nearly enough to warrant their use.