Stat 4202 Notes (PDF)

Statistics 4202 Lecture Notes
Professor Chang
SP17

February 27th, 2017
Section 12.6

Likelihood Ratio Tests

Recall that NP lemma gave most powerful test when both the null and alternative hypotheses were simple. However, often one or both will be composite.
In this case, it is common to construct a test based on the ratio of likelihoods,
which generally has good properties, though may not be Maximum Power.

Let X1 ...X2 be a RS of size n from a density f (x; θ).
Note: for testing simple v simple case, H0 : θ = θ0 vs
Example Let X1 , ..., Xn be a RS from a normal population with mean µ and
known variance σ 2 . Find the critical region f LRT for testing H0 : µ = µ0 vs
H0 : µ 6= µ0 .
Soln: ω = {µ0 }, MLE: µ
˜ = µ0
ˆ
Ω=Rµ
ˆ = µ0 .

Λ=

1
( √2πσ
)n exp(− 2σ1 2 Σ(xi − µ0 )2 )
n
maxL0
=
= · · · = exp(− 2 Σ(X − µ0 )2 )
1
maxL
2σ
)n exp(− 2σ1 2 Σ(xi − X)2 )
( √2πσ

LRT: reject H0 when Λ ≤ k
⇔ − 2σn2 Σ(X − µ0 )2 ≤ lnk
p
˜
⇔ (X − µ0 )2 ≥ − 2σn2 lnk ⇔ |X − µ0 | ≥ − 2σn2 lnk ≡ K

1

˜ is determined so that the critical region has size α.
K
˜
Q: How to find K
2

X ∼ N (µ0 , σn ) (Since we want P (Type 1 Error) = α which is under H0 .
˜ =α
⇒ P (|X − µ0 | ≥ K)
0
⇒ P (| X−µ
|≥
σ
√
n

˜
K

σ
√
n

)=α

0
Since P (| X−µ
| ≥ Zα/2 ) = α
σ
√
n

⇒ let

˜
K

σ
√
n

˜ =
= Z α2 ⇒ K

√σ Zα/2
n

LT: Reject H0 when |X − µ0 | ≥ Zα/2 √σn ( a typical example of a two sided
test)

Q In this example, the RS is from N (µ, σ 2 ) so that we can get the distribution of X. What if the distribution of the population is not normal?

Thm 12.8 (Wilkes Thm) For large n, the distribution of −2ln(Λ) approaches the χ21 distribution under very general conditions. (You can assume
these conditions are satisfied during the exam)
LRT: Reject H0 if −2ln(Λ) ≥ χ2α,1 , where alpha is the area to the right. Eric Yanchenko

Check with our example:
0 2
−2ln(Λ) = −2(− 2σn2 (X − µ0 )2 ) = − σn2 (X − µ0 )2 ) = ( X−µ
) (the denominator
σ
√
n

is N(0,1) and the whole term including the square is χ21 .

Problem 12.22: Let Xi ’s be iid N (µ, σ 2 ). Both mu and σ 2 are unknown.
FInd LRT of H0 : µ = µ0 vs H1 : µ =
6 µ0 with level α.
Soln: Ω = R × {σ 2 : σ 2 > 0}
ω = {µ0 } × {σ 2 : σ 2 > 0}
(1) Find MLE’s:

2

ˆ = X, σ
ˆ
Under Ω, µ
ˆ
ˆ 2 = n1 Σ(xi − X)2
Under ω, µ
ˆ = µ0 , σ
ˆ 2 = n1 Σ(xi − µ0 )2

Λ = uj78 =

( σˆ1 )n exp(− σˆ12 Σ(xi − µ0 )2 )
( σ1ˆˆ )n exp(− σˆˆ12 Σ(xi − X)2 )


≤k⇔

σ
ˆ2
σ
ˆˆ 2


≤k

Σ(xi − µ0 )2 −n/2
σˆ2 −n/2
]
)
=[
ˆ
Σ(xi − X)2
σˆ2
ˆ2
⇐ the numerator of ( σˆˆ )−n/2 = Σ(xi − µ0 )2 = Σ(xi − X + X − µ0 )2 =
And (

σ2

Σ(xi − X)2 + 2Σ(((xi − X)(X − µ0 )) + Σ(X − µ0 )2 = Σ(xi − X)2 + n(X − µ0 )2
Σ(xi − X)2 + n(X − µ0 )2 −n/2
n(X − µ0 )2
⇐[
]
⇔
≥ k −2/n − 1
Σ(xi − X)2
Σ(xi − X)2
X − µ0 )2
⇔
≥ (k −2/n − 1)(n − 1) By definition of S 2 .
S2
⇔

n
X−µ
√0
S/ n

˜
≥ K.

˜ = t α ,n−1 √S . We reject H0 if |X − µ0 | ≥ t α ,n−1 √S .
So K
n
n
2
2

March 1st, 2017
Recall: Likelihood Ratio Test LRT
Consider H0 : θ ∈ Ω vs H1 : θ ∈
/ Ω. Let X1 , ..., Xn iid f (x; θ).
0
LRT: reject H0 if Λ = maxL
≤ k for some k ∈ (0, 1).
maxL
Qn
ˆ θ:
ˆ MLE of θ over ω.
where maxL0 = maxθ∈ω i=1 f (xi ; θ),
Qn
ˆˆ ˆˆ
maxL0 = maxθ∈Ω i=1 f (xi ; θ),
θ: MLE of θ over Ω.

Problem 12.20 : Let X: # of successes in n trials of Bin(n; θ).
Find LRT of H0 : θ = 21 vs HA : θ 6= 21
ˆ
Soln: Ω = (0, 1), ω = {1/2} ⇒ θˆ = 1/2, θˆ = X/n.
⇒ maxL0 = ( nx )( 12 )x (1 − 1/2)n−x = ( nx )(1/2)n
⇒ maxL = ( nx )( X2 )x (1 − X/2)n−x
Λ=

maxL0
maxL

n
(n
x )(1/2)
X x
n−x ≤ k
)(
)
(1−X/2)
(n
x
2
X
− Xln( n − (n − X)ln(1

=

⇔ nln(1/2)
−X
n ) ≤ lnk
⇔ X(lnX − lnn) + (n − X)(ln(n − X) − lnn) ≥ −lnk + nln(1/2)
⇔ XlnX + (n − X)ln(n − X) − Xlnn − (n − X)lnn ≥ −lnk + nln(1/2) where
−Xlnn − (n − X)lnn = −nlnn
⇔ XlnX + (n − X)ln(n − X) ≥ nlnn − lnk + nln(1/2) ≡ k˜
3

Consider g(x) = xlnx + (n − x)ln(n − x), define 0ln(0) = 0.
g(x) ≥ k˜
⇔ X ≥ k˜ + n THERE IS MORE HERE
2

Chapter 13: Testing Hypotheses for Means, Variance, and Proportions In the previous chapter, we developed general methods for finding tests.
In this chapter, we examine specific tests for parameters that are commonly of
interest.

Section 13.1 and 13.2
ples.

I’ll introduce terminology by working through exam-

Example Consider X1 , ..., Xn iid N (µ, σ 2 ) , with σ 2 known.
Suppose that we want to test H0 : µ = µ0 vs HA : µ 6= µ0 .
We showed that LRT: Reject H0 if |X − µ0 | ≥ Zα/2 √σn
or equivalently:
0
Reject H0 if |Z| = | X−µ
| ≥ Zα/2
σ
√
n

Intuition: Reject H0 if the distance between X and µ0 is large enough.
H1 : µ 6= µ0
This is a ”two-sided alternative” and leads to a ”two-sided” test (two-tailed)
We could also formulate a one-sided alternative.
H0 : µ = µ0 vs HA : µ > µ0
0
Then our test is: Reject H0 if X−µ
≥ Zα/2 (for critical region of size α
σ
√
n

⇔ Xµ0 + Zα/2 √σn This gives a ”one-sided” test (one tailed).

Ex.
Consider a RS of 100 bags of dog food and we find that the average
weight is 0.955 lbs. Suppose that we know σ = 0.17lbs
Test H0 : µ = 1.0lbs vs HA : µ 6= 1.0lbs at level α = 0.05.
Our test is to reject H0 if |Z| = | X−1
0.17 | ≥ Zα/2 = 1.96
√
100

X = 0.955, |Z| = | 0.955−1
| = | − 2.65| = 2.65
0.17
√
100

Since |Z| = 2.65 ≥ 1.96, we reject H0 .
An alternative way to make the decision is to compute a p-value.
4

Idea: Is a z-value of -2.65 unusual when H0 is true?
p-value = P (Z ≤ −2.65 or Z ≥ 2.65) = P (Z ≤ −2.65) + P (Z ≥ 2.65) =
0.004 + 0.004 = 0.008
So the p-value is the smallest value of α for which we may still reject H0 . (Informal definition).

P-Value = Probability of observing something as extreme as or more extreme than what we observe, assuming that the null hypothesis is true.
(”extreme” is defined in the direction of alternative hypothesis)

Recitation: March 3rd 2017
Likelihood Ratio Test
Defn: ω and ω c subsets of Ω (the overall parameter space). We can define the
Likelihood Ratio Test statistic as:
Λ=

maxθ∈ω L(θ)
maxθ∈Ω L(θ)

where the critical region λ ≤ k, k ∈ [0, 1].
H0 : θ ∈ ω(H0 : θ = 0).
H1 : θ ∈ Ω(H1 : θ 6= 0).

Ex.
Let X1 , ..., Xn N (θ, 1) iid.
Ω=R
ω = R \ {θ0 }
ω c = {θ0 }
H0 : θ = θ 0 .
H1 : θ 6= θ0 .
MLE of θ : θˆM LE = X
f (x) = (2π)−1/2 exp(− 12 (x − θ)2 )
Qn
L(θ) = i=1 (2π)−1/2 exp(− 12 (xi − θ)2
= (2π)−n/2 exp(− 21 Σ(xi − θ)2 )
(2π)−n/2 exp(− 21 Σ(xi − θ0 )2 )
maxθ∈ω L(θ)
L(θ)
⇔λ=
=
=
maxθ∈Ω L(θ)
L(X
(2π)−n/2 exp(− 21 Σ(xi − X)2 )
1
2
= exp(− 2 Σ(xi − θ0 ) ) − Σ(xi − X)2 ))
2

2

= exp(− 12 (Σx2i − 2nXθ0 + nθ02 − Σx2i + 2nX − nX ))
2

= exp(− 12 (nθ02 − 2nXθ0 + nX )) = exp(− n2 (x − θ0 )2 )
⇔ λ = exp(− n2 (x − θ0 )2 ) ≤ k
⇔ ln(exp(− n2 (x − θ0 )2 )) ≤ lnk
5

⇔ − n2 (x − θ0 )2 ≤ lnk
⇔ (x − θ0 )2 ≥ −2lnk
q n
⇔ |x − θ0 | ≥ −2lnk
n
⇔ X tilde N (θ, n1 )
q
−2lnk √
0|
⇔ |x−θ
≥
n
1
√
n
n

⇔

|x−θ0 |
√1
n

≥ Zα/2

⇔ |x − θ0 | ≥ Zα/2 √1n

Lecture March 3rd 2017
P-Value: Probability of observing something as extreme as or more extreme
than what we observed, assuming null hypothesis is true.

Ex.
H0 : µ = µ0 vs HA : µ 6= µ0 . In this case, closer to |mu0 is less extreme,
father away from µ0 is more extreme.

Ex.
Let the null be that µ is even. The most extreme is the odd numbers,
less extreme as we approach even numbers.

NOTE:
true!

The p-value does not give you the probability of a hypothesis being

Interpretation:
If p = p-value, when H0 is true, there is a prob.=p of observing the test statistic as extreme or more extreme than the one we obtain
from the sample.
So if the pvalue ≤ α, we reject H0 in favor of H1 .
i.e. P-value being small makes H0 a poor explanation of the data.
In general, we will select a cut-off value for the p-value ahead of time.
And this cut-off value is the α-value or α-level of the test!

Q: What about testing H0 : µ = 1.0 vs HA : µ < 1?
For the previous example with dog food, the p value is = P (Z ≤ −2.65) = 0.004.
We get more extreme as we approach −∞ from 1.
pvalue = P ( more extreme or as extreme as— H0 true) = P (X ≤ 0.955) =
P (Z ≤ −2.65) = 0.004.
During an exam, our interpretation would be: ”If H0 is true, then probability
6

of observing X as extreme or more extreme than what we observed is 0.4%”
If α = 0.05, we reject H0 since p-value = 0.004 < α = 0.05.
Ex.
The lifetime in hours of a 75-watt bulb is known to be approximately
normally distributed with std. dev. σ = 45hrs.
A RS of 20 bulbs has a mean lifetime X = 1014hrs.
Is this evidence to support the claim that mean bulb lifetime exceeds 1000
hours? (use α = 0.05)
Soln: H0 : µ = 1000 vs HA : µ > 1000.
Remember! We always want to use something simple for H0 !
Z=

X − µ0
√σ
n

=

1014 − 1000
√45
20

= 1.39

⇔ pvalue = P (Z ≥ 1.39). This is because as X, the observation, becomes
larger, the evidence becomes more extreme, which is what we want the p-value
to quantify, so we take the probability that Z is greater than 1.39 since as Z
becomes greater than this value, data becomes more extreme.
⇔ pvalue = P (Z ≥ 1.39) = 0.0823 > α = 0.05 ⇒ we fail to reject the null
hypothesis, i.e. the data is not extreme enough!
P-Value interpretation for this example: If the mean lifetime is 1000 hours
(i.e. H0 is true) then the probability of observing the sample mean of 1014
hours or larger than 1014 hours is about 8%. It thus seems possible that the
mean is 1000 hours given the level α = 0.05. We fail to reject H0 and conclude
that the data are consistent with a mean lifetime of 1000 hours.

So far, we’ve considered the case where we assumed a normal distribution
and σ 2 was known. What is σ 2 is unknown?
• If n is large, we use the procedure with S 2 substituted for σ 2 (even if the
distribution is not normal by CLT).
• If n is small and the data are normally distributed, we use t-distribution.

Four Steps for Tests of Hypotheses:
1. Formula H0 and HA , and specify α.
2. (method 1): Using the sampling distribution of an appropriate test statistic, determine a critical region of size α.
(method 2): Specify the test statistic.
7

3. Determine the value of the test statistic (and corresponding p-value) from
the sample data. (for method 2)
4. (method 1) check whether the value of the test statistic falls into the critical region and accordingly, reject the null hypothesis or reserve judgment.
(method 2) check whether the given p-value is less than or equal to α
and accordingly, reject the null hypothesis or reserve judgment.

8