MTH3260 Exam Sheet (PDF)

Sigma Algebra on Ω: Σ ⊆ 2Ω
1. ∅ ∈ Σ
2. A ∈ Σ =⇒ Ω \ A ∈ Σ
∞
[
3. A1 , · · · ∈ Σ =⇒
Ai ∈ Σ
i=1

Topology (Ω, τ )
1. ∅, Ω ∈ τ
[
2. X ⊆ τ =⇒
A∈τ
A∈X

3. A1 , . . . , An ∈ τ =⇒

n
\

Ai ∈ τ

i=1

Probability Measure P : Σ → [0, 1]
1. P(∅)
=1
 [= 0,P(Ω)
X
2. P
Ai =
P(Ai )
i∈I

i∈I

Conditional Probability
P(A, B) = P(A | B)P(B)
Independence (A1 , . . . , An )
P(A1 , . . . , An ) = P(A1 ) . . . P(An )
f (x1 , . . . , xn ) = f1 (x1 ) . . . fn (xn )
E[X1 . . . Xn ] = E[X1 ] . . . E[Xn ]
ExpectationZ
E[h(X)] =

Confidence Interval (α significance)
PΘ (a(X) 6 φ 6 b(X)) = 1 − α
Centred
α
PΘ (φ 6 a(X)) = PΘ (φ > b(X)) =
2

θ∈Θ1

where γ is chosen such that φC has
size α.

X1 , .h. . , Xn ∼ N (µ, σ 2 ) (100(1 −
i α)%)
λ Test (H0 : θ = θ0 vs H1 : θ ∈ Θ1 )
µ ∈ X − zα/2 √σn , X + zα/2 √σn
Determine θbM LE (x) ∈ Θ1 .
C = {x ∈ Ω :
Critical Region C ⊂ R
b
Lx (θ(x))/L
P(T ∈ C | φ = φ0 ) = α
x (θ0 ) > 1/λ}
where λ is chosen such that φC has
Composite Alternative Hypothesis
size α.
sup P(T ∈ C | φ) = α
φ∈Φ0

Wald’s Decision Theory
State Space (Ω, Σ)
Statistical Model (Ω, Σ, P)
Decision Space (D, E)
Decision Rules R 3 d : Σ → E
P = {Pθ : θ ∈ Θ}
Loss, Risk Functions
R(θ, d) = E[L(θ,
d(X))]
Z
=

L(θ, d(x)) dPθ (x)
Ω

Preferable Decision Functions d1 ≺ d2
R(θ, d1 ) 6 R(θ, d2 ) ∀(θ ∈ Θ)

∞

γ Test[
(H0 : θ = θ0 vs H1 : θ ∈ Θ1 )
{x ∈ Ω : fθ1 (x)/fθ0 (x) > γ}
C=

h(x)fX (x) dx

Karlin-Rubin Theorem
Θ ⊂ R, Θ0 = {θ ∈ Θ : θ 6 θ0 }
Θ1 = {θ ∈ Θ : θ > θ0 }
Let φC be a deterministic test with
C = {x ∈ Ω : U (x) > k} for a k ∈ R
α = Eθ0 (φC )
of all tests of size 6 α, φC is the most
powerful
Wilks’ Theorem
Suppose X = (X1 , . . . , Xn )
with Θ0 ⊂ Θ and |Θ| − |Θ0 | = p
Then 2 log LX (H0 , H1 ) ∼ χ2p
as n → ∞

−∞

Optimality
d ≺ r ∀(r ∈ R)
Admissibility
@(r ∈ R) : r ≺ d ∧ d 6≺ r
Transformation
Unbias ∀(θ, θ0 ∈ Θ)
−1
(Y1 , . . . , Yn ) = ν (X1 , . . . , Xn )
Eθ [L(θ, d(X))] 6 Eθ [L(θ0 , d(X))]


g(y) = J(y1 , . . . , yn ) f (ν(y1 , . . . , yn ))
<∞
 dν 
⇐⇒
E
[d(X)]
= µθ
θ
g(y) =   f (ν(y))
dy
Hypothesis Testing
cov(X, Y ) = E[XY ] − µX µY
Null Hypothesis H0 : θ ∈ Θ0
cov(X, Y )
p
Alternative Hypothesis H1 : θ ∈ Θ1
corr(X, Y ) = p
var(X) var(Y )
Memorylessness
P(X > t + s | X > t) = P(X > s)

Estimator Bias
(unbiased) E[T ] = θ
X1 , X2 , . . .
Weak Law of Large Numbers
lim P(|Sn /n − µ| > ε) = 0
n→∞
Strong Law of Large Numbers
P(Sn /n → µ) = 1
Central
Limit Theorem
√
P( n(Sn /m − µ) 6 x) → P(σZ 6 x)
Maximum Likelihood Estimation
n
Y
L(Θ; x1 , . . . , xn ) =
f (xi | Θ)
i=1

b = arg max L(Θ; x1 , . . . , xn )
Θ
Θ

= arg max log L(Θ; x1 , . . . , xn )
Θ

Method of Moments
Mk (Θ) = E[X k ], Solve
n
1X k
Mk (Θ) =
X
n i=1 i

Observation x ∈ Ω
Decision Space D = {0, 1}
Decision Rule φ(x) : Ω → D
Size α = αφ = sup Eθ (φ)
θ∈Θ0

Power βφ : θ 7→ Eθ [φ]
Hypothesis Likelihood
Lx (H) = sup f (x | θ)
θ∈Θ

Likelihood Ratio
Lx (H1 )
Lx (H0 , H1 ) =
Lx (H0 )
sup Eθ (φ) = α = sup Pθ (C)
θ∈Θ0

θ∈Θ0

Neyman-Pearson Lemma
Θ0 = {θ0 }, Θ1 = {θ1 }. φ is a
likelihood ratio with critical region
C = {x ∈ Ω : Lx (H0 , H1 ) > k}
Of all tests of size 6 α, φ is the most
powerful.

Sufficient Statistics
Factorization Theorem
Suppose Pθ has density f (x | θ)
∀θ ∈ Θ
Statistic T is sufficient for θ iff
f (x | θ) = g(T (x), θ)h(x)
for some functions g : R × Θ → R,
h:Ω→R
Rao-Blackwell Theorem
Let S estimate θ such that
Eθ [S] < ∞ ∀(θ ∈ Θ). Let T be a
sufficient statistic for θ.
Then Sb = E[S | T ] is preferable to S
wrt quadratic risk and is unbiased.
Fisher Information


∂ log Lx
I(θ) = varθ
∂θ


∂ log Lx 2
= Eθ
∂θ
Cramer-Rao Inequality
T : Rn → R estimates g(θ) without
[g 0 (θ)]2
bias. varθ (T ) >
I(θ)
Efficiency e(T ) =

[g 0 (θ)]2 /I(θ) ?
=1
varθ (T )

Consistency (T1 , . . . , Tn )
∀(ε > 0) lim Pθ (|Tn − g(θ)| > ε) = 0
n→∞
Strong Consistency
Pθ (Tn → g(θ)) = 1

Stationarity
E[Xh ] = constant
cov(Xh+t , Xh+s ) = cov(Xt , Xs )
Autocovariance
γt = E(X0 Xt ), t ∈ Z

Poisson Processes
Arrival Times
0 < T1 < T2 < · · · < Tn < · · ·
Arrival Count in [0, t], Nt
Interarrival Time Sn

Binomial(n,
p)
 
n k
p (1 − p)n−k
k
np
np(1 − p)

Markovianity
P(Xn+1 ∈ B | Xn , . . . , X0 ) =
P(Xn+1 ∈ B | Xn )

{Nt = n} = {Tn 6 t < Tn+1 }
{Nt > n} = {Tn 6 t}
{s < Tn 6 t} = {Ns < n 6 Nt }

Projection
n
X
ai γj−i = γn+1−j ,

|λt|n
n!
E(Nt ) = λt, var(Nt ) = λt

Poisson(λ)
λk e−λ
k!
λ
λ

P(Nt = n) = e−λt

j = 1, . . . , n

i=1

∗
E((Xn+1

− Xn+1 )Xj ) = 0

Spectral Density
1 X
γt cos λt,
f (λ) =
2π
Z π t∈Z
γt =
f (λ) cos λt dλ

λ ∈ [−π, π]

−π

Wold Decomposition
∞
X
Xt =
aj εt−j + Yt ,

t∈Z

j=0

Linearity
∞
X
Xt =
aj εt−j ,
j=1
n
X
2

γt = σ

λn tn−1 e−λt
(n − 1)!
n
n
E(Tn ) = , var(Tk ) = 2
λ
λ

fTn (t) =

t∈Z

fSn (t) = λe−λt
1
1
E(Sn ) = , var(Sn ) = 2
λ
λ

Normal(µ, σ 2 )
2
2
1
√
e−(x−µ) /(2σ )
2πσ 2
µ
σ2

Square-Integrable Continuous Time
Processes
Wiener Process (Brownian Motion)
Wt ∼ N (0, σ 2 t)
(Wt ) has independent increments
Wt−s ⊥
⊥ Ws

Exponential(λ)
λe−λx
1/λ
1/λ2

cov(Ws , Wt ) = σ 2 min(s, t)

aj aj+t

j=1

ARMA(p, q)
p
q
X
X
Xt +
φj Xt−j = εt +
θj εt−j
j=1

j=1

Φ(B)Xt = Θ(B)εt
Yule-Walker Equation
p
X
φj γk−j = γk , k = 1, 2, . . .
j=1
p
X

φj γj = γ0

j=1

Autocovariance
γk = E(Xt Xt−k )
p
q
X
X
=
φj γk−j +
θj E(εt−j Xt−k )
j=1

j=1

Spectral Density
Yt = Φ(B)Xt = Θ(B)εt

2

 2
p
X

 σ
fY (λ) = 1 −
φj eλj 

 2π
j=1
Weak Stationarity
Φ(z) 6= 0 ∀|z| 6 1
Invertibility
Θ(z) 6= 0 ∀|z| 6 1

Geometric(p)
(1 − p)k−1 p
1/p
(1 − p)/p2

Geometric Series
X
1
=
an
1−a

Chi-Squared(k)
1
xk/2−1 e−x/2
k/2
2 Γ(k/2)
k
2k

n∈N

Gamma(α, β)
β α α−1 −βx
x
e
Γ(k)
α/β
α/β 2