STAT 3460 University of Toronto Intermediate Statistical Theory Questions The sample questions and notes are attached. Please carefully check out all of th

STAT 3460 University of Toronto Intermediate Statistical Theory Questions The sample questions and notes are attached. Please carefully check out all of that first before accept this work. 10 questions( it’s similar to the practice final) will be post on 4/16/2020(GMT-3 Timeline) 8:30AM, and will due in 3 hours on 11:30 AM.Please feel free to ask if you have any questions. MATH/STAT 3460, Intermediate Statistical Theory
Hong Gu
Brief of the contents and In Class Examples
March 19, 2020
1 / 147
Basic definitions
POPULATION. A set of numbers from which a sample is drawn is
referred to as a population. The distribution of the numbers
constituting a population is called the population distribution.
RANDOM SAMPLE. If X1 , X2 , …, Xn are independent and
identically distributed random variables, we say that they
constitute a random sample from the infinite population given by
their common distribution.
statistics: random variables that are functions of a set of random
variables X1 , X2 , …, Xn .
sample mean X? and sample variance S 2 .
concept of sampling distribution.
March 19, 2020
2 / 147
Point Estimation: the Method of Moments
The method of moments consists of equating the first few
moments of a population to the corresponding moments of a
sample, thus getting as many equations as are needed to solve for
the unknown parameters of the population.
The kth sample moment is defined as mk0 =
Pn
k
k =1 xi
n
.
March 19, 2020
3 / 147
Point Estimation: the Method of Maximum Likelihood
Data and probability model with parameter ?: f (x, ?).
maximum likelihood estimate of ?:
?? is the parameter that best explains the data
The likelihood function L(?; X ) and log-likelihood function l(?; X )
The score and information functions
S(?; X ) = l 0 (?; X ) =
dl(?; X )
d?
d 2 l(?; X )
d?2
The expected or Fisher information function is given by
I(?; X ) = ?l 00 (?; X ) = ?S 0 (?; X ) = ?
J(?) = E[I(?; X )] = E[?
d 2 l(?; X )
]
d?2
March 19, 2020
4 / 147
Point Estimation
Question 1
You toss a coin n (n=100) times, and get k (k=37) heads.
(a) What is the moment estimator for the probability of getting heads?
(b) What is the maximum likelihood estimator for the probability of
getting heads?
March 19, 2020
5 / 147
Point Estimation
Question 2
We purchase a bag of a new type of candy, that comes in different
colours. Of the first 10 sweets we take out of the bag, 3 are red, 2 are
blue, 2 are green, 1 is brown, 1 is purple, and 1 is white. If we assume
all colours are equally likely,
(a) What is the moment estimate for the number of colours?
(b) what is the maximum likelihood estimate for the number of colours?
March 19, 2020
6 / 147
Point Estimation
Question 3
A team of ecologists wants to know how many of a certain species of
birds lives in a forest. They perform the following experiment: They
capture a group of birds and mark them. They then release the birds,
and capture more birds a week later, and count how many of those are
marked.
Suppose the first group captured (and marked) contains 124 birds, and
the second group contains 138 birds, of which 17 are marked.
(a) What is the moment estimator for the number of birds in the forest?
(b) What is the maximum likelihood estimator for the number of birds in
the forest?
March 19, 2020
7 / 147
Point Estimation
Question 4
Let X1 , . . . , Xn be samples from a binomial distribution B(10, p).
(a) What is the moment estimator for p?
(b) What is the maximum likelihood estimator for p?
March 19, 2020
8 / 147
Point Estimation
Question 5
You are conducting a survey. One of the questions is potentially
sensitive — the answer “YES” might be embarassing. To avoid
embarrassment, you attempt one of the following schemes:
1
Ask the participant to roll a die (out of sight) and if it is 6, answer
“YES” regardless of the true answer. Otherwise answer the
question truthfully.
2
Ask the participant to roll a die (out of sight) and if it is 6, give the
opposite of the true answer. Otherwise answer the question
truthfully.
What are the moment estimate and the maximum likelihood estimate
for the number of people who should really answer “YES” in each
case?
March 19, 2020
9 / 147
Point Estimation
Question 6
The lifetime of a light-bulb is thought to be exponentially distributed
with parameter ?. 1000 light bulbs are left on for 24 hours. Within that
time 8 of them break. What are the moment estimate and the
maximum likelihood estimate for ??
March 19, 2020
10 / 147
Combining Independent Events
Experiment 1 gives data E1 with P(E1 ; ?); Experiment 2 gives data
E2 with P(E2 ; ?)
Two experiments are independent
The joint probability is P(E1 , E2 ; ?) = P(E1 ; ?)P(E2 ; ?)
The combined log-likelihood function L(?) = c 0 L1 (?)L2 (?)
The log-likelihood function is L(?) = l1 (?) + l2 (?)
The score function is S(?) = S1 (?) + S2 (?)
The information function is I(?) = I1 (?) + I2 (?)
March 19, 2020
11 / 147
Combining Independent Events
Question 7
The lifetime of a light-bulb is thought to be exponentially distributed
with parameter ?. 1000 light bulbs are left on for 24 hours. Within that
time 8 of them break. Another set of 500 light bulbs are left on for 72
hours. Within that time 14 of them break. What are the moment
estimate and maximum likelihood estimate for ? from the combined
data?
March 19, 2020
12 / 147
R code used to solve the equation
install.packages(“animation”)
library(animation)
obj=newton.method(FUN=function(x) x^3+2*x-2.956,
init=1.1, rg=c(0.9,1.1), tol=0.0001)
obj$root
obj=newton.method(FUN=function(x) 60000*x^3+192*x^2
+192*x-58800, init=1.1, rg=c(0.9,1.1), tol=0.0001)
obj$root
March 19, 2020
13 / 147
Combining Independent Events
Question 8
Two people are conducting a survey with a potentially sensitive
question. One of them uses technique 1 and gets 43 “YES” answers
out of 200. The other uses technique 2 and gets 20 “YES” answers out
of 100. What are their individual MLEs and what is the combined
MLE? What is the combined moment estimate? (Refer the two
techniques to Question 5.)
March 19, 2020
14 / 147
Likelihood for Continuous Models
X continuous R.V. has pdf f (x) and cdf F (x).
P(E; ?) = ?ni=1 P(ai < X ? bi ) = ?ni=1 [F (bi ) ? F (ai )] L(?) = [?ni=1 ?i ]?ni=1 f (xi ) = c?ni=1 f (xi ) March 19, 2020 15 / 147 Likelihood for Continuous Models Question 9 Let X1 , . . . , Xn be uniformly distributed on the interval [0, a] for some unknown a. (a) What is the moment estimator for a? (b) What is the maximum likelihood estimate for a? March 19, 2020 16 / 147 Likelihood for Continuous Models Question 10 Let X1 , . . . , Xn be normally distributed with mean µ and variance 1. What is the maximum likelihood estimate for µ? March 19, 2020 17 / 147 Likelihood for Continuous Models Question 11 Let X1 , . . . , Xn be normally distributed with mean 0 and variance ? 2 . (a) What is the moment estimator for ?? (b) What is the maximum likelihood estimate for ?? March 19, 2020 18 / 147 Likelihood for Continuous Models Question 12 Let X1 , . . . , Xn be exponentially distributed with parameter ?. (a) What is the moment estimator for ?? (b) What is the maximum likelihood estimate for ?? March 19, 2020 19 / 147 Censoring in Lifetime Experiments Suppose that m specimens fail and n ? m do not, we have m failure times x1 , x2 , . . . , xm and n ? m censoring times T1 , T2 , . . . , Tn?m , then the likelihood function is proportional to n?m [?m i=1 f (xi )?i ] ?j=1 [1 ? F (Tj )] n?m L(?) = c ?m i=1 f (xi ) ?j=1 [1 ? F (Tj )] March 19, 2020 20 / 147 Censoring in Lifetime Experiments Question 13 Let X1 , . . . , Xn be exponentially distributed with parameter ?. However, the values are censored above 1, so for any Xi > 1, we do not know
the value of Xi , only that it is at least 1. What is the maximum
likelihood estimate of ?? what is the moment estimate of ??
March 19, 2020
21 / 147
Censoring in Lifetime Experiments
Question 14
A company is interested in how frequently customers visit its website.
When a customer visits the website, it leaves a unique cookie in the
user’s browser. When the user returns, it records the time since the
cookie was issued. It records the following times (in days)
Days
Frequency Returned
Frequency Censored
Days
Frequency Returned
Frequency Censored
1
10
8
8
33
27
2
25
15
9
29
29
3
31
18
10
18
36
4
45
30
11
21
35
5
53
33
12
13
31
6
49
34
13
8
38
7
51
31
14
6
29
Assume that the number of days until a customer returns has a
geometric distribution with probability p. What is the maximum
likelihood estimate for p?
March 19, 2020
22 / 147
Invariance
Suppose that ? = g(?), where g is invertible.
P(E; ?) = P(E; g(?));
L(?) = L? (?)
The MLE: ?? = g(??)
This means that if we know MLE of ? then we know MLE of any
one-to-one function of ?. It is one reason why MLE is widely used.
March 19, 2020
23 / 147
Invariance
Question 15
Under a certain model of evolution, for two species with phylogenetic
distance t between them, the probability that a given nucleotides will
4
be the same is 14 + 34 e? 3 t . If for two given species, and a given gene,
there are 532 nucleotides, of which, 346 are the same between the two
species.
What is the maximum likelihood estimate of the phylogenetic distance
between them?
March 19, 2020
24 / 147
Invariance
Question 16
The lifetime of a light-bulb is thought to be exponentially distributed
with parameter ?. 1000 light bulbs are left on for 24 hours. Within that
time 8 of them break. What is the maximum likelihood estimate for the
mean lifetime of a lightbulb?
March 19, 2020
25 / 147
Invariance
Question 17
Suppose the number of car accidents on a typical day has a Poisson
distribution with mean ?. The average time between car accidents is ?1 .
Suppose we observe 234 car accidents over a period of 44 typical
days. Calculate the MLE for the average time between car accidents.
March 19, 2020
26 / 147
Newton’s Method
To solve equation g(?) = 0, we start at ?0 , using Taylor’s
expansion g(?) ? g(?0 ) + (? ? ?0 )g 0 (?0 ).
iteratively using ?n+1 = ?n ? g(?n )/g 0 (?n ) until convergence.
This can be used to find MLE by solving S(?) = 0.
The estimate can be updated by ?n+1 = ?n + s(?n )/I(?n ).
when I(?n ) is repalced by J(?n ), the method is called Fisher’s
method of scoring.
Good initial value is necessary. (The moment estimates are often
used as starting values)
It is difficult to use Newton’s Method if maximum occurs on or near
a boundary.
March 19, 2020
27 / 147
Newton’s Method
Question 18
Let X1 , X2 , X3 be independent samples whose distribution is that of a
sum of two independent exponential distributions with parameters ?
and 2?. That is,
fXi (x) = 2?(e??x ? e?2?x )
Find the maximum likelihood estimate for ? if X1 = 2, X2 = 3.1 and
X3 = 1.5.
March 19, 2020
28 / 147
Two-Parameter Maximum Likelihood Estimation
The likelihood is L(?, ?) = cP(E; ?, ?).
The MLE (??, ??) maximizes L(?, ?) and l(?, ?).
?l ?l T
, ?? ) .
The score function is S(?, ?) = ( ??
The information matrix I(?, ?) is a 2 × 2 matrix. I(??, ??) > 0
(positive definite).
Likelihoods are invariant under 1-1 parameter transformations.
Newton’s method: (?1 , ?1 )T = (?0 , ?0 )T + I(?0 , ?0 )?1 S(?0 , ?0 ),
iterate until convergence.
March 19, 2020
29 / 147
Two-Parameter Maximum Likelihood Estimation
Question 19
Let X1 , . . . , Xn be independant samples from a normal distribution with
mean µ and variance ? 2 , where µ and ? are unknown. What are the
moment estimator and the maximum likelihood estimator for the pair
(µ, ?)?
March 19, 2020
30 / 147
Two-Parameter Maximum Likelihood Estimation
Question 20
Let X1 = 4, X2 = 7, X3 = 5 be independant samples from a binomial
distribution, with parameters n and p. What is the maximum likelihood
estimate of n and p?
March 19, 2020
31 / 147
Two-Parameter Maximum Likelihood Estimation
Question 21
We roll a die 100 times, and get 18 sixes. In another experiment, we
roll the same die 100 times and get 19 fives. What is the maximum
likelihood estimate for the probabilities of getting a six, and getting a
five?
March 19, 2020
32 / 147
Two-Parameter Maximum Likelihood Estimation
Question 22
Suppose that Y1 , Y2 and Y3 are independent Poisson variates with
means µ1 , µ2 and µ1 + µ2 , respectively. Derive formulae for the
maximum likelihood estimates (µ?1 , µ?2 ) based on nonzero observed
values y1 , y2 , y3 .
March 19, 2020
33 / 147
Properties of point estimation: Unbiased Estimators
Unbiased Estimator: A statistic ?? is an unbiased estimator of the
parameter ? of a given distribution if and only if E(??) = ? for all
possible values of ?.
Bias: bn (?) = E(??) ? ?
Asymptotically unbiased estimator: ?? is an asymptotically
unbiased estimator of ? if and only if
limn?? bn (?) = 0.
March 19, 2020
34 / 147
Unbiased Estimators
Question 23
A collection of light bulbs have lifetime following an exponential
distribution with parameter ?. We leave 1000 light bulbs on until 10
have failed. Let ?? be the maximum likelihood estimate of ?, based on
these data.
(a) Calculate the bias of ?? as an estimator for ?.
(b) By invariance, ??1 is the maximum likelihood estimator for the mean
lifetime of a light bulb. Calculate the bias of this estimator.
March 19, 2020
35 / 147
Unbiased Estimators
Question 24
Are the moment estimators or maximum likelihood estimators in
Questions 5, 6, 11 ,12 unbiased?
March 19, 2020
36 / 147
Unbiased Estimators
Question 25
If S 2 is the variance of a random sample from an infinite population
with the finite variance ? 2 , show that S 2 is an unbiased estimator for
? 2 . Is S an unbiased estimator of ??
March 19, 2020
37 / 147
Unbiased Estimators
Question 26
If X1 , X2 , …, Xn constitute a random sample from a uniform population
[0, a], show that the largest sample value (that is, the nth order
statistic, Yn ) is a biased estimator of the parameter a. Also, modify this
estimator of a to make it unbiased. (hint: we need to know the
distribution of order statistics to calculate the expectation).
March 19, 2020
38 / 147
Order Statistics
Distribution of the r th order statistic
For random samples of size n from an infinite population that has the
value f (x) at x, the probability density of the r th order statistic Yr is
given by
n!
gr (yr ) =
(r ? 1)!(n ? r )!
Z
r ?1
yr
f (x)dx
??
Z
n?r
?
f (yr )
f (x)dx
yr
for ?? < yr < ?. March 19, 2020 39 / 147 Efficiency Minimum variance unbiased estimator (MVUE): The estimator for the parameter ? of a given distribution that has the smallest variance of all unbiased estimators for ? is called the minimum variance unbiased estimator, or the best unbiased estimator for ?. Efficiency of unbiased estimatoes ??2 relative to ??1 is measured as var (??1 ) var (??2 ) For biased estimator, the mean squared error (MSE) is used as a measure of how good the estimator ?? for ?: MSE(??) = E[(?? ? ?)2 ] = [bias(??)]2 + var (??) March 19, 2020 40 / 147 Efficiency Question 27 If X1 , X2 , ..., Xn constitute a random sample from a uniform population [0, a], (a) compare the efficiency of the moment estimator 2X? and the unbiased estimator n+1 n Yn , where Yn = max{X1 , X2 , ..., Xn }. (b) compare the mean squared error of three estimators 2X? , n+1 n Yn and Yn . March 19, 2020 41 / 147 Rao-Cramér Lower Bound Let X be a random variable with pdf f (x; ?), ? ? ? where the parameter space ? is an open interval. Under certain regularity conditions, the score statistic S(?) = ?logf??(?;x) has mean 0 and variance (called Fisher information): J(?) = E[( ? 2 logf (?; x) ?logf (?; x) 2 ) ] = ?E[ ] ?? ??2 For a random sample X1 , ..., Xn from population f (x; ?), the score P function S(?) = ni=1 ?logf??(?;xi ) , from CLT, S(?) approximately follows N(0, nJ(?)). March 19, 2020 42 / 147 Rao-Cramér Lower Bound Theorem (Rao-Cramér Lower Bound). Let X1 , X2 , ..., Xn be iid with common pdf f (x; ?) for ? ? ?. Also, suppose that the set of x values, where f (x; ?) 6= 0, does not depend on ? (the pdfs have common support for all ?). Assume certain regularity conditions hold. Let Y = u(X1 , X2 , ..., Xn ) be a statistic with mean E(Y ) = E[u(X1 , X2 , ..., Xn )] = k (?). Then Var (Y ) ? [k 0 (?)]2 nJ(?) Corollary: if Y = u(X1 , X2 , ..., Xn ) is an unbiased estimator of ?, so that k (?) = ?, then the Rao-Cramér inequality becomes Var (Y ) ? 1 nJ(?) Thus If ?? is an unbiased estimator of ? and Var (??) = 1 nJ(?) , then ?? is a minimum variance unbiased estimator of ?. March 19, 2020 43 / 147 Efficiency Question 28 Show that X? is a minimum variance unbiased estimator of the mean µ of a normal population. March 19, 2020 44 / 147 Efficiency Question 29 Let X1 , X2 , ..., Xn denote a random sample from a Poisson distribution that has the mean ?. Show that the mle X? is a minimum variance unbiased estimator of the mean ?. March 19, 2020 45 / 147 Efficiency Question 30 If Y1 , Y2 , ..., Yn is a random sample from fY (y ; ?) = 2y /?2 , 0 ? y ? ?. (a) show that ?? = 32 Y? is an unbiased estimator for ? (b) Show that the variance of ?? is less than the Rao-Cramér lower bound for fY (y ; ?). March 19, 2020 46 / 147 Consistency Consistent estimator: The statistic ?? is a consistent estimator of the parameter ? of a given distribution if and only if for each c > 0
limn?? p(|?? ? ?| < c) = 1 The kind of convergence expressed by the above definition is generally called convergence in probability. Theorem If ?? is an unbiased estimator of the parameter ? and var (??) ? 0 as n ? ?, then ?? is a consistent estimator of ? . March 19, 2020 47 / 147 Consistency Question 31 Let X1 , X2 , ..., Xn denote a random sample from a distribution with mean µ and variance ? 2 . Show that the sample mean X? is a consistent estimator of µ and the sample variance is a consistent estimator of ? 2 . (Assuming Var (S 2 ) < ?.) March 19, 2020 48 / 147 Consistency Question 32 Let X1 , X2 , ..., Xn denote a random sample from a uniform distribution on (0, ?). The ??n = Xmax was shown to be a biased estimator, is it consistent? March 19, 2020 49 / 147 Asymptotic Properties of MLE-One parameter Suppose X = (X1 , · · · , Xn ) be a random sample from f (x; ?0 ), where ?0 is the true but unknown value of ?. Let ??n = ??n (X1 , · · · , Xn ) be the M.L. estimator of ?0 based on X . Then under certain (regularity) conditions ??n ?p ?0 [nJ(?0 )]1/2 (??n ? ?0 ) ?D Z ? N(0, 1) D = 2[l(??n ; X ) ? l(?0 ; X )] ?D W ? ?2 (1) This theorem implies that for sufficiently large n, ??n has an approximately N(?0 , [nJ(?0 )]?1 ) distribution. Of course J(?0 ) is unknown because ?0 is unknown. It can also be proved [nJ(??n )]1/2 (??n ? ?0 ) ?D Z ? N(0, 1) [nI(??n )]1/2 (??n ? ?0 ) ?D Z ? N(0, 1) March 19, 2020 50 / 147 Asymptotic Properties of MLE- multiparameter case ? = (?1 , · · · , ?k )T is a vector. Let ??n = ??n (X1 , · · · , Xn ) be the M.L. estimator of ? based on X . Similarly we have ??n ?p ?0 [nJ(?0 )]1/2 (??n ? ?0 ) ?D Z ? MVN(0k , Ik ) D = 2[l(??n ; X ) ? l(?0 ; X )] ?D W ? ?2 (k ) [nJ(??n )]1/2 (??n ? ?0 ) ?D Z ? MVN(0k , Ik ) [nI(??n )]1/2 (??n ? ?0 ) ?D Z ? MVN(0k , Ik ) Note: These results do not hold if the support set of X depends on ?. March 19, 2020 51 / 147 Sufficiency Any valid statistical inference should not be affected by features of the data which are irrelevant to the question of interest. Outcomes of the same experiment which give rise to proportional likelihood functions for ? should lead to the same inferences about ?. sufficient estimator The statistic ?? is a sufficient estimator of the parameter ? of a given distribution if and only if for each value of ?? the conditional probability distribution or density of the rando... Purchase answer to see full attachment