|
Lecture Notes 9 Econ 29000,
Principles of Statistics Kevin R Foster, CCNY Spring 2011 |
|
|
|
Details of
Distributions T-distributions, chi-squared, etc. |
|
|
Take the basic methodology of Hypothesis Testing and figure out how to deal with a few complications.
T-tests
The first complication is if we have a small sample and we're estimating the standard deviation. In every previous example, we used a large sample. For a small sample, the estimation of the standard error introduces some additional noise – we're forming a hypothesis test based on an estimation of the mean, using an estimation of the standard error.
How "big" should a "big" sample be? Evidently if we can easily get more data then we should use it, but there are many cases where we need to make a decision based on limited information – there just might not be that many observations. Generally after about 30 observations is enough to justify the normal distribution. With fewer observations we use a t-distribution.
To work with t-distributions we need the concept of
"Degrees of Freedom" (df). This just takes account of the fact that, to
estimate the sample standard deviation, we need to first estimate the sample
average, since the standard deviation uses
If we have thousands of observations then we don't really need to worry. But when we have small samples and we're estimating a relatively large number of parameters, we count degrees of freedom.
The family of t-distributions with mean of zero looks basically like a Standard Normal distribution with a familiar bell shape, but with slightly fatter tails. There is a family of t-distributions with exact shape depending on the degrees of freedom; lower degrees of freedom correspond with fatter tails (more variation; more probability of seeing larger differences from zero).
This chart compares the Standard Normal PDF with the t-distributions with different degrees of freedom.
This table shows the different critical values to use in place of our good old friend 1.96:
|
Critical
Values for t vs N |
|
|||||
|
df |
95% |
90% |
99% |
|||
|
5 |
2.57 |
|
2.02 |
|
4.03 |
|
|
10 |
2.23 |
1.81 |
3.17 |
|||
|
20 |
2.09 |
1.72 |
2.85 |
|||
|
30 |
2.04 |
1.70 |
2.75 |
|||
|
50 |
2.01 |
1.68 |
2.68 |
|||
|
100 |
1.98 |
1.66 |
2.63 |
|||
|
Normal |
1.96 |
|
1.64 |
|
2.58 |
|
The higher numbers for lower degrees of freedom mean that the confidence interval must be wider – which should make intuitive sense. With just 5 or 10 observations a 95% confidence interval should be wider than with 1000 or 10,000 observations (even beyond the familiar sqrt(N) term in the standard error of the average).
T-tests with two
samples
When we're comparing two sample averages we can make either of two assumptions: either the standard deviations are the same (even though we don't know them) or they could be different. Of course it is more conservative to assume that they're different (i.e. don't assume that they're the same) – this makes the test less likely to reject the null.
Assuming that the standard errors are different, we compare
this test statistic against a t-distribution with degrees of freedom of the
minimum of either
Sometimes we have paired data, which can give us more powerful tests.
We can test if the variances are in fact equal, but a series of hypothesis tests can give us questionable results.
Other Distributions
There are other sampling distributions than the Normal Distribution and T-Distribution. There are χ2 (Chi-Squared) Distributions (also characterized by the number of degrees of freedom); there are F-Distributions with two different degrees of freedom. For now we won't worry about these but just note that the basic procedure is the same: calculate a test statistic and compare it to a known distribution to figure out how likely it was, to see the actual value.
(On Car Talk they joked, "I once had to learn the
entire Greek alphabet for a college class.
I was taking a course in ... Statistics!")
Complications from a
Series of Hypothesis Tests
Often a modeler will make a series of hypothesis tests to attempt to understand the inter-relations of a dataset. However while this is often done, it is not usually done correctly. Recall from our discussion of Type I and Type II errors that we are always at risk of making incorrect inferences about the world based on our limited data. If a test has a significance level of 5% then we will not reject a null hypothesis until there is just a 5% probability that we could be fooled into seeing a relationship where there is none. This is low but still is a 1-in-20 chance. If I do 20 hypothesis tests to find 20 variables that significantly impact some variable of interest, then it is likely that one of those variables is fooling me (I don't know which one, though). It is also likely that my high standard of proof meant that there are other variables which are important but which didn't seem to be.
Sometimes you see very stupid people who collect a large number of possible explanatory variables, run hundreds of analyses, and find the ones that give the "best-looking" test statistics – the ones that look good but are actually entirely fictitious. Many statistical programs have procedures that will help do this; help the user be as stupid as he wants.
Why is this stupid? It completely destroys the logical basis for the hypothesis tests and makes it impossible to determine whether or not the data are fooling me. In many cases this actually guarantees that, given a sufficiently rich collection of possible explanatory variables, I can form hypothesis tests and show that some variables have "good" test statistics – even though they are completely unconnected. Basically this is the infamous situation where a million monkeys randomly typing would eventually write Shakespeare's plays. A million earnest statisticians, doing random statistical tests, will eventually find test statistics that look great with very low p-values. But that's just due to persistence; it doesn't reflect anything about the larger world.
In finance, which throws out gigabytes of data, this phenomenon is common. For instance there used to be a relationship between which team won the Super Bowl (in January) and whether the stock market would have a good year. It seemed to be a solid result with decades of supporting evidence – but it was completely stupid and everybody knew it. Analysts still work to get slightly-less-implausible but still completely stupid results, which they use to sell their securities.
Consider the logical chain of making a number of hypothesis tests in order to find one supposedly-best model. When I make the first test, I have 5% chance of making a Type I error. Given the results of this test, I make the second test, again with a 5% chance of making a Type I error. The probability of not making an error on either test is (.95)(.95) = .9025 so the significance level of the overall test procedure is not 5% but actually (1 - .9025) = 9.75%. If I make three successive hypothesis tests, the probability of not making an error is .8574 so the significance level is 14.26%. If I make 10 successive tests then the significance level is over 40%! This means that there is a 40% chance that the tester is being fooled and there is not actually the relationship that was hypothesized – and worse, the stupid tester believes that the significance level is just 5%.