In this section, I just want to give a brief note and talk about computation of confidence intervals for population quantities based on small samples. And this is a detail that will be handled by the computer, and it's not something to worry about having to deal with by hand, etc. But I just want to put it out there, so that when you see things in the literature that don't quite fit with what I've told you about creating confidence intervals, by adding plus or minus two standard errors. You'll just be aware of this correction that needs to be made in relatively small sample sizes. So I just want you to appreciate and note the role of corrections to the central limit theorem methods when estimating the confidence interval mean from a small data sample. And appreciate and note the role of exact computer-based computations as an alternative to the CLT methods when estimating confidence interval for a proportion or incidence rate from a small data sample. So, I didn't give you the full debriefing on the Central Limit Theorem. And the one piece I left out is that it requires, quote-unquote, enough data to kick in and the sampling distribution to be theoretically normal. For example, with sample means, a sample size cutoff often used for normality to guarantee the normality of the theoretical sampling distribution is when the means are based on samples of size 60 or larger. So in sampling distribution back in lecture set six, we saw situations where we had samples of smaller than 60, and the sampling distribution still looked normal. And I'll tell you why that is in a minute, it was pretty much normal with a slight caveat. So why does the Central Limit Theorem require quote and unquote larger samples? Well, the sample needs to be large enough. Remember means are highly susceptible to outline points, potentially, unless the sample size is large enough so that the influence of any one point is negligible. And so that's the idea of the Central Limit Theorem. Requires a relatively large sample to kick in, so that the influence of extreme points in some of the samples we may get across all possible random samples is mitigated and relegated to a small amount. So large enough depends on data type, for means it's 60. For binary outcomes, there's slightly different criteria but. So I bring this up not to cause concern, or worry, or make you think that you always have to be monitoring the sample size. And worrying about how to create a confidence interval depending on what your rent is. Anyway, if you're using a computer, it will take care of that detail for you. The more important thing is, I just want to point out sometimes you'll see confidence intervals that are wider than you'd expect. Or slightly different than you expect as they're not symmetric around the estimate for proportions or incidents rates for example. And, you might be thinking, why is that? Because if I take in the estimate plus or minus two standard errors, I should get another result. And that's because some other process was used to create the confidence interval because the results, the sample size is not large enough to use the Central Limit Theorem based approach. That's all I want you to know. The computer will handle this, the interpretation of the confidence level is still the same. I just want you to be aware of this when you see situations like that in the literature, or get results that look different than what you thought you had computed by hand. So let recap the results of the CLT, or Central Limit Theorm for example means. Common taking a random sample of size n from a population. The theoretical sampling distribution of a sample mean across all random samples is approximately normal, it's centered at the true mean. And most of the observations are s under the curve. The estimated sample means would fall within plus or minus two standard errors of the true mean. So it turns out, this is almost true in smaller samples. You wouldn't be able to detect it if I showed you a single curve. But the curve for this theoretical sample distribution of means based on smaller samples is not quite normal. It's slightly wider and flatter than a normal curve, although it is symmetric and bell shaped. So, unless you had it side-by-side compared to a normal curve, it would still look, you wouldn't be able to distinguish it from a normal curve. But it's what's called a t-distribution, so it's sort of a little fatter in the tails normal distribution. So in smaller samples, in order to get 95% coverage under this slightly fatter version, the normal distribution, we may have to go out more than two standard errors to achieve 95% confidence. And how much bigger this needs to be than two? How many standard errors more than two? We need to add and subtract depends on the size of our sample. And this could be looked up in what's called a t-table or a t-distribution. A t-distribution is uniquely, it's a mean zero. Distribution is uniquely defined by what is called our degrees of freedom. And for any single sample, the degrees of freedom we have is n- 1. So if I have 20 observations, I have what are called 19 degrees of freedom. But in reality, we're not going to look things up on these tables. I never do it as practice, the computer will handle this detail. So how many standard errors we need to go depends on the degrees of freedom? And this is linked to sample size and the appropriate degrees of freedom for any given study or n- 1. Let me just try and give you some insight where this idea comes from degrees of freedom. Let's start with a very simple example where I have a sample of size 2. I suppose I told you I had a sample of ages for people. A sample, two people from a larger population. I wanted to lift the age distribution in the sample, and the sample means is 30 years old. And I ask you, how old is person number 1? And how old is person number two? Well, there's no way, if I only told you the mean of the two persons was 30, there's no way you could tell me specifically how old person one or two were. You could come up with examples that would work, two values that would have a mean of 30, but there's no way you could specifically identify the ages for the two persons. But then if I told you, look person number 1 is 20, and you know that the mean is 30, then person number two the value for them has to be whatever makes the mean of these two numbers, Turn out to be 30, so they would be 40 years old. So once you've specified that first value, if I know the mean, the second value is no longer random, it has to be whatever makes the mean 30. So the first observation's random without knowing, if we only know the mean, you couldn't tell me the value of the first or second observation. But as soon as you know the mean and the first observation, the second one is fixed. So of these two observations, once I know the mean, only one is free to vary. That's degrees of freedom, and you could extend this idea to larger groups. So if I sample 10, Ages, I knew the mean age, and the first 9 ages, then the tenth ages fix. So only 9 of those 10 are truly random. In any case, you could look up the correct t-number in a t-table or t-distribution, For a t-distribution with n- 1 degrees of freedom. And the constructed confidence interval, we'll do the same because actual approach we take the mean plus or minus a fix number of standard errors. But the number of standard errors would be depend on our sample size to get 95% coverage. So here's an example t-table to some values on a t-table for given degrees of freedom. Here's the number of standard errors we would need to add and subtract to get 95% coverage for given different sample sizes. So if I had 12 observations, if n = 12, I'd have 11 degrees of freedom. I need to take, in order to get 95% confidence interval for the population mean based on sample of 12, I'd take my mean and subtract 2.201 standard error. So larger than 2, Because of this smaller sample. You can see it's the degrees of freedom increases, the values decrease, they get close and closer to 2. And at 60 they hit 2 exactly which is essentially we've said is what we call the number of standard errors under a normal curve needed to achieve 95% confidence. Truly it's 1.96, so if you were a stickler for being proper, then you can see as the degrees of freedom approaches infinity. This converges to a true normal distribution but for practical purposes, we get what we need with samples of size 60 or more. Again, I don't want you to worry about this, you will never use this table in this class. I don't look things up in these tables, the computer will handle it. So let me just show you an example though of where this would be necessary to make this correction. So here's a small study on a response to treatment among 12 patients with hyperlipidemia, high LDL cholesterol level. So I took a sample of persons with high LDL, Gave them a treatment and then measured the changing cholesterol post treatment measurement, pre-treatment for each of the 12 patients. And the mean change across the 12 patients was a decrease of 54.1 milligrams per deciliter in LDL, it's sizable negative. So the change was negative because it was a decrease, but there was a lot of variability in the 12 changes across the 12 people, and that's the standard deviation. If we wanted to get a 95% confidence interval for the true mean change, we will start with, as we always have done before, we start with our mean change, -54.1 and add and subtract not two standard errors but 2.2 standard errors. We can compute standard error the same way by taking the standard deviation of our 12 measurements divided by the square root of 12. You can easily find the t-tables for other cutoffs in any stats text or by searching the Internet, or just by using a software package on the computer. The point is, not to spend a lot of time, or any time actually looking up t-values. More importantly is the basic understanding of why slightly more needs to be added to the sample means smaller samples to get a valid 95% confidence interval. And the interpretation, the resulting confidence interval is, the same as discussing the earlier lectures section in this lecture set. For small samples of binary and time to event data, there's no adjustment analogous to the t-distribution for creating confidence intervals. Exact methods need to be employed when creating smaller sample intervals for proportions and incidence rates, these are handled by the computer. Traditionally, these were done with only small samples, and the CLT results were used otherwise, even by computers. Because its exact computations take up a lot of computer memory, but in this day and age, computer have amazing amounts of memory. So now, computers generally report the results of these exact methods but the results are pretty much identical to the Central Limit Theorem, results for all but the smallest of samples. So let me just give you an example of a small sample. What happens if we applied the Central Limit Theorem to a small sample of binary data? Why we need the correction to the Central Limit Theorem? So suppose a random sample of 20 Hopkins students was taken in February 2017. The students were asked if they currently had a cold. Now these 20 students, 3 had cold symptoms. So I might ask, what is a 95% confidence interval for the true prevalence or proportion of colds among Hopkins students in February 2017? So the sample proportion is 3 out of 20 or 15%. If I wanted to do a confidence interval based on the Central Limit Theorem base methods and I add or subtract two estimates standard errors, I get a confidence interval whose lower bound was negative. So negative 1% to positive 31%, we know that proportions can't be less than 0. So this doesn't make sense scientifically. This would be a tip off that something was wrong. So the exact 95% confidence interval for this based on the computer goes from 3% to 38%. So this is correct on both ends of having a positive value. And you can see that the lower end is a possibility. And the upper end is slightly larger than what we would have gotten by the Central Limit Theorem approach. Both these intervals are wide because we have a small sample. But this one yields correct values for the entire range of possibilities. So with a small sample, adjustments need to be made to the Central Limit Theorem-based approaches to estimating confidence intervals for means, proportions, and incidence rates. Computers can handle these computations, but the interpretation of the resulting 95% and other level confidence interval is the same, regardless of the how the confidence interval is constructed. So one thing to think about this, why is this the case? Why do we need to be more conservative in smaller samples? When you think about the theoretical standard error for quantities of interest versus a true standard error, right. So for example, for sample means, or for sample proportions. If we compute the true standard error, Using the underlying population matters, as we don't know, then we won't need this corrections. But for example, the standard error of a sample mean, the theoretical ones based on the standard error of the values in the population for which the sample is taken, divided by the square root of the sample size. We don't know sigma, because we're not observing everyone in the population. We can only estimate this and do an approximate standard error using our best estimate. And similarly for, Proportions, we can only estimate standard bearer as a function of our observed proportion, not the truth. So just like means and proportions in smaller samples, the uncertainty in our estimates of the variability in our sample estimates becomes more higher, gets higher. So the last data they're based on them more on stable they are. And some point we have to counter act, the fact that what they are estimated standard errors based on these estimates from the samples are not very precise. And so we have to add a more uncertainty to get 95% coverage, because the standard error estimates are ripe with their own type of uncertainty that comes from being based on a small sample.
