Clinical significance

so in this video we're going to talk about the difference between clinical significance and statistical significance so so far we've been discussing statistical significance in the last few videos we've talked about power which is the ability to reject the null hypothesis when it isn't true or in other words the ability of our test to detect a statistically significant effect and we saw that power depends on Beta Alpha the effect size in the sample size so let's use an example let's say that we have a drug that we think will affect body weight and we're going to test it against a placebo to see if it reduces weight more than a placebo so here are the distributions for the drug which we're going to call the alternate distribution and the placebo which were calling the null distribution and in this study we have 200 people so we have a hundred people that are given the drug and a hundred people that are given the placebo and we're looking for a change in percent change in weight between before they took the drug or placebo and after they took the drug or placebo so you can see here this is the null hypothesis distribution meaning there is no change so the placebo would be that distribution so that's why the mean of the distribution is at zero so there's no change because there was no essentially a drug given and then this is the alternate distribution which is the drug now in this case you can see that the mean of the alternate distribution is negative one meaning that there was a change in weight of one percent one percent weight loss among the people who took the drug now the question is is are those is that different statistically significant and so how do we tell well again we have a cut-off point on the null distribution which in this case would be probably a and here right and then alpha is on that side and so what we see is that most of the alternate distribution is overlapping with the null distribution so therefore it's very unlikely that we're going to reject the null hypothesis and say that they're statistically significant so the likely outcome here is going to be the acceptance of the null and rejection of the alternate distribution meaning there is no difference between the drug and the placebo so now remember what happens when you increase the sample size of a distribution you you narrow the distribution because what you do is you decrease the standard error so here we have a massive increase in the sample size so we multiplied the sample size by 100 so now we have 10,000 people taking the drug and 10,000 taking the placebo and so therefore the difference between the drug and the placebo is the same right the effect size is still one one percent change but in this case the distributions are so narrow that we will are very very likely in this case to reject the null hypothesis right because the very little of the alternate distribution is overlapping with the null distribution so the point is is that the sample size can have a big impact on whether a result is statistically significant so in both cases we had a change in weight of 1 percent so here this shows this this effect size right here this difference is the same in each study right but in one case is statistically significant down here because the sample size is so big the standard is so small and therefore the distributions are so narrow and do not overlap and in the other case the sample size was smaller and so the reverse is true these distributions are much wider because the standard error is bigger and therefore they overlap more so what is clinical significance clinical significance is whether that difference that you detect so in this case it's not it is statistically significant so you say oh yeah my study worked my drug worked its effective so do you want to sell this drug say the drug has some side effects it's expensive or even it's not that expensive it reduces in it reduces way by 1% well there may not be any clinic clinically meaningful impact of that small change in weight and therefore even though this drug is statistically significantly different from a placebo it's not clinically significantly different from placebo so the effect is statistically significant but not clinically significant so just to review real quickly statistical significance is just simply that the difference that you got in the you know the difference between the control group and the test group the experimental group is the probability of finding that if the null hypothesis is true is less than 5% right so we've been over that clinical significance is not a mathematical sort of it's not related to the hypothesis testing or the p-values right it's it's basically a clinical judgement that says this difference that I found in my study that significally significant but is it also clinically meaningful does it result in a long-term clinical benefit for example does say the 1% reduction in weight result in long-term reduction of risk of cardiovascular disease or more tab so usually we're when we power a study but in other words when we decide how many people are going to be in this study we base that as you saw before power is based on an effect size that you're anticipating and so usually we want that effect size to be clinically significant so we would never say okay we wanna we're gonna include you know a thousand people to see a difference of one pound we don't care about a difference of one pound because it's not clinically significant so like in the case of weight usually a five percent change in weight is has been shown in other studies to be related to clinical benefit long term and so therefore in a weight loss study you would always want to have a difference of at least five percent in the change in weight eat otherwise it's not clinically significant even if it is statistically significant so hopefully that helps

1 comment

  1. So you are saying that although something is statistically significant it doesn't really define the "impact" it has or the "magnitude" it has on the patient? Does this also mean that studies that determine number needed to treat in terms of lowering cardiovascular event risk, rather, determines this magnitude?

Leave a Reply

(*) Required, Your email will not be published