Clinical Significance

Statistical Significance

Definition

Assigned to a result where there are meaningful and quantifiable effects weighed against potential costs, inconveniences, and harmful effects.

Determines if an effect or event likely happened due to random chance.

Terms to Know

Effect size is a measure of the observed phenomenon’s magnitude. This might include the correlation between two variables, the risk of an event occurring, or the mean difference between two treatments, groups, or variables.

Number Needed to Treat (NNT) is a type of effect size that measures the average number of patients who need to be treated to prevent one additional bad outcome, or the number who need to be treated for one to benefit over the control treatment.

Confidence intervals provide information about the range in which the true value lies with a specified degree of probability. It also indicates direction and the strength of the effect.  In medicine, confidence intervals are often set at 95% or 99%. For example, at 95% CI means there is 95% chance that range (the interval) contains the true population mean. If we  repeated the same study 100x from many samples, we’d get the same values 95% of the time.

For most tests or studies, the null hypothesis is when there is no relationship between variables or when there is no difference between groups.

P value, or probability value, tells you how likely your data could have occurred if the null hypothesis is true. It’s a proportion, so if p=.039, then there is a 3.9% chance your data occurred due to random chance. The p-value can only tell you whether or not the null hypothesis is supported.  It does not mean your alternative hypothesis is true or why.

Significance level is set at the start of a study, before data collection, and is chosen based on discipline (some go as high as .09, but in medicine we stick to .01 and .05). After data collection, if the p value is less than or equal to the significance level, the results are considered statistically significant.

Why does it matter?

Clinical relevance facilitates the understanding and interpretation of results for clinicians. It’s not enough for a drug just to reduce a patient’s level of pain. Is it cheaper than other pain-reducing drugs? How much does it impact pain? Are the side effects more mild and less frequent than with other drugs?

Statistical significance helps us understand how strongly the results of a test should impact our decisions. It’s not the only factor we should consider, when assessing the importance of our data. Context, sample size, cost, effect on patient’s QoL all matter. Statistically significant results may not have meaning impact in clinical settings.

Bottom Line

Clinical significance is about the size and scope of an effect, not just whether or not something happened.

Statistical significance tries to verify an effect is taking place. However, it is not the primary determinant of truth, efficacy, or importance.

More Resources

¨ Statistical or Clinical Significance?

¨ Probability, Proof, and Clinical Significance

¨ Stats Refresher from Harvard Business Review

¨ Common Pitfalls in Stats Analysis