The Study
Machine learning algorithm validation with a limited sample size
This study is like a science experiment about science experiments. It uses fake data to test how different ways of checking a computer program's guesses can trick us into thinking it's better than it really is. It shows that some common checking methods give too-good-to-be-true results, especially when there isn't much data.
Analysis score
Maximum 0 for a computational/algorithm study.
Where the score came from
When scientists use small amounts of data to train AI, they can accidentally trick themselves into thinking it works better than it does. This happens when the AI gets to 'peek' at the test data during training, especially when picking which data features to use.
Where does this study sit?
Reviews of RCTs (Meta-analyses)
Max 100Randomized Trials
Max 90Reviews of Cohort Studies
Max 85Cohort Studies
Max 72Reviews of Case-Control Studies
Max 63Case-Control Studies
Max 58Cross-Sectional & Case Series
Max 50Expert Opinion
Max 50 / 100
Quality score
Based on clinical experience or non-systematic literature reviews. The lowest level of evidence as they are most susceptible to bias and personal perspective.
Key takeaways
Summary
Based on the study abstract and findings.
- 1If a study says AI can diagnose a disease with 90% accuracy using only 50 people, it might be wrong—especially if it used the wrong testing method.
- 2With small data, regular testing tricks made AI seem 10-30% more accurate than it really was.
- 3Nested testing and simple train/test splits gave honest results.
- 4Choosing features before testing caused bigger lies than adjusting settings.
Score breakdown, methodology, conflicts of interest, evidence analysis & raw study data
Publication
Journal
PLoS ONE
Year
2019
Authors
A. Vabalas, E. Gowen, E. Poliakoff, A. Casson
Related Content
Claims (6)
When a study doesn't include enough people, the results might just be due to chance and not reflect what's really going on for most people.
If you're testing a machine learning model using a common method called K-fold cross-validation, you might think it's working better than it really is—especially if you're tuning the model using all your data first. This can trick you into believing your model is accurate when it won't work as well on new data.
Nested cross-validation is like having two layers of checkups when testing a model—it keeps the test data totally separate so the model doesn’t cheat, giving a fairer score no matter how much data you have.
If you pick the most important features using all your data — including the test set — your model might look better than it really is, because it's secretly cheating by seeing data it shouldn't see yet.
Splitting data into training and testing sets gives a fair measure of how well a machine learning model works—just as reliable as more complex methods—because it keeps the test data completely separate so the model doesn't cheat by seeing it early.
If a machine learning model has more features (like traits or measurements) than data points (like people or samples), it can 'cheat' by finding fake patterns in random noise, especially when tested the wrong way — making it look better than it really is.
Not medical advice. For informational purposes only. Always consult a qualified healthcare professional before making health decisions.