The Claim
K-fold cross-validation produces optimistically biased performance estimates in machine learning models when sample sizes are small, and this bias persists even at sample sizes up to 1,000, especially when feature selection or hyperparameter tuning is performed on the entire dataset prior to validation; this occurs because data used for model development leaks into the validation process, resulting in overfitting and inflated accuracy estimates that do not reflect true generalization performance.
What the research says
Not yet evaluated
We are still looking at what the research says.
These are independent scores, not a percentage. Higher-grade studies count more, so a single strong opposing study can outweigh several weaker ones.
If you're testing a machine learning model using a common method called K-fold cross-validation, you might think it's working better than it really is—especially if you're tuning the model using all your data first. This can trick you into believing your model is accurate when it won't work as well on new data.
See the scientific wording
K-fold cross-validation produces optimistically biased performance estimates in machine learning models when sample sizes are small, with the bias persisting even at sample sizes as large as 1,000, particularly when feature selection or hyperparameter tuning is performed on the full dataset before validation. This bias arises because data used for model development leaks into the validation process, leading to overfitting and inflated accuracy estimates that do not reflect true generalization performance.
What the research says
1 studyStudy: Machine learning algorithm validation with a limited sample size
The study shows that a common method for testing AI models can make them look better than they really are, especially when there isn’t much data, which is exactly what the claim says.
Score breakdown, mechanism chain, raw evidence, ideal studies needed & 1 supporting studies
Not medical advice. For informational purposes only. Always consult a qualified healthcare professional before making health decisions.