Validation - 7jw9/MASc GitHub Wiki

Presentation slides from Wright State University giving an overview of various validation techniques and how they differ. Short and simple with great figures.

http://research.cs.tamu.edu/prism/lectures/iss/iss_l13.pdf

Seminal paper that showed LOOCV does not lead to confident estimate of model (probability of choosing model with best predictive ability does not converge to 1 as n->infinity; leave-k-out cross-validation strategy rectifies this):

http://www.jstor.org/stable/2290328?seq=1#page_scan_tab_contents

Another perspective on LOOCV:

"In a famous paper, Shao (1993) showed that leave-one-out cross validation does not lead to a consistent estimate of the model. That is, if there is a true model, then LOOCV will not always find it, even with very large sample sizes. In contrast, certain kinds of leave-k-out cross-validation, where k increases with n, will be consistent. Frankly, I don’t consider this is a very important result as there is never a true model. In reality, every model is wrong, so consistency is not really an interesting property." - http://robjhyndman.com/hyndsight/crossvalidation/