Bayes Error Rate - niranjv/ml-notes GitHub Wiki

• In classification, test error rate is minimized when instance is assigned to most likely class, given its covariates. I.e,. instance with covariate vector x_0 will be assigned to class j for which:

Pr(Y = j | X = x_0)

is maximum

• This classifier is called Bayes Classifier and gives the lowest possible test error rate (Bayes error rate)
• Overall Bayes error rate is:

1 - E(max_j Pr(Y=j | X) )

where E is over all possible values for X

• Cannot use Bayes classifier in practice because we don't know P(Y|X)
• So estimate P(Y|X) and classify instance with highest estimated P(Y|X), e.g., KNN

• ISLR, Section 2.2 (Bayes classifier)