Machine Learning Tutorial

Kernelized SVM for Classification

Before applying the algorithm, is important to preprocess the data, otherwise it will perform poorly. Use MinMaxScalar function or do it by hand, like the code below:

# compute the minimum value per feature on the training set
min_on_training = X_train.min(axis=0)

# compute the range of each feature (max - min) on the training set 
range_on_training = (X_train - min_on_training).max(axis=0)

# subtract the min, and divide by range
# afterward, min=0 and max=1 for each feature
X_train_scaled = (X_train - min_on_training) / range_on_training

The algorithm:

from sklearn.svm import SVC

svm = SVC(kernel='rbf', C=10, gamma=0.1).fit(X, y)

# To get the support vectors
sv = svm.support_vectors_

Tips

• The gamma parameter controls the width of the Gaussian kernel. It determines the scale of what it means for points to be close together.
• The C parameter is a regularization parameter, similar to that used in the linear models. It limits the importance of each point (or more precisely, their dual_coef_).

Good settings for the two parameters are usually strongly correlated.

Parameters effects on a model

A small gamma means a large radius for the Gaussian kernel, which means that many points are considered close by. This is reflected in very smooth decision boundaries on the left, and boundaries that focus more on single points further to the right. In other words, it means low complexity of the model.

As with the linear models, a small C means a very restricted model, where each data point can only have very limited influence. Increasing C, as shown on the bottom right, allows these points to have a stronger influence on the model and makes the decision boundary bend to correctly classify them.