KNN (K-Nearest Neighbors)

Intro

K-Nearest Neighbors is a supervised machine learning algorithm for classification and regression that finds the “k” closest data points

k-NN classifier

Application of a k-NN classifier considering k = 3 neighbors. Left - Given the test point ”?”, the algorithm seeks the 3 closest points in the training set, and adopts the majority vote to classify it as “class red”. Right - By iteratively repeating the prediction over the whole feature space (X1, X2), one can depict the “decision surface”

Algorithm

How it works

1. Choose K (number of neighbors)

2. Compute distance (e.g. Euclidean, Manhattan, or cosine (often for embeddings)) between the new point and all training points

3. Select the K nearest points

4. Aggregate:

   • Classification → majority vote
   • Regression → average of values