In machine learning, sometimes the prediction of our model may not be satisfactory. Although there may be many reasons for that, often it is due to either overfitting or underfitting of the model.
If we have too many features, the machine learning model may fit the training data very well, but fail to generalize to the test data. It is because our model is too complex for the amount of training data available thus not generalize to new examples. This problem is known as overfitting.
Think of overfitting as memorizing as opposed to learning.
On the other hand, underfitting occurs when our model is too simple to capture the underlying trend of the data thus doesn’t even perform well on the training data and not likely to generalize well on the testing data as well. For example, when fitting a linear model to non-linear data. Such a model would have poor predictive performance.
The bias is the difference between the model’s predictions and the true values. High bias can cause the model to underfit.
The variance is the model’s sensitivity to small fluctuations in the training set. High variance can cause the model to overfit.
To get good predictions, you’ll need to find a balance of bias and variance the minimizes the total error.
In order to address the problem of overfitting, there are a number of techniques as:
Image source: http://scott.fortmann-roe.com/docs/BiasVariance.html