Skip to content

Handling Missing Values

Description

Missing data is a common problem that occurs in many machine-learning projects. Dealing with missing data is important because ML models cannot handle missing data and will either produce errors or provide inaccurate results.

There are several methods for dealing with missing data in ML projects:

  • Dropping rows: Addressing missing data can involve a straightforward approach of discarding rows that contain such values. Nevertheless, exercising caution is paramount when employing this method as excessive row removal may result in the loss of valuable data, impacting the overall accuracy of the model. We usually use this method when we have a few rows in our dataset, and we have a few rows with missing values. In this case, removing a few rows can be a good and easy approach to training our model while the final performance will not be affected significantly.
  • Dropping columns: Another approach is to drop the columns that contain missing values. This method can be effective if the missing values are concentrated in a few columns and if those columns are not important for the analysis. However, dropping important columns can lead to a loss of valuable information. It is better to perform some sort of correlation analysis to see the correlation of the values in these columns with the target class or value before dropping these columns.
  • Mean/median/mode imputation: Mean, median, and mode imputation entail substituting missing values with the mean, median, or mode derived from the non-missing values within the corresponding column. This method is easy to implement and can be effective when the missing values are few and randomly distributed. However, it can also introduce bias and affect the variability of the data.
  • Regression imputation: Regression imputation involves predicting the missing values based on the values of other variables in the dataset. This method can be effective when the missing values are related to other variables in the dataset, but it requires a regression model to be built for each column with missing values.
  • Multiple imputation: Multiple imputation encompasses generating multiple imputed datasets through statistical models, followed by amalgamating the outcomes to produce a conclusive dataset. This approach proves efficacious, particularly when dealing with non-randomly distributed missing values and a substantial number of gaps in the dataset.
  • K-nearest neighbor imputation: K-nearest neighbor imputation entails identifying the k-nearest data points to the missing value and utilizing their values to impute the absent value. This method can be effective when the missing values are clustered together in the dataset. In this approach, we can find the most similar records to the dataset to the record that has the missing value, and then use the mean of the values of those records for that specific record as the missed value.