Data Preparation

This chapter focuses on data preparation, a crucial step in the analytics process to ensure that the data used for modeling is of the highest quality. The chapter covers various aspects of data preparation, including obtaining the needed data, data cleaning, handling missing values, detecting and dealing with outliers, and feature engineering. The chapter starts by highlighting that a company’s data warehouse may not always provide the required data in the correct form, and data must be assembled, cleaned, and tailored to the analytics problem. It emphasizes the importance of having the right type of data, such as predictors and outcome variables for predictive modeling, and the need for external data integration from various sources. Data cleaning is discussed in detail, acknowledging that datasets may contain inaccurate, incomplete, or inconsistent values, among other issues. The chapter lists several activities for data cleaning, such as removing duplicate records, dealing with missing values, identifying and handling outliers, and ensuring consistency in formats and units of measurement. The presence of missing values is recognized as one of the most challenging problems in analytics. The chapter discusses the types of missing values (MCAR, MAR, MNAR). It outlines various techniques for handling missing data, such as listwise deletion, imputation methods, and using indicator variables for missingness. Outliers are discussed in the context of both univariate and multivariate methods for detection. The chapter emphasizes that outliers should not be routinely removed without careful consideration and domain knowledge. Instead, different techniques for handling outliers, such as replacing them with missing value indicators or Winsorizing, are provided. Feature engineering is introduced to improve predictive model performance by creating new variables or transforming existing ones. The chapter presents examples of feature engineering, including creating new variables from existing ones, adding polynomial terms, and using transformations to reduce skewness. Throughout the chapter, the KNIME data analytics platform is demonstrated to carry out various data preparation tasks, such as missing value handling, outlier detection, and transformations. In conclusion, this chapter emphasizes the significance of data preparation in the analytics process and provides practical guidance and examples for data cleaning, missing value handling, outlier detection, and feature engineering using KNIME.