Check out our blogs where we cover topics such as Python, Data Science, Machine Learning, Deep Learning. A Best place to start your AI career for beginner, intermediate peoples.


Feature Engineering

If you want to be a successful data scientist and to make your model predict the most accurate result, then the following article is for you.

 What is Feature Engineering?

As per Wikipedia, Feature engineering is the process of using domain knowledge to extract features from raw data via data mining techniques. These features can be used to improve the performance of machine learning algorithms. Feature engineering can be considered as applied machine learning itself.

Okay, so we saw what Wikipedia said about it but still, we want to know more about it. So let’s divide this term into 2 parts for better understanding i.e Feature and Engineering.

 What are the Features in a dataset?

Basically, all machine learning algorithms use some input data (independent data) to generate output. These input data are features, which are usually in the form of structured columns.

So, why do we engineer it?

To get the most accurate or I can say to get the most precise output from our machine learning the algorithm we need to give it cleanest data possible which should be compatible with our machine learning algorithm.

So the process of preparing the proper input dataset for our machine learning model is known as Feature Engineering.

 Now you will ask how important is this process of feature engineering to generate any machine-learning project?

 According to Forbs survey, data scientist spends around 80% of their time on data preparation.

Forbes Data Science Survey

Link of the complete survey by Forbes.

Data scientists spend 60% of their time cleaning and organizing data. Collecting data sets come second at 19% of their time, meaning data scientists spend around 80% of their time on preparing and managing data for analysis.

76% of data scientists view data preparation as the least enjoyable part of their work

57% of data scientists regard cleaning and organizing data as the least enjoyable part of their work and 19% say this about collecting data sets.

List of Techniques you can find in this blog:

1.) Techniques of Imputation of numerical and categorical data

2.) Dealing with outliers

3.) Binning

4.) Techniques of dealing with Gaussian-Distribution / Skewness

5.) OneHotEncoding and OrdinalEncoder

6.) Feature splitting & extraction

7.) Group By

8.) Concat, Merge, Join

9.) Scaling

10.) Extracting Date


1.) Techniques of Imputation of numerical and categorical data

Deal with missing data


In this we deal with missing values that are present in our dataset.

The most simple solution to deal with missing values is to drop that row or column, suppose that you have more than 75% of values missing that you can definitely drop that row or column. 

 threshold = 0.75  
 # Columns:  
 data = data[data.columns[data.isnull().mean() < threshold]]  
 # Rows:  
 data = data.loc[data.columns[data.isnull().mean(axis=1) < threshold]]

 The most common way to deal with missing values is through fill 0, Mean, Mean, or Mode value in that missing space. 

 # fill NaN values with 0  
 data = data.fillna(0)  
 # fill NaN values with median  
 data = data.fillna(data.mean())  
 # fill NaN values with median: Better than Mean  
 data = data.fillna(data.median())  
 # fill NaN values with mode  
 data = data.fillna(data.mode()) 

 To deal with numerical data specifically you can use imputers they are very useful to get the perfect value to fill in that missing space.

 Above mention is too naïve way of imputation, but we are ninjas we have some of our own ninja techniques we can use

(i) Simple Imputer

-A simple imputer is univariate which means it will take only a single feature in the count. 

 # using simple imputer  
 import numpy as np  
 from sklearn.impute import SimpleImputer  
 imputer = SimpleImputer(missing_values=np.nan, strategy='median') # can be mean, median, mode  
 imputer =  
 data = imputer.transform(x)  
 print('Imputed Data:',data)

 -This is quite a blunt approach.

-What if the feature “x” which has NaN values is very well co-correlated with the features such as “y” or “z”. Let’s say, people with higher “Age” give more “Rent” and people with lower “Age” give less “Rent”. So that suggests a multivariate approach. Which just means it takes multiple features into count.

-So the Iterative Imputer and KNN Imputer comes into the concentration. 

(ii) Iterative Imputer

-In iterative imputer for all the rows in which “Age” is not missing sklearn trains a regression model.

-Where other features such as “Gender” and “Rent”  are considered as independent input features and “Age” is considered as the target output feature.

 -Now for all rows in which “Age” is missing it makes predictions for “Age” by using features “Gender” and “Rent” to trained model. 

 # IterativeImputer  
 from sklearn.experimental import enable_iterative_imputer  
 from sklearn.impute import IterativeImputer  
 imputer_it = IterativeImputer()  


NOTE: This model used by iterative imputer is totally independent of the model you are using this dataset as training data. 

(iii) KNN Imputer

-Let’s pretend we have a row in which “Age” is missing.

-Then sklearn finds the 2 most similar rows measured by how close the “Gender” and “Fare” values are to this row. 

  # KNNImputer   
  from sklearn.experimental import enable_iterative_imputer   
  from sklearn.impute import KNNImputer   
  imputer_knn = KNNImputer(n_neighbors=2)   


Dealing with the categorical and nominal data with the help of mode

data['categorical_col'] = data['categorical_col'].fillna(data['categorical_col'].mode()[0])


2.) Dealing with outliers

Here is an Outlier

Detecting outliers:

Using Z-score:

Data that fall outside of the 3rd standard deviation is considered as an outlier. We can use a Z-score if Z-score falls outside of μ+3 or μ-3 then we will consider it as an outlier.


 import numpy as np  
 box = [10,20,15,25,11,12,16,26,19,29,3000]  
 outliers = []  
 def detect_outliers(data):  
   Threshold = 3  
   mean = np.mean(data)  
   std = np.std(data)  
   for i in data:  
     z_score = (i-mean)/std  
     if np.abs(z_score) > Threshold:  
   return outliers  
 out = detect_outliers(box)  

Using IQR:

Datapoint that fall outside of 1.5 times od Inter Quartile Range above 1st  the quartile and the 3rd quartile.

Inter Quartile Range

 import numpy as np  
 box = [10,20,15,25,11,12,16,26,19,29,3000]  
 box = sorted(box)  
 Q1,Q3 = np.percentile(box, [25,75])  
 iqr = Q3 - Q1  
 # Lower bound and higher bound values  
 lower_bound_val = Q1 - (1.5*iqr)  
 higher_bound_val = Q3 + (1.5*iqr)  


Using Box-plot:

The values which are beyond the Max or Min are considered as the outliers.

Box plot

   for feature in data:  
   dataset = data.copy()  
   if 0 in dataset[feature].unique():  
     dataset[feature] = np.log(dataset[feature])  

 Dropping outliers with the Standard Deviation: 

 fact = 3  
 upper_bound_val = data['column'].mean() + data['column'].std() * fact  
 lower_bound_val = data['column'].mean() - data['column'].std() * fact  
 data = data[(data['column'] < upper_bound_val) & (data['column'] > lower_bound_val)]
 data = pd.DataFrame(np.random.randn(500,4))
from scipy import stats
data[(np.abs(stats.zscore(data)) < 3).all(axis=1)]

 Dropping the outliers rows with percentile: 

 upper_bound_val = data['column'].quantile(0.95)  
 lower_bound_val = data['column'].quantile(0.05)  
 data = data[(data['column'] < upper_bound_val) & (data['column'] > lower_bound_val)]  


3.) Binning

Base to Binned data

Image Credit

-Binning can be applied to both categorical and numerical data.

- The main reason behind binning is to make the model more robust and prevent overfitting, however, it has a performance cost.


#Numerical Binning Example

Value      Bin      

0-30   =>  Fail      

31-70  =>  Average      

71-100 =>  Excelent


#Categorical Binning Example

Value                   Bin      

Mumbai   =>  Maharashtra     

Pune     =>  Maharashtra      

Bikaner  =>  Rajasthan

Jaipur   =>  Rajasthan


Numerical Binning Example:

data['bin'] = pd.cut(data['value'], bins=[0,30,70,100], labels=["Fail", "Average", "Excelent"]) 


   value    | bin

0 |  2    | Fail
1 | 45   | Average
2 |  7    | Fail
3 |  85  | Excellent
4 |  28  | Fail


Categorical Binning Example:

  conditions = [  
  choices = ['Maharashtra', 'Maharashtra', 'Rajasthan', 'Rajasthan']  
  data['Continent'] =, choices, default='Other') 


  value    | bin 

0 |  Mumbai |  Maharashtra
1 | Pune     |  Maharashtra
2 | Bikaner   |  Rajasthan
3 |   Delhi        | Other
4 |  Jaipur  | Rajasthan


4.)Techniques of dealing with Gaussian-Distribution / Skewness


Skewed data

-Helps to handles skewed data and after transformation, the distribution becomes more approximate to the normal.

-Decreases the effect of the outliers to the normalization of magnitude difference and the model becomes more robust.


(i)Log Transformation:

Log transformation is a data transformation method in which it replaces by log(x) with base 10, base 2, or natural log.

 data['log_column'] = np.log(data['column']+1)  

(ii)Reciprocal Transformation:

In Reciprocal Transformation, x will replace by the inverse of X(1/X). The reciprocal transformation will give little effect on the shape of the distribution. This transformation can be only used for non-zero values. The skewness for the transformed data is increased.

 data['Reciprocal_column'] = 1/(data['column']+1)  

(iii)Square-Root Transformation:

This transformation will give a moderate effect on distribution. The main advantage of square root transformation is, it can be applied to zero values. Here the x will replace by the square root(x). It is weaker than the Log Transformation.

  data['sqr_column'] = data['column']**(1/2)  

5.)OneHotEncoding and OrdinalEncoder

One Hot Encoding Example

Since our computer systems only understand numerical data. So categorical data makes no sense for them unless they are converted into the numerical data and for that we use techniques such as OneHotEncoding and OrdinalEncoders.

# For single column

  encoded_col = pd.get_dummies(data['column'], drop_first = True)  

 # For multi column

  encoded_multi_col = pd.get_dummies(data,columns = ['column1', 'column2', 'columnN'], drop_first=True)

 The above mentioned is too naïve way of doing this, but we are ninjas we have a secret technique to get the optimum result.

For unordered (Nominal) data we will be using OneHotEncoding

Example: Male, Female

 # OneHotEncoding is for unordered (Nominal) data  
 # Example: Male, Female  
 from sklearn.preprocessing import OneHotEncoder  
 ohe = OneHotEncoder(sparse=False)  

For ordered (Ordinal) data we will be using OrdinalEncoder

Example: First, Second, Third

 # OrdinalEncoder is for ordered (Ordinal) data  
 # Example: First, Second, Third  
 from sklearn.preprocessing import OrdinalEncoder  
 oe = OrdinalEncoder(categories=[['First', 'Second', 'Third'],  
 oe.fit_transform(data[['Rank', 'Grade']])  

6.)Feature splitting & extraction

Data Splitting and Extraction

-By extracting the utilizable part of a column into a new feature:

*We enable the machine-learning algorithms to comprehend them.

*Make possible to bin and group them.

*Improve model performance by uncovering potential information.

Splitting a function is a good option, but however, there is one way of splitting features.

It depends on the characteristics of the column, how to split it.

Feature Splitting:

 import pandas as pd  
 import numpy as np  
 data = [('Arnold Schwarzenegger','M'),   
     ('Natasha Romanova','F'),   
     ('Sylvester Stallone','M'),   
     ('Gal Gadot','F'),   
     ('Dwayne Johnson','M')]  
 df = pd.DataFrame(data, columns=['name', 'gender'])  

output 1

 # Extracting First Name" ").map(lambda X : X[0])  
First name output

 # Extracting Last Name" ").map(lambda X : X[-1])  
Last name output

Feature Extraction:

 weather_data = [('1/1/2017', 32, 6, 'Rain'),  
         ('1/2/2017', 30, 7, 'Sunny'),  
         ('1/3/2017', 32, 2, 'Snow'),  
         ('1/4/2017', 34, 6, 'Snow'),  
         ('1/5/2017', 32, 4, 'Rain'),  
         ('1/6/2017', 32, 2, 'Sunny')  
 df = pd.DataFrame(weather_data, columns=['day', 'temp', 'windspeed', 'weather'])  
weather data


date when it rained[df.temp == df.temp.max()]  

max temperature

7.) Group By

Group data

groupby() function is used to split the data into groups based on some criteria. pandas objects can be split on any of their axes. The abstract definition of grouping is to provide a mapping of labels to group names.

  weather_data = [('1/1/2017', 'Mumbai' , 32, 6, 'Rain'),  
         ('1/2/2017', 'Pune' , 30, 7, 'Sunny'),  
         ('1/3/2017', 'Mumbai' , 32, 2, 'Snow'),  
         ('1/4/2017', 'Pune' , 34, 6, 'Snow'),  
         ('1/5/2017', 'Mumbai' , 32, 4, 'Rain'),  
         ('1/6/2017', 'Delhi' , 32, 2, 'Sunny')  
 data = pd.DataFrame(weather_data, columns=['day', 'city', 'temp', 'windspeed', 'weather'])  

weather data

 grp_city = data.groupby('city')  
 for city, city_data in grp_city:  

City Data

 # Get specific group  

get specific group


City Max


City Mean


Group city describe

8.) Concat, Merge, Join

Concat, Merge, &  Join

Works like how we had worked on SQL.

  Maharashtra_weather_data = pd.DataFrame({  
   'city': ['Mumbai', 'Pune', 'Thane'],  
   'temperature': [32, 45, 30],  
   'humidity': [80, 60, 78]  

Maharashtra weather data

 Gujarat_weather_data = pd.DataFrame({  
   'city': ['Surat', 'Rajkot', 'Mehsana'],  
   'temperature': [21, 24, 35],  
   'humidity': [68, 65, 75]  

Gujarat weather data

(i) Concat:

-Column Wise Concatenation

 df = pd.concat([Maharashtra_weather_data, Gujarat_weather_data])  

-Row Wise Concatenation

 df = pd.concat([Maharashtra_weather_data, Gujarat_weather_data], axis=1)  

Row wise

(ii) Merge

  temp_data = pd.DataFrame({  

Temperature city

 humidity_data = pd.DataFrame({  

Humidity city

 # Merge 2 dataframe without explicitly mentioning the index  
 df = pd.merge(temp_data, humidity_data, on='city')  

Merge Data

(iii) Join

 df = pd.merge(temp_data, humidity_data, on='city',how='outer')  

join data

9.) Scaling

Data Scaling

Scaling  are done basically in 2 ways first is Normalization and the second is Standardization.

 data = pd.DataFrame({  
   'name':['Ram', 'Lakhan', 'Shiva', 'Ria', 'Lucy', 'Suraj', 'Rohan', 'Anny', 'Priya', 'Niraj'],  
   'Salary':[25000, 32000, 50000, 33000, 20000, 29000, 22000, 52000, 23000, 26000],  

Data for Normalization

(i) Normalization (MinMaxScaler)

* X_norm = X – X_min / X_max – X_min

* Range 0 to 1

* Due to the decrease in standard deviation the effect of outliers increases.

*Before Normalization, it is recommended to handle the outliers.

 from sklearn.model_selection import train_test_split  
 X = data.drop(['name','purchased'], axis=1)  
 y = data['purchased']  
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) 

 from sklearn.preprocessing import MinMaxScaler  

 Min_Max_scaler = MinMaxScaler()  
 Min_Max_X_train = Min_Max_scaler.fit_transform(X_train)  
 Min_Max_X_test = Min_Max_scaler.transform(X_test)  

Min Max Scaled Train


Min Max Scaled Test 

(ii) Standardization (StandardScaler)


*Z = X - μ / σ

*μ = 0 and σ = 1

*Range -1 to 1

*If the standard deviation of feature is different, their range also would differ from each other.

 from sklearn.preprocessing import StandardScaler  
 StandardScaler_scaler = StandardScaler()  
 StandardScaler_X_train = StandardScaler_scaler.fit_transform(X_train)  
 StandardScaler_X_test = StandardScaler_scaler.transform(X_test)  

Standard Scaler Train


Standard Scaler Test

10.) Extracting Date


There are 3 ways we can preprocessing date

*Extracting the parts of the date into the different columns: YEAR, MONTH, DAY.

*Extracting the time period between the current date and column in terms of YEAR, MONTH, DAY.

*Extracting some specific features from the date such as the name of the weekday, weekend, etc.

*With dealing with date features like that our machine learning model can easily understand data and deal with the data.

 from datetime import date  
 data = pd.DataFrame({  
   'date': ['01/01/2017', '04/12/2000', '23/04/2011', '11/02/2008', '08/08/2018']  

Time series data

Time series data info

 # Transform String to Date  
 data['date'] = pd.to_datetime(, format='%d/%m/%Y')

Transforming string to date

 # Extracat year  
 data['year'] = data['date'].dt.year  

Year extraction

 # Extract month  
 data['month'] = data['date'].dt.month  

Month extraction

 # Extract day  
 data['day'] = data['date']  

Day extraction

 # Extracting passed year since the date   
 data['passed_years'] = - data['date'].dt.year  

Passed Year

 # Extracting passed month since the date   
 data['passed_months'] = ( - data['date'].dt.year)*12 + ( - data['date'].dt.month)  

Extracting passed month since the date

 Here is the Github link of all the code which is mentioned here:

Github Link of the Notebook: Link

Use all the tips and tricks which are mentioned above and you will surely get amazing data to train your model on.

So we hope that you enjoyed this session. If you did then please share it with your friends and spread this knowledge. 

Follow us at :

Instagram :

Facebook :

Twitter :


No comments:

No Spamming and No Offensive Language

Powered by Blogger.