1. Introduction¶
1.1 Project Summary¶
To ensure the safety and reliability of each and every unique car configuration before they hit the road, Daimler’s engineers have developed a robust testing system. But, optimizing the speed of their testing system for so many possible feature combinations is complex and time-consuming without a powerful algorithmic approach. As one of the world’s biggest manufacturers of premium cars, safety and efficiency are paramount on Daimler’s production lines.
1.2 Objective¶
The objective is to tackle the curse of dimensionality and reduce the time cars spend on test bench.
1.3 Data Dictionary¶
The dataset contains variables that are anonymus each representing a custome feature in a Mercedes car. The ground truth is labeled ‘y’ and represents the time (in seconds) that the car took to pass testing for each variable.
Variables with letters are categorical. Variables with 0/1 are binary values.
- train.csv - the training set
- test.csv - the test set, you must predict the 'y' variable for the 'ID's in this file
- sample_submission.csv - a sample submission file in the correct format
1.4 Steps¶
We will be performing various steps to achieve the objective
- Loading data
- Exploratory data analysis
- Data wrangling
- Modeling and Predicting
- Formatting output
""" importing required packages """
%matplotlib inline
""" packages for data manipulation, munging and merging """
import pandas as pd
import numpy as np
""" packages for visualiztion and exploratory analysis """
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style="whitegrid", color_codes=True)
""" packages for running machine learning algorithms """
from time import time
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVR, LinearSVC
from sklearn.ensemble import RandomForestRegressor
from sklearn.neighbors import KNeighborsRegressor
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from xgboost import XGBClassifier
from sklearn.decomposition import PCA
from sklearn.cluster import FeatureAgglomeration
from sklearn.feature_selection import VarianceThreshold
Loading data¶
We will be loading training and test datasets using pandas read_csv function.
""" file paths """
train_file = 'train.csv'
test_file = 'test.csv'
""" function to read training data into pandas dataframe """
def read_training_data(filepath):
train_df = pd.read_csv(filepath)
return train_df
""" function to read test data into pandas dataframe """
def read_test_data(filepath):
test_df = pd.read_csv(filepath)
return test_df
""" calling function to read training data into a dataframe """
train_df = read_training_data(train_file)
test_df = read_test_data(test_file)
def data_summary(df):
print(df.shape)
print(df.info())
print(df.head(5))
""" calling functions to print summary statistics of training data """
data_summary(train_df)
data_summary(test_df)
In the training dataset, there are 4209 rows with 378 columns.
- ground truth varible y is of type float
- X0,X1,X2,X3,X4,X5,X6,X8 are of type object
- rest of the columns are int type
We will convert [X0,X1,X2,X3,X4,X5,X6,X8] to categorical types and plot to see the distribution of values.
Imputing missing values¶
We will check if there are any missing values in the training and testing datasets. If there are any we will use suitable methods to impute missing values.
def check_missing_values(df):
if df.isnull().any().any():
print("There are missing values in the dataframe")
else:
print("There are no missing values in the dataframe")
""" calling functions to check missing values on training and test datasets """
check_missing_values(train_df)
check_missing_values(test_df)
Looks like there are no missing values in the dataframe.
def initial_datatype_conversion(df):
cols = ['X0','X1','X2','X3','X4','X5','X6','X8']
for col in cols:
df[col] = df[col].astype('category')
return df
""" datatype conversion """
ret_train_df = initial_datatype_conversion(train_df)
ret_test_df = initial_datatype_conversion(test_df)
""" combining categorical attributes from training and test datasets """
train_df_cat = ret_train_df.loc[:,['X0','X1','X2','X3','X4','X5','X6','X8']]
test_df_cat = ret_test_df.loc[:,['X0','X1','X2','X3','X4','X5','X6','X8']]
train_df_cat = train_df_cat.add_prefix('train_')
test_df_cat = test_df_cat.add_prefix('test_')
combined = train_df_cat.append(test_df_cat, ignore_index=True)
Exploratory Data Analysis¶
def visualize_categories(df, **kwargs):
row = kwargs.get('row',None)
col = kwargs.get('col',None)
hue = kwargs.get('hue',None)
df_types = ['train_','test_']
y_val = ['X0','X1','X2','X3','X4','X5','X6','X8']
for df_type in df_types:
for val in y_val:
yval = df_type + val
plt.figure()
sns.countplot(y=yval, data=df, color="c");
plt.show()
visualize_categories(combined)
The plots reveal that values among X0, X1, X2, X5, X6 and X8 are fairly distributed, where as values in X3 is moderately distributed. X4 seems to have only one value 'd'. If X4 is part of the training features dataset, the model outcomes may be skewed. In order to avoid that, we will de dropping it from the training and test datasets that will be passed to modeling step.
The character values in X0, X1, X2, X5, X6 and X8 are converted to numerical values so that we can use them in modeling steps.
def value_assignment(df):
unique_values = np.unique(df[['X0', 'X1', 'X2','X3','X4','X5','X6','X8']].values)
dict_vals = {'a':0,'b':1,'c':2,'d':3,'e':4,'f':5,'g':6,'h':7,'i':8,'j':9,'k':10,'l':11,'m':12,'n':13,'o':14,
'p':15,'q':16,'r':17,'s':18,'t':19,'u':20,'v':21,'w':22,'x':23,'y':24,'z':25,'aa':26,'ab':27,
'ac':28,'ad':29,'ae':30,'af':31,'ag':32,'ah':33,'ai':34,'aj':35,'ak':36,'al':37,'am':38,'an':39,
'ao':40,'ap':41,'aq':42,'ar':43,'as':44,'at':45,'au':46,'av':47,'aw':'48','ax':49,'ay':50,'az':51,
'ba':52,'bb':53,'bc':54}
cols = ['X0', 'X1', 'X2','X3','X4','X5','X6','X8']
for col in cols:
df[col] = df[col].astype('object')
df[col].replace(dict_vals, inplace=True)
#df[col] = df[col].astype('int64')
df[cols] = df[cols].apply(pd.to_numeric, errors='coerce')
return df
""" calling value_assignment function to assign numerical values to categorical attributes """
train_df_prep = value_assignment(ret_train_df)
test_df_prep = value_assignment(ret_test_df)
Modeling and Prediction¶
""" data preparation for modeling and prediction steps """
train_X = train_df_prep.drop(['ID','y','X4'], axis=1)
train_Y = train_df_prep['y']
train_Y = train_Y.values
test_X = test_df_prep.drop(['ID','X4'],axis=1)
def perform_svc(train_X, train_Y, test_X):
svr_clf = SVR()
svr_clf.fit(X=train_X, y=train_Y)
pred_Y = svr_clf.predict(test_X)
accuracy = round(svr_clf.score(train_X, train_Y) * 100, 2)
returnval = {'model':'SVR', 'accuracy':accuracy}
return returnval
Random Forest Regressor¶
def perform_rfc(df_X, df_Y, test_df_X):
rfr_clf = RandomForestRegressor(n_estimators = 100 ,oob_score=True, max_features="auto")
rfr_clf.fit(df_X, df_Y)
pred_Y = rfr_clf.predict(test_df_X)
accuracy = round(rfr_clf.score(df_X, df_Y) * 100, 2)
returnval = {'model':'RandomForestRegressor','accuracy':accuracy}
return returnval
K-Nearest Neighbors Regressor¶
def perform_knn(df_X, df_Y, test_df_X):
knn = KNeighborsRegressor(n_neighbors=5)
knn.fit(df_X, df_Y)
pred_Y = knn.predict(test_df_X)
accuracy = round(knn.score(df_X, df_Y) *100,2)
returnval = {'model':'KNeighborsRegressor','accuracy':accuracy}
return returnval
Naive Bayes¶
def perform_gnb(df_X, df_Y, test_df_X):
gnb = GaussianNB()
gnb.fit(df_X, df_Y)
pred_Y = gnb.predict(test_df_X)
accuracy = round(gnb.score(df_X, df_Y)*100,2)
returnval = {'model':'GaussianNB','accuracy':accuracy}
return returnval
XGBoost¶
def perform_xgboost(df_X, df_Y, test_df_X):
model = XGBClassifier()
model.fit(df_X, df_Y)
pred_Y = model.predict(test_df_X)
accuracy = round(model.score(df_X, df_Y)*100,2)
returnval = {'model':'XGBoost','accuracy':accuracy}
return returnval
Dimensionality Reduction Techniques¶
Principal component analysis (PCA)¶
Linear dimensionality reduction using Singular Value Decomposition (SVD) of the data to project it to a lower dimensional space. There are 4 types of SVD solvers
auto : the solver is selected by a default policy based on X.shape and n_components: if the input data is larger than 500x500 and the number of components to extract is lower than 80% of the smallest dimension of the data, then the more efficient ‘randomized’ method is enabled. Otherwise the exact full SVD is computed and optionally truncated afterwards
full: run exact full SVD calling the standard LAPACK solver via scipy.linalg.svd and select the components by postprocessing
arpack: run SVD truncated to n_components calling ARPACK solver via scipy.sparse.linalg.svds. It requires strictly 0 < n_components < X.shape[1]
randomized: run randomized SVD by the method of Halko et al.
We will run principal component analysis for multiple n_component values to see how the models get affected.
def perform_pca(train_X, train_Y, test_X):
n_components = [10]
#solvers = ['auto']
pca_model_accuracies = pd.DataFrame()
pca = PCA(n_components = 10,svd_solver='auto')
pca.fit(train_X)
train_X_pca = pca.transform(train_X)
test_X_pca = pca.transform(test_X)
svc_acc_val = perform_svc(train_X_pca, train_Y, test_X_pca)
rfc_acc_val = perform_rfc(train_X_pca, train_Y, test_X_pca)
knn_acc_val = perform_knn(train_X_pca, train_Y, test_X_pca)
pca_model_accuracies = pca_model_accuracies.append([svc_acc_val,rfc_acc_val,knn_acc_val])
cols = list(pca_model_accuracies.columns.values)
cols = cols[-1:] + cols[:-1]
pca_model_accuracies = pca_model_accuracies[cols]
pca_model_accuracies = pca_model_accuracies.sort_values(by='accuracy')
print(pca_model_accuracies)
""" calling function to perform principal component analysis """
perform_pca(train_X, train_Y, test_X)
Feature Agglomeration¶
Similar to AgglomerativeClustering, but recursively merges features instead of samples.
Default connectivity value is None, i.e, the hierarchical clustering algorithm is used which is unstructured.
def perform_feature_agglomeration(train_X, train_Y, test_X):
n_clusters = [32]
fagg_model_accuracies = pd.DataFrame()
for n_cluster in n_clusters:
agglo = FeatureAgglomeration(connectivity=None, n_clusters=n_cluster)
agglo.fit(train_X)
train_X_reduced = agglo.transform(train_X)
test_X_reduced = agglo.transform(test_X)
svc_acc_val = perform_svc(train_X_reduced, train_Y, test_X_reduced)
svc_acc_val['no of clusters'] = n_cluster
rfc_acc_val = perform_rfc(train_X_reduced, train_Y, test_X_reduced)
rfc_acc_val['no of clusters'] = n_cluster
knn_acc_val = perform_knn(train_X_reduced, train_Y, test_X_reduced)
knn_acc_val['no of clusters'] = n_cluster
fagg_model_accuracies = fagg_model_accuracies.append([svc_acc_val,rfc_acc_val,knn_acc_val])
cols = list(fagg_model_accuracies.columns.values)
cols = cols[-1:] + cols[:-1]
fagg_model_accuracies = fagg_model_accuracies[cols]
fagg_model_accuracies = fagg_model_accuracies.sort_values(by='accuracy')
print(fagg_model_accuracies)
""" calling function to perform feature agglomeration """
perform_feature_agglomeration(train_X, train_Y, test_X)
Variance Threshold¶
def perform_variance_threshold(train_X, train_Y, test_X):
vt_model_accuracies = pd.DataFrame()
selector = VarianceThreshold(threshold=0.03)
selector.fit(train_X)
train_X_reduced = selector.transform(train_X)
test_X_reduced = selector.transform(test_X)
svc_acc_val = perform_svc(train_X_reduced, train_Y, test_X_reduced)
rfc_acc_val = perform_rfc(train_X_reduced, train_Y, test_X_reduced)
knn_acc_val = perform_knn(train_X_reduced, train_Y, test_X_reduced)
vt_model_accuracies = vt_model_accuracies.append([svc_acc_val,rfc_acc_val,knn_acc_val])
cols = list(vt_model_accuracies.columns.values)
cols = cols[-1:] + cols[:-1]
vt_model_accuracies = vt_model_accuracies[cols]
vt_model_accuracies = vt_model_accuracies.sort_values(by='accuracy')
print(vt_model_accuracies)
""" calling function to perform variance threshold analysis """
perform_variance_threshold(train_X, train_Y, test_X)
Output¶
- Model accuracy rates based on PCA show that values do not differ much between full, arpack and random solver types. Among the models used with reduced dimensionality datasets from PCA, random forest classifier has the highest accuracy rate.
- Model accuracy rates based on Feature Agglomeration show gradual increase in values as n_clusters value increases. Among the models Random Forest Classifier generates highest accuracy rate of 93.61 for n_clusters = 32.
- Model accuracy rates based on Variance Threshold with threshold = 0.03 returns highest accuracy rate for random forest classifier which is 93.61.
def write_to_csv(train_X, train_Y, test_X, test_df):
agglo = FeatureAgglomeration(connectivity=None, n_clusters=32)
agglo.fit(train_X)
train_X_reduced = agglo.transform(train_X)
test_X_reduced = agglo.transform(test_X)
rfr_clf = RandomForestRegressor(n_estimators = 100 ,oob_score=True, max_features=None)
rfr_clf.fit(train_X_reduced, train_Y)
pred_Y = rfr_clf.predict(test_X_reduced)
pred_Y_list = pred_Y.tolist()
test_X['y'] = pred_Y
test_X['ID'] = test_df['ID']
final_df = test_X[['ID','y']]
final_df.to_csv('submission_1.csv',sep=',',index=False)
write_to_csv(train_X, train_Y, test_X, test_df_prep)
Summary¶
This is my first competition besides titanic machine learning exercise. I had a great time working on this competition. Since I am still trying to get a grip of machine learning concepts and techniques, I request fellow kagglers to provide constructive feedback to improve myself.