Assignment - 1 (Machine Learning)
Linear Regression
Author: Jimut Bahan Pal
As usual, importing the necessary libraries¶
In [76]:
import numpy as np # just an opensource version of mini-matlab for matrix mult and stuffs
import matplotlib.pyplot as plt # plotting library
import matplotlib.patches as mpatches # cool patches
from mpl_toolkits.mplot3d import Axes3D # 3D stuffs
from sklearn import datasets, linear_model # ML lib
from sklearn.datasets import make_regression # autocreation of dataset and stuffs
from sklearn.linear_model import LinearRegression # Linear Regression module
from sklearn.preprocessing import PolynomialFeatures # ploynomial regression
from sklearn.metrics import mean_squared_error, r2_score # for generating RMSE and other metrics for eval.
Part - A¶
Creation of 1D dataset¶
Artificially generated 1D dataset, using y = $m$x + $\epsilon$. Here, Gaussian noise $\epsilon$ = 6.
In [77]:
X, y = make_regression(n_samples=100, n_features=1, noise=6)
In [78]:
# plot regression dataset
fig, ax = plt.subplots()
ax.set_title('The plot for the whole dataset')
scatter = ax.scatter(X,y,color='#4224eb')
handles, labels = scatter.legend_elements(prop="sizes", alpha=0.6)
red_patch = mpatches.Patch(color='#4224eb', label='1D dataset')
plt.legend(handles=[red_patch],loc="lower right", title="Legend")
ax.grid(True)
plt.show()
In [79]:
# Split the data into training/testing sets
X_train = X[:-20]
X_test = X[-20:]
# Split the targets into training/testing sets
y_train = y[:-20]
y_test = y[-20:]
In [80]:
# Create linear regression object
regr = LinearRegression()
# Train the model using the training sets
regr.fit(X_train, y_train)
Out[80]:
In [81]:
# Make predictions using the testing set
y_pred = regr.predict(X_test)
print(y_pred)
Root mean squared error¶
In [82]:
# The coefficients
print('Coefficients: \n', regr.coef_)
# The mean squared error
print('Root Mean squared error [RMSE]: %.2f'
% np.sqrt(mean_squared_error(y_test, y_pred)))
# The coefficient of determination: 1 is perfect prediction
print('Coefficient of determination: %.2f'
% r2_score(y_test, y_pred))
In [83]:
# Plot outputs
fig, ax = plt.subplots()
ax.set_title('The plot for the test set')
scatter = ax.scatter(X_test, y_test,color='#4224eb')
handles, labels = scatter.legend_elements(prop="sizes", alpha=0.6)
red_patch = mpatches.Patch(color='#4224eb', label='Test set')
plt.legend(handles=[red_patch],loc="lower right", title="Legend")
plt.plot(X_test, y_pred, color='#f90909', linewidth=3)
# plt.xticks(())
# plt.yticks(())
ax.grid(True)
plt.show()
Part - B¶
Creation of 4D dataset¶
${h _\theta }\left( x \right) = {\theta _0} + {\theta _1}{x _1} + {\theta _2}x _1^2 + {\theta _3}x _1^3 + ... + {\theta _n}x _1^n$
In [84]:
X, y = make_regression(n_samples=100, n_features=4, noise=6)
So we can see this has 4 features, i.e. 4 columns¶
In [85]:
X
Out[85]:
We can select the first column .i.e. $x_1$¶
In [86]:
X[:,0]
Out[86]:
Creating the train and test dataset¶
In [87]:
# Split the data into training/testing sets
X_train = X[:-20]
X_test = X[-20:]
# Split the targets into training/testing sets
y_train = y[:-20]
y_test = y[-20:]
Training¶
In [88]:
polynomial_features= PolynomialFeatures(degree=4)
x_poly = polynomial_features.fit_transform(X_train)
x_test_poly = polynomial_features.fit_transform(X_test)
model = LinearRegression()
model.fit(x_poly, y_train)
y_poly_pred = model.predict(x_test_poly)
Root Mean Squared Error¶
In [89]:
rmse = np.sqrt(mean_squared_error(y_test,y_poly_pred))
r2 = r2_score(y_test,y_poly_pred)
print("Root Mean Squared Error [RMSE] : ",rmse)
print("R-2 score : ",r2)
One of the problems for visualising 4D data is, it can't be visualised spatially, so, for the sake of experiment, I have tried to use heatmap as another dimension. Just trying to recreate this stuff. Column 1,2,3 and 4 are the four features and they represent some of the dimension in this plot.¶
In [90]:
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
img = ax.scatter(X_test[:,0], X_test[:,1], X_test[:,2], c=X_test[:,3], cmap=plt.hot())
fig.colorbar(img)
plt.show()
Now a struggle to display the plot for 4D surface. Just trying to recreate this stuff for visualising 4D.¶
In [91]:
import matplotlib
from scipy.interpolate import griddata
from matplotlib import cm
name_color_map = 'seismic';
list_name_variables = ['x', 'y', 'z', 'c'];
index_x = 0; index_y = 1; index_z = 2; index_c = 3;
# X-Y are transformed into 2D grids. It's like a form of interpolation
X_test_1 = np.linspace(X_test[:,0].min(), X_test[:,0].max(), len(np.unique(X_test[:,0])));
y_test = np.linspace(y_test.min(), y_test.max(), len(np.unique(y_test)));
x2, y2 = np.meshgrid(X_test_1, y_test);
# Interpolation of Z: old X-Y to the new X-Y grid.
# Note: Sometimes values can be < z.min and so it may be better to set
# the values too low to the true minimum value.
z2 = griddata( (X_test[:,0], y_test), X_test[:,1], (x2, y2), method='cubic', fill_value = 0);
z2[z2 < X_test[:,1].min()] = X_test[:,1].min();
# Interpolation of C: old X-Y on the new X-Y grid (as we did for Z)
# The only problem is the fact that the interpolation of C does not take
# into account Z and that, consequently, the representation is less
# valid compared to the previous solutions.
c2 = griddata( (X_test[:,0], y_test), X_test[:,2], (x2, y2), method='cubic', fill_value = 0);
c2[c2 < X_test[:,2].min()] = X_test[:,2].min();
#--------
color_dimension = c2; # It must be in 2D - as for "X, Y, Z".
minn, maxx = color_dimension.min(), color_dimension.max();
norm = matplotlib.colors.Normalize(minn, maxx);
m = plt.cm.ScalarMappable(norm=norm, cmap = name_color_map);
m.set_array([]);
fcolors = m.to_rgba(color_dimension);
# At this time, X-Y-Z-C are all 2D and we can use "plot_surface".
fig = plt.figure(); ax = fig.gca(projection='3d');
surf = ax.plot_surface(x2, y2, z2, facecolors = fcolors, linewidth=0, rstride=1, cstride=1,
antialiased=False);
cbar = fig.colorbar(m, shrink=0.5, aspect=5);
cbar.ax.get_yaxis().labelpad = 15; cbar.ax.set_ylabel(list_name_variables[index_c], rotation = 270);
ax.set_xlabel(list_name_variables[index_x]); ax.set_ylabel(list_name_variables[index_y]);
ax.set_zlabel(list_name_variables[index_z]);
plt.title('%s in fcn of %s, %s and %s' % (list_name_variables[index_c], list_name_variables[index_x], list_name_variables[index_y], list_name_variables[index_z]) );
plt.show();
References¶
- Dripta Maharaj, (2020), Slides, availabe on web https://sites.google.com/view/da220-2019-20 , last accessed on 16.1.2020.
- Brownlee, J., (2018), How to Generate Test Datasets in Python with scikit-learn, availabe on web https://machinelearningmastery.com/generate-test-datasets-python-scikit-learn/ , last accessed on 16.1.2020.
- Scikit-learn documentation, Linear Regression Example, availabe on web https://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html#sphx-glr-auto-examples-linear-model-plot-ols-py , last accessed on 16.1.2020.
- user2386081, (2013), Matplotlib scatter plot legend, https://stackoverflow.com/questions/17411940/matplotlib-scatter-plot-legend , last accessed on 16.1.2020.
- Dixon & Moe, (2015), Searching for that perfect color has never been easier, use our HTML color picker to browse millions of colors and color harmonies. https://htmlcolorcodes.com/color-picker/ , last accessed on 16.1.2020.
- Matplotlib documentation, Scatter plots with a legend, https://matplotlib.org/3.1.1/gallery/lines_bars_and_markers/scatter_with_legend.html , last accessed on 16.1.2020.
- Tengis, (2013), How to make a 4d plot with matplotlib using arbitrary data, https://stackoverflow.com/questions/14995610/how-to-make-a-4d-plot-with-matplotlib-using-arbitrary-data , last accessed on 16.1.2020.
- Agarwal, A., (2018), Polynomial Regression, Towards datascience, https://towardsdatascience.com/polynomial-regression-bbe8b9d97491 , last accessed on 16.1.2020 .
Acknowledgements¶
- Dripta Maharaj