Gfyuki

I am doing a random forest regression on my dataset (which has abut 15 input features and 1 target feature). I am getting a decently low R^2 of <1 for both the train and test sets (please do let me know if <1 is not a good-enough R^2 score).

import pandas as pd

import numpy as np

from sklearn.ensemble import RandomForestRegressor

from sklearn.model_selection import train_test_split



# load dataset

df = pd.read_csv('Dataset.csv')



# split into input (X) and output (Y) variables

X = df.drop(['ID_COLUMN', 'TARGET_COLUMN'], axis=1)

Y = df.TARGET_COLUMN



# Split the data into 67% for training and 33% for testing

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.33)



# Fitting the regression model to the dataset

regressor = RandomForestRegressor(n_estimators = 100, random_state = 50)

regressor.fit(X_train, Y_train.ravel()) # Using ravel() to avoid getting 'DataConversionWarning' warning message





print("Predicting Values:")

y_pred = regressor.predict(X_test)



print("Getting Model Performance...")



# Get regression scores

print("R^2 train = ", regressor.score(X_train, Y_train))

print("R^2 test = ", regressor.score(X_test, Y_test))

This outputs the following:

Predicting Values:

Getting Model Performance...

R^2 train =  0.9791000275450427

R^2 test = 0.8577464692386905

Then, I checked the difference between the actual target column values in the test dataset versus the predicted values, like so:

diff = 

for i in range(len(y_pred)):

    if Y_test.values[i]!=0: # a few values were 0 which was causing the corresponding diff value to become inf

        diff.append(100*np.abs(y_pred[i]-Y_test.values[i])/Y_test.values[i]) # element-wise percentage error

I found that the majority of the element-wise differences were between 40-60% and their mean was almost 50%!

np.mean(diff)

>>> 49.07580695857447

So, which one is correct? Is the regression score correct and my model is good for this data, or is the element-wise error I calculated correct and the model didn't do well for this data? If its the latter, please advise on how to increase the prediction accuracy.

I also checked the rmse score:

import math

rmse = math.sqrt(np.mean((np.array(Y_test) - y_pred)**2))

rmse

>>> 3.67328471827293

This seems quite high for the model to have done a good job, but please correct me if I'm wrong.

And I also checked the R^2 scores for different number of estimators:

import matplotlib.pyplot as plt

model = RandomForestRegressor(n_jobs=-1)

# Try different numbers of n_estimators

estimators = np.arange(10, 200, 10)

scores = 

for n in estimators:

    model.set_params(n_estimators=n)

    model.fit(X_train, Y_train)

    scores.append(model.score(X_test, Y_test))

plt.title("Effect of n_estimators")

plt.xlabel("n_estimator")

plt.ylabel("score")

plt.plot(estimators, scores)

Please advise.

I tried using linear regression first, and got a very high MSE (which is why I was trying out random forest):

from sklearn.linear_model import LinearRegression

from sklearn.metrics import mean_squared_error, r2_score



lr = LinearRegression()

lr.fit(X_train, y_train)

y_pred = lr.predict(X_test)



# The coefficients

print('Coefficients: n', lr.coef_)

# The mean squared error

print("Mean squared error: %.2f" % mean_squared_error(y_test, y_pred))

# Explained variance score: 1 is perfect prediction

print('Variance score: %.2f' % r2_score(y_test, y_pred))





Coefficients: 

 [ 1.93829229e-01 -4.68738825e-01  2.01635420e-01  6.35902010e-01

  6.57354434e-03  5.13180293e-03  2.84015810e-01 -1.31469084e-06

  1.95335035e+00]

Mean squared error: 86.92

Variance score: 0.08

This line looks wrong to me:

diff.append(100*np.abs(y_pred[i]-Y_test.values[i])/Y_test.values[i])

Shouldn't the abs be around the entire calculation?

diff.append(100*np.abs((y_pred[i]-Y_test.values[i])/Y_test.values[i]))

That aside, the RMSE calculation looks accurate and is in the scale of the error, and the $R^2$ is great, so all things being equal, I would lean towards looking for something you did wrong in assessing the errors. That's why I was focused on your calculation.

One other thought, have you checked for outliers? This could affect some measures and not others as drastically.