Gfyuki

Hello and thank you in advance for any answer. I am building a NN for a multi class classification problem. I have pretrained my data through a Word2vec and generated a 300 dimension vector with values. I have 6 classes .. My data consist of 15000 rows(x300 dimensions). My first question is :

What is the number of units? Is it something that we can extract from theory?
Also i have managed to gain 0.85-0.90 training set accuracy but the validation set accuracy is always low: 0.22-0.25. I do not know what way to follow as I am newbie to these kind of stuff. My code is:

import numpy as np

import pandas as pd

import keras

from sklearn.preprocessing import Imputer

from sklearn.model_selection import train_test_split

from keras.models import Sequential

from keras.layers import Dense,Dropout

from sklearn.metrics import confusion_matrix

import matplotlib.pyplot as plt

from keras import regularizers







dataset = pd.read_csv('word2vec.csv')





X = dataset.iloc[:, 0:300].values

y = dataset.iloc[:, 300].values



imputer = Imputer(missing_values = 'NaN', strategy = 'mean', axis = 0)

imputer = imputer.fit(X[:, 0:300])

X[:, 0:300] = imputer.transform(X[:, 0:300])



counter = 0

for iterations in range(1):



    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)



    classifier = Sequential()



    classifier.add(Dense(units=70, kernel_initializer='uniform',kernel_regularizer=regularizers.l2(0.0001),

                activity_regularizer=regularizers.l1(0.001), activation='relu', input_dim=300))



    classifier.add(Dropout(0.3))



    # classifier.add(Dense(units=70, kernel_initializer='uniform', activation='relu'))





    classifier.add(Dense(units=6, kernel_initializer='uniform', activation='softmax'))



    classifier.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])



    history = classifier.fit(X_train, y_train,validation_split=0.1, batch_size=10, epochs=100,shuffle=True)



    y_pred = classifier.predict(X_test)

    # y_pred = (y_pred > 0.1666)



    classifier.summary()

    counter+=1

    print(classifier.layers)

    print("Running RNN with Dropout Layer")

    print("Number of layers used: "+str(len(classifier.layers)))

    # list all data in history

    print(history.history.keys())

    # summarize history for accuracy

    plt.plot(history.history['acc'])

    plt.plot(history.history['val_acc'])

    plt.title('model accuracy')

    plt.ylabel('accuracy')

    plt.xlabel('epoch')

    plt.legend(['train', 'test'], loc='upper left')

    plt.show()

    # summarize history for loss

    plt.plot(history.history['loss'])

    plt.plot(history.history['val_loss'])

    plt.title('model loss')

    plt.ylabel('loss')

    plt.xlabel('epoch')

    plt.legend(['train', 'test'], loc='upper left')

    plt.show()

Yep - you've overfitted !

Can you see what your accuracy is as a function of the number of training cycles? It would be interesting to see how that looks.

Is there any way you could increase your data? 15000x300 isn't a huge amount of depth for a neural network, considering the width of the data set.

You could look at the impact of increasing your regularisation parameters, or decreasing the depth of your network