Gfyuki

I'm building a sentiment analysis program in python using Keras Sequential model for deep learning

my data is 20,000 tweets:

• positive tweets: 9152 tweets


• negative tweets: 10849 tweets

I wrote a sequential model script to make the binary classification as follows:

model=Sequential()

model.add(Embedding(vocab_size, 100, input_length=max_words))

model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))

model.add(MaxPooling1D(pool_size=2))

model.add(Flatten())

model.add(Dense(250, activation='relu'))

model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])



# Fit the model



print(model.summary())

history=model.fit(X_train[train], y1[train], validation_split=0.30,epochs=2, batch_size=128,verbose=2)

however I get very strange results! The model accuracy is almost perfect (>90)
whereas the validation accuracy is very low (<1) (shown bellow)

Train on 9417 samples, validate on 4036 samples

 Epoch 1/2

- 13s - loss: 0.5478 - acc: 0.7133 - val_loss: 3.6157 - val_acc: 0.0243

 Epoch 2/2

- 11s - loss: 0.2287 - acc: 0.8995 - val_loss: 5.4746 - val_acc: 0.0339

I tried to increase the number of epoch, and it only increases the model accuracy and lowers the validation accuracy

Any advice on how to overcome this issue?

Update:

this is how I handle my data

#read training data

pos_file=open('pos2.txt', 'r', encoding="Latin-1")

 neg_file=open('neg3.txt', 'r', encoding="Latin-1")

# Load data from files

pos = list(pos_file.readlines())

neg = list(neg_file.readlines()) 

x = pos + neg

docs = numpy.array(x)

#read Testing Data

pos_test=open('posTest2.txt', 'r',encoding="Latin-1")

posT = list(pos_test.readlines())

neg_test=open('negTest2.txt', 'r',encoding="Latin-1")

negT = list(neg_test.readlines())

xTest = posT + negT

total2 = numpy.array(xTest)



CombinedDocs=numpy.append(total2,docs)



# Generate labels

positive_labels = [1 for _ in pos]

negative_labels = [0 for _ in neg]

labels = numpy.concatenate([positive_labels, negative_labels], 0)



# prepare tokenizer

t = Tokenizer()

t.fit_on_texts(CombinedDocs)

vocab_size = len(t.word_index) + 1

# integer encode the documents

encoded_docs = t.texts_to_sequences(docs)

#print(encoded_docs)



# pad documents to a max length of 140 words

max_length = 140

padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')

Here I used Google public word2vec

# load the whole embedding into memory

embeddings_index = dict()

f = open('Google28.bin',encoding="latin-1")

for line in f:

values = line.split()

word = values[0]

coefs = asarray(values[1:], dtype='str')

embeddings_index[word] = coefs

f.close()



# create a weight matrix for words in training docs

embedding_matrix = zeros((vocab_size, 100))



for word, i in t.word_index.items():

embedding_vector = embeddings_index.get(word)

if embedding_vector is not None:

    embedding_matrix[i] = embedding_vector





#Convert to numpy

NewTraining=numpy.array(padded_docs)

NewLabels=numpy.array(labels)

encoded_docs2 = t.texts_to_sequences(total2)



# pad documents to a max length of 140 words



padded_docs2 = pad_sequences(encoded_docs2, maxlen=max_length, padding='post')





# Generate labels

positive_labels2 = [1 for _ in posT]

negative_labels2 = [0 for _ in negT]

yTest = numpy.concatenate([positive_labels2, negative_labels2], 0)

NewTesting=numpy.array(padded_docs2)

NewLabelsTsting=numpy.array(yTest)

You should try to shuffle all of your data and split them to the train and test and valid set then train again.

I had the same condition: High acc and low vad_acc.

It was because the parameter of Keras.model.fit, validation_split.

This will separate the last section of data as validation data.
Therefore, if your data was in order, your validity data will be in the same case.
Try to shuffle the training data.

It seems that with validation split, validation accuracy is not working properly. Instead of using validation split in fit function of your model, try splitting your training data into train data and validate data before fit function and then feed the validation data in the feed function like this.

Instead of doing this

history=model.fit(X_train[train], y1[train], validation_split=0.30,epochs=2, batch_size=128,verbose=2)

Split your train data into validation and train data by any method, and then say your validation data is (X_val,Y_val), then replace the above line of code with this one:

history=model.fit(X_train[train], y1[train], validation_data=(X_val,Y_val),epochs=2, batch_size=128,verbose=2)

When a machine learning model has high training accuracy and very low validation then this case is probably known as over-fitting. The reasons for this can be as follows:

1. The hypothesis function you are using is too complex that your model perfectly fits the training data but fails to do on test/validation data.


2. The number of learning parameters in your model is way too much that instead of generalizing the examples your model learns those examples and hence the model performs badly on test/validation data.

To solve the above problems a number of solutions can be tried depending on your dataset:

1. Use a simple cost and loss function.


2. Use regulation which helps in reducing over-fitting i.e Dropout.


3. Reduce the number of learning parameters in your model.

These are the 3 solutions that are most likely to improve the validation accuracy of your model and still if these don't work check your inputs whether they have the right shapes and sizes.

I have a similar issue with my model. I'm trying to use the most basic Conv1D model to analyze review data and output a rating of 1-5 class, therefore the loss is categorical_crossentropy.
Model structure is as below

# define model

model = Sequential()

model.add(Embedding(vocab_size, 100, input_length=max_length))

model.add(Conv1D(filters=32, kernel_size=8, activation='relu'))

model.add(MaxPooling1D(pool_size=2))

model.add(Flatten())

model.add(Dense(10, activation='relu'))

model.add(Dense(5, activation='softmax'))

Total params: 15,865,417

Trainable params: 15,865,417

Non-trainable params: 0

Training acc increases and loss decreases as expected. But validation loss and validation acc decrease straight after the 1st epoch itself. The overall testing after training gives an accuracy around 60s.

I've already shuffled, downsampled (all classes have the number of data samples) and splitted the data properly to training(70%) / validation(10%) / testing(20%).

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])



model.fit(final_X_train, final_Y_train, epochs=5, batch_size=128, validation_data=(final_X_val, final_Y_val), callbacks=callback)

Any help is highly appreciated.