Hfrxdjt

Hello and thank you in advance for any answer. I am building a NN for a multi class classification problem. I have pretrained my data through a Word2vec and generated a 300 dimension vector with values. I have 6 classes .. My data consist of 15000 rows(x300 dimensions). My first question is :

What is the number of units? Is it something that we can extract from theory?
Also i have managed to gain 0.85-0.90 training set accuracy but the validation set accuracy is always low: 0.22-0.25. I do not know what way to follow as I am newbie to these kind of stuff. My code is:

import numpy as np
import pandas as pd
import keras
from sklearn.preprocessing import Imputer
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Dense,Dropout
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
from keras import regularizers



dataset = pd.read_csv('word2vec.csv')


X = dataset.iloc[:, 0:300].values
y = dataset.iloc[:, 300].values

imputer = Imputer(missing_values = 'NaN', strategy = 'mean', axis = 0)
imputer = imputer.fit(X[:, 0:300])
X[:, 0:300] = imputer.transform(X[:, 0:300])

counter = 0
for iterations in range(1):

 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

 classifier = Sequential()

 classifier.add(Dense(units=70, kernel_initializer='uniform',kernel_regularizer=regularizers.l2(0.0001),
 activity_regularizer=regularizers.l1(0.001), activation='relu', input_dim=300))

 classifier.add(Dropout(0.3))

 # classifier.add(Dense(units=70, kernel_initializer='uniform', activation='relu'))


 classifier.add(Dense(units=6, kernel_initializer='uniform', activation='softmax'))

 classifier.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

 history = classifier.fit(X_train, y_train,validation_split=0.1, batch_size=10, epochs=100,shuffle=True)

 y_pred = classifier.predict(X_test)
 # y_pred = (y_pred > 0.1666)

 classifier.summary()
 counter+=1
 print(classifier.layers)
 print("Running RNN with Dropout Layer")
 print("Number of layers used: "+str(len(classifier.layers)))
 # list all data in history
 print(history.history.keys())
 # summarize history for accuracy
 plt.plot(history.history['acc'])
 plt.plot(history.history['val_acc'])
 plt.title('model accuracy')
 plt.ylabel('accuracy')
 plt.xlabel('epoch')
 plt.legend(['train', 'test'], loc='upper left')
 plt.show()
 # summarize history for loss
 plt.plot(history.history['loss'])
 plt.plot(history.history['val_loss'])
 plt.title('model loss')
 plt.ylabel('loss')
 plt.xlabel('epoch')
 plt.legend(['train', 'test'], loc='upper left')
 plt.show()

Yep - you've overfitted !

Can you see what your accuracy is as a function of the number of training cycles? It would be interesting to see how that looks.

Is there any way you could increase your data? 15000x300 isn't a huge amount of depth for a neural network, considering the width of the data set.

You could look at the impact of increasing your regularisation parameters, or decreasing the depth of your network