Hfrxdjt

Last 4-6 weeks, I have been learning and working for the first time on ML. Reading blogs, articles, documentations, etc. and practising. Have asked lot of questions here on Stack Overflow as well.

While I have got some amount of hands-on experience, but still got a very basic doubt (confusion) --
When I take my input data set with 1000 records, the model prediction accuracy is say 75%. When I keep 50000 records, the model accuracy is 65%.

1) Does that mean the model responds completely based on the i/p data being fed into?

2) If #1 is true, then in real-world where we don't have control on input data, how will the model work?

Ex. For suggesting products to a customer, the input data to the model would be the past customer buying experiences. As the quantity of input data increases, the prediction accuracy will increase or decrease?

Please let me know if I need to add further details to my question.

Thanks.

Edit - 1 - Below added frequency distribution of my input data:

It looks like your model overfits did you try to do a train/test split?

To answer your first question, the accuracy of the model highly depends on the "quality" of the input data. Basically, your training data should represent the same scenario as that of the final model deployment environment.

There are two reasons why the scenario you mentioned is happening,

1. When you added more data, maybe there is no good relationship between input features and label for the new examples. It is always said that less and clean data is better than large and messy data.



2. 
   

   If 49000 records added afterward are from the same set(i.e. have a good relationship between label and features) as that of 1000 before, there are again two possible reasons

   
   
   

   A. If accuracy on the train dataset is small along with test dataset. e.g. training accuracy is 70% and test accuracy is 65%, then you are underfitting data. Model is very complex and dataset is small in terms of the number of examples.

   
   
   

   B. If your training accuracy is near 100% and test accuracy is 65%, you are overfitting data. Model is complex, so you should go with some simple algorithm.

   
   
   

   NOTE* Since you haven't mentioned about training accuracy, it is difficult to say what out of two above is happening.

   
   

Now coming to your second question about real-world deployment. There is something called model staleness over time which is basically the problem of reducing accuracy of a model over time. https://medium.com/thelaunchpad/how-to-protect-your-machine-learning-product-from-time-adversaries-and-itself-ff07727d6712, this is the article by a product manager at Google how staleness problem and how it can be solved. This will answer your second question.

Let me know if something is not clear.