Optimizing decision threshold on model with oversampled/imbalanced data Unicorn Meta Zoo #1: Why another podcast? Announcing the arrival of Valued Associate #679: Cesar Manara 2019 Moderator Election Q&A - Questionnaire 2019 Community Moderator Election ResultsOverfitting/Underfitting with Data set sizeError in Model with caret PackageIs Gini coefficient a good metric for measuring predictive model performance on highly imbalanced dataImbalanced classification data with a top decile conversion metricChoosing a model for dataset with categorical variablesCross validation for highly imbalanced data with undersamplingDoes a precision score increasing with a higher number of folds mean the model will improve with more data?How to Work with Imbalanced DataProbabilistic Machine Learning model to match spatial data
How was Lagrange appointed professor of mathematics so early?
What were wait-states, and why was it only an issue for PCs?
Bright yellow or light yellow?
Why do people think Winterfell crypts is the safest place for women, children & old people?
`FindRoot [ ]`::jsing: Encountered a singular Jacobian at a point...WHY
Did war bonds have better investment alternatives during WWII?
Israeli soda type drink
Raising a bilingual kid. When should we introduce the majority language?
Where to find documentation for `whois` command options?
Why I cannot instantiate a class whose constructor is private in a friend class?
How to keep bees out of canned beverages?
Determinant of a matrix with 2 equal rows
Retract an already submitted Recommendation Letter (written for an undergrad student)
Is a self contained air-bullet cartridge feasible?
What helicopter has the most rotor blades?
Coin Game with infinite paradox
Is there a verb for listening stealthily?
How would it unbalance gameplay to rule that Weapon Master allows for picking a fighting style?
Like totally amazing interchangeable sister outfit accessory swapping or whatever
Mechanism of the formation of peracetic acid
Why did Israel vote against lifting the American embargo on Cuba?
What to do with someone that cheated their way though university and a PhD program?
"Working on a knee"
Marquee sign letters
Optimizing decision threshold on model with oversampled/imbalanced data
Unicorn Meta Zoo #1: Why another podcast?
Announcing the arrival of Valued Associate #679: Cesar Manara
2019 Moderator Election Q&A - Questionnaire
2019 Community Moderator Election ResultsOverfitting/Underfitting with Data set sizeError in Model with caret PackageIs Gini coefficient a good metric for measuring predictive model performance on highly imbalanced dataImbalanced classification data with a top decile conversion metricChoosing a model for dataset with categorical variablesCross validation for highly imbalanced data with undersamplingDoes a precision score increasing with a higher number of folds mean the model will improve with more data?How to Work with Imbalanced DataProbabilistic Machine Learning model to match spatial data
$begingroup$
I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a workflow in mind which I wanted to share and get an opinion on if I'm heading in the right direction or maybe I missed something.
- Split Train/Test/Val
- Setup pipeline for GridSearch and optimize hyper-parameters (pipeline will only oversample training folds)
- Scoring metric will be AUC as training set is balanced at that point
- Since model was trained on balanced dataset, it will probably be very conservative and predict a lot of false positives
- Taking above into consideration, model will be calibrated to have more accurate probabilities (CalibratedClassifierCV)
- View precision/recall curve with calibrated probability thresholds on validation set and determine optimal point
Does this process sound reasonable? Would appreciate any feedback/suggestions
cross-validation model-selection smote grid-search
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
$endgroup$
bumped to the homepage by Community♦ 32 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a workflow in mind which I wanted to share and get an opinion on if I'm heading in the right direction or maybe I missed something.
- Split Train/Test/Val
- Setup pipeline for GridSearch and optimize hyper-parameters (pipeline will only oversample training folds)
- Scoring metric will be AUC as training set is balanced at that point
- Since model was trained on balanced dataset, it will probably be very conservative and predict a lot of false positives
- Taking above into consideration, model will be calibrated to have more accurate probabilities (CalibratedClassifierCV)
- View precision/recall curve with calibrated probability thresholds on validation set and determine optimal point
Does this process sound reasonable? Would appreciate any feedback/suggestions
cross-validation model-selection smote grid-search
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
$endgroup$
bumped to the homepage by Community♦ 32 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a workflow in mind which I wanted to share and get an opinion on if I'm heading in the right direction or maybe I missed something.
- Split Train/Test/Val
- Setup pipeline for GridSearch and optimize hyper-parameters (pipeline will only oversample training folds)
- Scoring metric will be AUC as training set is balanced at that point
- Since model was trained on balanced dataset, it will probably be very conservative and predict a lot of false positives
- Taking above into consideration, model will be calibrated to have more accurate probabilities (CalibratedClassifierCV)
- View precision/recall curve with calibrated probability thresholds on validation set and determine optimal point
Does this process sound reasonable? Would appreciate any feedback/suggestions
cross-validation model-selection smote grid-search
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
$endgroup$
I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a workflow in mind which I wanted to share and get an opinion on if I'm heading in the right direction or maybe I missed something.
- Split Train/Test/Val
- Setup pipeline for GridSearch and optimize hyper-parameters (pipeline will only oversample training folds)
- Scoring metric will be AUC as training set is balanced at that point
- Since model was trained on balanced dataset, it will probably be very conservative and predict a lot of false positives
- Taking above into consideration, model will be calibrated to have more accurate probabilities (CalibratedClassifierCV)
- View precision/recall curve with calibrated probability thresholds on validation set and determine optimal point
Does this process sound reasonable? Would appreciate any feedback/suggestions
cross-validation model-selection smote grid-search
cross-validation model-selection smote grid-search
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
edited Sep 24 '18 at 14:52
rayven1lk
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
asked Sep 21 '18 at 20:36
rayven1lkrayven1lk
666
666
bumped to the homepage by Community♦ 32 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 32 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
I am not sure if in the last point, you meant the validation set instead of the testing set.
Here is my advice:
1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need to handle the data imbalance problem.
2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.
3- use the training set for training, the validation set for tuning the hyper parameters , BUT do not touch the testing set
4- use the testing set for testing only
5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f38621%2foptimizing-decision-threshold-on-model-with-oversampled-imbalanced-data%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
I am not sure if in the last point, you meant the validation set instead of the testing set.
Here is my advice:
1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need to handle the data imbalance problem.
2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.
3- use the training set for training, the validation set for tuning the hyper parameters , BUT do not touch the testing set
4- use the testing set for testing only
5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
$endgroup$
add a comment |
$begingroup$
I am not sure if in the last point, you meant the validation set instead of the testing set.
Here is my advice:
1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need to handle the data imbalance problem.
2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.
3- use the training set for training, the validation set for tuning the hyper parameters , BUT do not touch the testing set
4- use the testing set for testing only
5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
$endgroup$
add a comment |
$begingroup$
I am not sure if in the last point, you meant the validation set instead of the testing set.
Here is my advice:
1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need to handle the data imbalance problem.
2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.
3- use the training set for training, the validation set for tuning the hyper parameters , BUT do not touch the testing set
4- use the testing set for testing only
5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
$endgroup$
I am not sure if in the last point, you meant the validation set instead of the testing set.
Here is my advice:
1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need to handle the data imbalance problem.
2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.
3- use the training set for training, the validation set for tuning the hyper parameters , BUT do not touch the testing set
4- use the testing set for testing only
5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
answered Sep 24 '18 at 6:31
Bashar HaddadBashar Haddad
1,2821413
1,2821413
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f38621%2foptimizing-decision-threshold-on-model-with-oversampled-imbalanced-data%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
