How much data is needed for a GBM to be more reliable than logistic regression for binary classification? Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 00:00UTC (8:00pm US/Eastern) 2019 Moderator Election Q&A - Questionnaire 2019 Community Moderator Election ResultsBinary classification model for sparse / biased dataPython : How to use Multinomial Logistic Regression using SKlearnLogistic regression on biased dataLogistic regression with high cardinality categorical variableStrategy for dealing with giant sample sizeHow to add more theta parameters into my logistic regression?Regression equation for ordinal dataMulti-Class Classification With Logistic Regression On Binary DataHow many coefficients does the Logistic regression model has as a function of the number of features?
What do you call the main part of a joke?
Amount of permutations on an NxNxN Rubik's Cube
Is grep documentation about ignoring case wrong, since it doesn't ignore case in filenames?
How fail-safe is nr as stop bytes?
Question about debouncing - delay of state change
Why do we need to use the builder design pattern when we can do the same thing with setters?
Crossing US/Canada Border for less than 24 hours
What does it mean that physics no longer uses mechanical models to describe phenomena?
I am having problem understanding the behavior of below code in JavaScript
How were pictures turned from film to a big picture in a picture frame before digital scanning?
What is "gratricide"?
How to play a character with a disability or mental disorder without being offensive?
Why do we bend a book to keep it straight?
How do I find out the mythology and history of my Fortress?
Illegal assignment from sObject to Id
Is there any word for a place full of confusion?
Should I use a zero-interest credit card for a large one-time purchase?
Significance of Cersei's obsession with elephants?
AppleTVs create a chatty alternate WiFi network
How much damage would a cupful of neutron star matter do to the Earth?
How many terms of the Maclaurin series for ln(1+x) do you need to use to estimate ln(1.4) to within 0.001?
Is CEO the "profession" with the most psychopaths?
Do I really need to have a message in a novel to appeal to readers?
When a candle burns, why does the top of wick glow if bottom of flame is hottest?
How much data is needed for a GBM to be more reliable than logistic regression for binary classification?
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 00:00UTC (8:00pm US/Eastern)
2019 Moderator Election Q&A - Questionnaire
2019 Community Moderator Election ResultsBinary classification model for sparse / biased dataPython : How to use Multinomial Logistic Regression using SKlearnLogistic regression on biased dataLogistic regression with high cardinality categorical variableStrategy for dealing with giant sample sizeHow to add more theta parameters into my logistic regression?Regression equation for ordinal dataMulti-Class Classification With Logistic Regression On Binary DataHow many coefficients does the Logistic regression model has as a function of the number of features?
$begingroup$
When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (number of rows) vs the reliability of the fit out-of-sample.
Obviously the more data, the more reliable the predictions will be out of sample (all else being equal). The more rows of data, the more likely that GBM will be more predictive on a real data set. So i'm wondering is there a rough rule of thumb when you would say the data is too small to use GBM, better of using a logistic regression. I know that logistic regressions can be more predictive in some cases, but my understanding is that GBM usually outperforms.
I'm particularly interested in the case that the number of predictive variables is far less than the total number of variables available. For example, we have 100 variables, 10 of which might be highly predictive of the target variable.
I know about all the benefits of train/test sets and CV. But that doesn't work all the time either. For example, your colleague has just run the train/test loop too many times, has added too many unreliable variables into the mix to improve the test AUC, and now you have a GBM model that is crazy good. Too good.
logistic-regression cross-validation gbm
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
$endgroup$
bumped to the homepage by Community♦ 32 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (number of rows) vs the reliability of the fit out-of-sample.
Obviously the more data, the more reliable the predictions will be out of sample (all else being equal). The more rows of data, the more likely that GBM will be more predictive on a real data set. So i'm wondering is there a rough rule of thumb when you would say the data is too small to use GBM, better of using a logistic regression. I know that logistic regressions can be more predictive in some cases, but my understanding is that GBM usually outperforms.
I'm particularly interested in the case that the number of predictive variables is far less than the total number of variables available. For example, we have 100 variables, 10 of which might be highly predictive of the target variable.
I know about all the benefits of train/test sets and CV. But that doesn't work all the time either. For example, your colleague has just run the train/test loop too many times, has added too many unreliable variables into the mix to improve the test AUC, and now you have a GBM model that is crazy good. Too good.
logistic-regression cross-validation gbm
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
$endgroup$
bumped to the homepage by Community♦ 32 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (number of rows) vs the reliability of the fit out-of-sample.
Obviously the more data, the more reliable the predictions will be out of sample (all else being equal). The more rows of data, the more likely that GBM will be more predictive on a real data set. So i'm wondering is there a rough rule of thumb when you would say the data is too small to use GBM, better of using a logistic regression. I know that logistic regressions can be more predictive in some cases, but my understanding is that GBM usually outperforms.
I'm particularly interested in the case that the number of predictive variables is far less than the total number of variables available. For example, we have 100 variables, 10 of which might be highly predictive of the target variable.
I know about all the benefits of train/test sets and CV. But that doesn't work all the time either. For example, your colleague has just run the train/test loop too many times, has added too many unreliable variables into the mix to improve the test AUC, and now you have a GBM model that is crazy good. Too good.
logistic-regression cross-validation gbm
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
$endgroup$
When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (number of rows) vs the reliability of the fit out-of-sample.
Obviously the more data, the more reliable the predictions will be out of sample (all else being equal). The more rows of data, the more likely that GBM will be more predictive on a real data set. So i'm wondering is there a rough rule of thumb when you would say the data is too small to use GBM, better of using a logistic regression. I know that logistic regressions can be more predictive in some cases, but my understanding is that GBM usually outperforms.
I'm particularly interested in the case that the number of predictive variables is far less than the total number of variables available. For example, we have 100 variables, 10 of which might be highly predictive of the target variable.
I know about all the benefits of train/test sets and CV. But that doesn't work all the time either. For example, your colleague has just run the train/test loop too many times, has added too many unreliable variables into the mix to improve the test AUC, and now you have a GBM model that is crazy good. Too good.
logistic-regression cross-validation gbm
logistic-regression cross-validation gbm
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
1061
bumped to the homepage by Community♦ 32 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 32 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have
- correlated variables
- nonlinearity
- interactions
If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance.
If you're worried that you've overfit your model to the test set, you'll need to find a new test set!
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f24966%2fhow-much-data-is-needed-for-a-gbm-to-be-more-reliable-than-logistic-regression-f%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have
- correlated variables
- nonlinearity
- interactions
If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance.
If you're worried that you've overfit your model to the test set, you'll need to find a new test set!
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
$endgroup$
add a comment |
$begingroup$
In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have
- correlated variables
- nonlinearity
- interactions
If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance.
If you're worried that you've overfit your model to the test set, you'll need to find a new test set!
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
$endgroup$
add a comment |
$begingroup$
In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have
- correlated variables
- nonlinearity
- interactions
If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance.
If you're worried that you've overfit your model to the test set, you'll need to find a new test set!
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
$endgroup$
In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have
- correlated variables
- nonlinearity
- interactions
If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance.
If you're worried that you've overfit your model to the test set, you'll need to find a new test set!
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
1716
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f24966%2fhow-much-data-is-needed-for-a-gbm-to-be-more-reliable-than-logistic-regression-f%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
