How much data is needed for a GBM to be more reliable than logistic regression for binary classification?
$begingroup$
When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (number of rows) vs the reliability of the fit out-of-sample.
Obviously the more data, the more reliable the predictions will be out of sample (all else being equal). The more rows of data, the more likely that GBM will be more predictive on a real data set. So i'm wondering is there a rough rule of thumb when you would say the data is too small to use GBM, better of using a logistic regression. I know that logistic regressions can be more predictive in some cases, but my understanding is that GBM usually outperforms.
I'm particularly interested in the case that the number of predictive variables is far less than the total number of variables available. For example, we have 100 variables, 10 of which might be highly predictive of the target variable.
I know about all the benefits of train/test sets and CV. But that doesn't work all the time either. For example, your colleague has just run the train/test loop too many times, has added too many unreliable variables into the mix to improve the test AUC, and now you have a GBM model that is crazy good. Too good.
logistic-regression cross-validation gbm
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
$endgroup$
bumped to the homepage by Community♦ 6 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (number of rows) vs the reliability of the fit out-of-sample.
Obviously the more data, the more reliable the predictions will be out of sample (all else being equal). The more rows of data, the more likely that GBM will be more predictive on a real data set. So i'm wondering is there a rough rule of thumb when you would say the data is too small to use GBM, better of using a logistic regression. I know that logistic regressions can be more predictive in some cases, but my understanding is that GBM usually outperforms.
I'm particularly interested in the case that the number of predictive variables is far less than the total number of variables available. For example, we have 100 variables, 10 of which might be highly predictive of the target variable.
I know about all the benefits of train/test sets and CV. But that doesn't work all the time either. For example, your colleague has just run the train/test loop too many times, has added too many unreliable variables into the mix to improve the test AUC, and now you have a GBM model that is crazy good. Too good.
logistic-regression cross-validation gbm
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
$endgroup$
bumped to the homepage by Community♦ 6 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (number of rows) vs the reliability of the fit out-of-sample.
Obviously the more data, the more reliable the predictions will be out of sample (all else being equal). The more rows of data, the more likely that GBM will be more predictive on a real data set. So i'm wondering is there a rough rule of thumb when you would say the data is too small to use GBM, better of using a logistic regression. I know that logistic regressions can be more predictive in some cases, but my understanding is that GBM usually outperforms.
I'm particularly interested in the case that the number of predictive variables is far less than the total number of variables available. For example, we have 100 variables, 10 of which might be highly predictive of the target variable.
I know about all the benefits of train/test sets and CV. But that doesn't work all the time either. For example, your colleague has just run the train/test loop too many times, has added too many unreliable variables into the mix to improve the test AUC, and now you have a GBM model that is crazy good. Too good.
logistic-regression cross-validation gbm
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
$endgroup$
When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (number of rows) vs the reliability of the fit out-of-sample.
Obviously the more data, the more reliable the predictions will be out of sample (all else being equal). The more rows of data, the more likely that GBM will be more predictive on a real data set. So i'm wondering is there a rough rule of thumb when you would say the data is too small to use GBM, better of using a logistic regression. I know that logistic regressions can be more predictive in some cases, but my understanding is that GBM usually outperforms.
I'm particularly interested in the case that the number of predictive variables is far less than the total number of variables available. For example, we have 100 variables, 10 of which might be highly predictive of the target variable.
I know about all the benefits of train/test sets and CV. But that doesn't work all the time either. For example, your colleague has just run the train/test loop too many times, has added too many unreliable variables into the mix to improve the test AUC, and now you have a GBM model that is crazy good. Too good.
logistic-regression cross-validation gbm
logistic-regression cross-validation gbm
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
asked Nov 20 '17 at 18:26
dule arnauxdule arnaux
1061
1061
bumped to the homepage by Community♦ 6 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 6 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have
- correlated variables
- nonlinearity
- interactions
If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance.
If you're worried that you've overfit your model to the test set, you'll need to find a new test set!
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f24966%2fhow-much-data-is-needed-for-a-gbm-to-be-more-reliable-than-logistic-regression-f%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have
- correlated variables
- nonlinearity
- interactions
If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance.
If you're worried that you've overfit your model to the test set, you'll need to find a new test set!
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
$endgroup$
add a comment |
$begingroup$
In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have
- correlated variables
- nonlinearity
- interactions
If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance.
If you're worried that you've overfit your model to the test set, you'll need to find a new test set!
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
$endgroup$
add a comment |
$begingroup$
In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have
- correlated variables
- nonlinearity
- interactions
If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance.
If you're worried that you've overfit your model to the test set, you'll need to find a new test set!
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
$endgroup$
In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have
- correlated variables
- nonlinearity
- interactions
If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance.
If you're worried that you've overfit your model to the test set, you'll need to find a new test set!
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
answered Nov 22 '17 at 22:40
Max FlanderMax Flander
1716
1716
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f24966%2fhow-much-data-is-needed-for-a-gbm-to-be-more-reliable-than-logistic-regression-f%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
