xgboost performance with predicted values as input
$begingroup$
I have predicted the probability of loss using different features. Now when I used this with a non-important feature to predict the probability of loss. The first one is very close. logloss was close to 0.11. However, I have few more other features, I wanted to know if the features are important or not. So, I used the new features with this predicted probability. I found volatile behavior. Not only did the performance (logloss) drop to 0.14, but the model didn't pick the predicted probability as an important feature.
My Primary Questions:
- What is the reason behind this outcome?
- Should I dump every single
feature into one model, then see which feature is important ?
beginner feature-extraction xgboost feature-engineering
edited Dec 6 '16 at 22:34
Society of Data Scientists
534515
asked Apr 29 '16 at 22:28
Kppatel PatelKppatel Patel
1063
$endgroup$
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I have predicted the probability of loss using different features. Now when I used this with a non-important feature to predict the probability of loss. The first one is very close. logloss was close to 0.11. However, I have few more other features, I wanted to know if the features are important or not. So, I used the new features with this predicted probability. I found volatile behavior. Not only did the performance (logloss) drop to 0.14, but the model didn't pick the predicted probability as an important feature.
My Primary Questions:
- What is the reason behind this outcome?
- Should I dump every single
feature into one model, then see which feature is important ?
beginner feature-extraction xgboost feature-engineering
edited Dec 6 '16 at 22:34
Society of Data Scientists
534515
asked Apr 29 '16 at 22:28
Kppatel PatelKppatel Patel
1063
$endgroup$
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
3
$begingroup$
You may want to add some more detail to your question: what features you are using, what you are predicting. Sometimes a minimal reproducible example is useful (see stackoverflow.com/questions/5963269/…) and will help others give you more instructive feedback and useful answers.
$endgroup$
– Brandon Loudermilk
Apr 30 '16 at 0:31
$begingroup$
@Kappatel Patel I don't think dumping those features will give you a better result because xgboost is very powerful and is immune to such things. Dropping features will hardly make any difference as per my opinion
$endgroup$
– Nain
Oct 27 '16 at 18:42
$begingroup$
Kaggle Quora? (Based on the logloss you mention). Give some examples, and consider that boosting requires some degree of variation, i.e use a different model or set of features.
$endgroup$
– GrimSqueaker
Jun 5 '17 at 12:45
$begingroup$
@Nain XGBoost and LightGBM are powerful tree algorithms, yet they are not robust, they are rather sensitive to overfitting, useless features etc. Their power emerges from their computational efficiency enabling the framework to ensemble models. Dump useless(if you are sure they are useless) features, be careful with XGBoost.
$endgroup$
– Ugur MULUK
Nov 28 '18 at 11:18
add a comment |
$begingroup$
I have predicted the probability of loss using different features. Now when I used this with a non-important feature to predict the probability of loss. The first one is very close. logloss was close to 0.11. However, I have few more other features, I wanted to know if the features are important or not. So, I used the new features with this predicted probability. I found volatile behavior. Not only did the performance (logloss) drop to 0.14, but the model didn't pick the predicted probability as an important feature.
My Primary Questions:
- What is the reason behind this outcome?
- Should I dump every single
feature into one model, then see which feature is important ?
beginner feature-extraction xgboost feature-engineering
edited Dec 6 '16 at 22:34
Society of Data Scientists
534515
asked Apr 29 '16 at 22:28
Kppatel PatelKppatel Patel
1063
$endgroup$
I have predicted the probability of loss using different features. Now when I used this with a non-important feature to predict the probability of loss. The first one is very close. logloss was close to 0.11. However, I have few more other features, I wanted to know if the features are important or not. So, I used the new features with this predicted probability. I found volatile behavior. Not only did the performance (logloss) drop to 0.14, but the model didn't pick the predicted probability as an important feature.
My Primary Questions:
- What is the reason behind this outcome?
- Should I dump every single
feature into one model, then see which feature is important ?
beginner feature-extraction xgboost feature-engineering
beginner feature-extraction xgboost feature-engineering
edited Dec 6 '16 at 22:34
Society of Data Scientists
534515
asked Apr 29 '16 at 22:28
Kppatel PatelKppatel Patel
1063
edited Dec 6 '16 at 22:34
Society of Data Scientists
534515
asked Apr 29 '16 at 22:28
Kppatel PatelKppatel Patel
1063
edited Dec 6 '16 at 22:34
Society of Data Scientists
534515
edited Dec 6 '16 at 22:34
Society of Data Scientists
534515
edited Dec 6 '16 at 22:34
Society of Data Scientists
534515
534515
asked Apr 29 '16 at 22:28
Kppatel PatelKppatel Patel
1063
asked Apr 29 '16 at 22:28
Kppatel PatelKppatel Patel
1063
asked Apr 29 '16 at 22:28
Kppatel PatelKppatel Patel
1063
1063
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
3
$begingroup$
You may want to add some more detail to your question: what features you are using, what you are predicting. Sometimes a minimal reproducible example is useful (see stackoverflow.com/questions/5963269/…) and will help others give you more instructive feedback and useful answers.
$endgroup$
– Brandon Loudermilk
Apr 30 '16 at 0:31
$begingroup$
@Kappatel Patel I don't think dumping those features will give you a better result because xgboost is very powerful and is immune to such things. Dropping features will hardly make any difference as per my opinion
$endgroup$
– Nain
Oct 27 '16 at 18:42
$begingroup$
Kaggle Quora? (Based on the logloss you mention). Give some examples, and consider that boosting requires some degree of variation, i.e use a different model or set of features.
$endgroup$
– GrimSqueaker
Jun 5 '17 at 12:45
$begingroup$
@Nain XGBoost and LightGBM are powerful tree algorithms, yet they are not robust, they are rather sensitive to overfitting, useless features etc. Their power emerges from their computational efficiency enabling the framework to ensemble models. Dump useless(if you are sure they are useless) features, be careful with XGBoost.
$endgroup$
– Ugur MULUK
Nov 28 '18 at 11:18
add a comment |
3
$begingroup$
You may want to add some more detail to your question: what features you are using, what you are predicting. Sometimes a minimal reproducible example is useful (see stackoverflow.com/questions/5963269/…) and will help others give you more instructive feedback and useful answers.
$endgroup$
– Brandon Loudermilk
Apr 30 '16 at 0:31
$begingroup$
@Kappatel Patel I don't think dumping those features will give you a better result because xgboost is very powerful and is immune to such things. Dropping features will hardly make any difference as per my opinion
$endgroup$
– Nain
Oct 27 '16 at 18:42
$begingroup$
Kaggle Quora? (Based on the logloss you mention). Give some examples, and consider that boosting requires some degree of variation, i.e use a different model or set of features.
$endgroup$
– GrimSqueaker
Jun 5 '17 at 12:45
$begingroup$
@Nain XGBoost and LightGBM are powerful tree algorithms, yet they are not robust, they are rather sensitive to overfitting, useless features etc. Their power emerges from their computational efficiency enabling the framework to ensemble models. Dump useless(if you are sure they are useless) features, be careful with XGBoost.
$endgroup$
– Ugur MULUK
Nov 28 '18 at 11:18
3
3
$begingroup$
You may want to add some more detail to your question: what features you are using, what you are predicting. Sometimes a minimal reproducible example is useful (see stackoverflow.com/questions/5963269/…) and will help others give you more instructive feedback and useful answers.
$endgroup$
– Brandon Loudermilk
Apr 30 '16 at 0:31
$begingroup$
You may want to add some more detail to your question: what features you are using, what you are predicting. Sometimes a minimal reproducible example is useful (see stackoverflow.com/questions/5963269/…) and will help others give you more instructive feedback and useful answers.
$endgroup$
– Brandon Loudermilk
Apr 30 '16 at 0:31
$begingroup$
@Kappatel Patel I don't think dumping those features will give you a better result because xgboost is very powerful and is immune to such things. Dropping features will hardly make any difference as per my opinion
$endgroup$
– Nain
Oct 27 '16 at 18:42
$begingroup$
@Kappatel Patel I don't think dumping those features will give you a better result because xgboost is very powerful and is immune to such things. Dropping features will hardly make any difference as per my opinion
$endgroup$
– Nain
Oct 27 '16 at 18:42
$begingroup$
Kaggle Quora? (Based on the logloss you mention). Give some examples, and consider that boosting requires some degree of variation, i.e use a different model or set of features.
$endgroup$
– GrimSqueaker
Jun 5 '17 at 12:45
$begingroup$
Kaggle Quora? (Based on the logloss you mention). Give some examples, and consider that boosting requires some degree of variation, i.e use a different model or set of features.
$endgroup$
– GrimSqueaker
Jun 5 '17 at 12:45
$begingroup$
@Nain XGBoost and LightGBM are powerful tree algorithms, yet they are not robust, they are rather sensitive to overfitting, useless features etc. Their power emerges from their computational efficiency enabling the framework to ensemble models. Dump useless(if you are sure they are useless) features, be careful with XGBoost.
$endgroup$
– Ugur MULUK
Nov 28 '18 at 11:18
$begingroup$
@Nain XGBoost and LightGBM are powerful tree algorithms, yet they are not robust, they are rather sensitive to overfitting, useless features etc. Their power emerges from their computational efficiency enabling the framework to ensemble models. Dump useless(if you are sure they are useless) features, be careful with XGBoost.
$endgroup$
– Ugur MULUK
Nov 28 '18 at 11:18
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
You could try the next thing:, get some top k predictions made with xgboost, and use only those as features and feed them into an LogisticRegression model.
I am currious if that will help in case you want to give a feedback on my proposed solution
answered Apr 30 '16 at 8:52
Daia AlexandruDaia Alexandru
11
$endgroup$
2
$begingroup$
There's no reason to think that important features in a tree based model are linearly related to the target.
$endgroup$
– Matthew Drury
Mar 2 '18 at 15:20
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f11516%2fxgboost-performance-with-predicted-values-as-input%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
You could try the next thing:, get some top k predictions made with xgboost, and use only those as features and feed them into an LogisticRegression model.
I am currious if that will help in case you want to give a feedback on my proposed solution
answered Apr 30 '16 at 8:52
Daia AlexandruDaia Alexandru
11
$endgroup$
2
$begingroup$
There's no reason to think that important features in a tree based model are linearly related to the target.
$endgroup$
– Matthew Drury
Mar 2 '18 at 15:20
add a comment |
$begingroup$
You could try the next thing:, get some top k predictions made with xgboost, and use only those as features and feed them into an LogisticRegression model.
I am currious if that will help in case you want to give a feedback on my proposed solution
answered Apr 30 '16 at 8:52
Daia AlexandruDaia Alexandru
11
$endgroup$
2
$begingroup$
There's no reason to think that important features in a tree based model are linearly related to the target.
$endgroup$
– Matthew Drury
Mar 2 '18 at 15:20
add a comment |
$begingroup$
You could try the next thing:, get some top k predictions made with xgboost, and use only those as features and feed them into an LogisticRegression model.
I am currious if that will help in case you want to give a feedback on my proposed solution
answered Apr 30 '16 at 8:52
Daia AlexandruDaia Alexandru
11
$endgroup$
You could try the next thing:, get some top k predictions made with xgboost, and use only those as features and feed them into an LogisticRegression model.
I am currious if that will help in case you want to give a feedback on my proposed solution
answered Apr 30 '16 at 8:52
Daia AlexandruDaia Alexandru
11
answered Apr 30 '16 at 8:52
Daia AlexandruDaia Alexandru
11
answered Apr 30 '16 at 8:52
Daia AlexandruDaia Alexandru
11
answered Apr 30 '16 at 8:52
Daia AlexandruDaia Alexandru
11
11
2
$begingroup$
There's no reason to think that important features in a tree based model are linearly related to the target.
$endgroup$
– Matthew Drury
Mar 2 '18 at 15:20
add a comment |
2
$begingroup$
There's no reason to think that important features in a tree based model are linearly related to the target.
$endgroup$
– Matthew Drury
Mar 2 '18 at 15:20
2
2
$begingroup$
There's no reason to think that important features in a tree based model are linearly related to the target.
$endgroup$
– Matthew Drury
Mar 2 '18 at 15:20
$begingroup$
There's no reason to think that important features in a tree based model are linearly related to the target.
$endgroup$
– Matthew Drury
Mar 2 '18 at 15:20
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f11516%2fxgboost-performance-with-predicted-values-as-input%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
3
$begingroup$
You may want to add some more detail to your question: what features you are using, what you are predicting. Sometimes a minimal reproducible example is useful (see stackoverflow.com/questions/5963269/…) and will help others give you more instructive feedback and useful answers.
$endgroup$
– Brandon Loudermilk
Apr 30 '16 at 0:31
$begingroup$
@Kappatel Patel I don't think dumping those features will give you a better result because xgboost is very powerful and is immune to such things. Dropping features will hardly make any difference as per my opinion
$endgroup$
– Nain
Oct 27 '16 at 18:42
$begingroup$
Kaggle Quora? (Based on the logloss you mention). Give some examples, and consider that boosting requires some degree of variation, i.e use a different model or set of features.
$endgroup$
– GrimSqueaker
Jun 5 '17 at 12:45
$begingroup$
@Nain XGBoost and LightGBM are powerful tree algorithms, yet they are not robust, they are rather sensitive to overfitting, useless features etc. Their power emerges from their computational efficiency enabling the framework to ensemble models. Dump useless(if you are sure they are useless) features, be careful with XGBoost.
$endgroup$
– Ugur MULUK
Nov 28 '18 at 11:18