How to determine best parameters after grid search?
$begingroup$
I'm optimizing the parameters for a single layer MLP. I've chosen to vary 4 parameters: hidden layer size, tolerance, activation, and regularization weights. Each of these has 4 possible values it can take (4^4 = 256 combinations).
So the question is, how does one determine that a set of parameters are statistically significantly better than another?
My stats is a little rusty, but my first thought was n-way ANOVA with 4 factors and 4 degrees of freedom in each factor. Is there something better?
python neural-network
asked Nov 28 '17 at 3:36
BraaedyBraaedy
61
$endgroup$
bumped to the homepage by Community♦ 1 min ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I'm optimizing the parameters for a single layer MLP. I've chosen to vary 4 parameters: hidden layer size, tolerance, activation, and regularization weights. Each of these has 4 possible values it can take (4^4 = 256 combinations).
So the question is, how does one determine that a set of parameters are statistically significantly better than another?
My stats is a little rusty, but my first thought was n-way ANOVA with 4 factors and 4 degrees of freedom in each factor. Is there something better?
python neural-network
asked Nov 28 '17 at 3:36
BraaedyBraaedy
61
$endgroup$
bumped to the homepage by Community♦ 1 min ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
4
$begingroup$
Welcome to the site! People usually just choose the one that performs the best on the validation set, assuming that the sample is large enough to achieve statistical significance. However, I can't imagine testing 256 hyperparameter combinations on a realistic data set so yours might be small, in which case statistical significance may indeed be a consideration... which in turn calls into question the choice of using a neural network. Just out of curiosity, what is the "tolerance"?
$endgroup$
– Emre
Nov 28 '17 at 4:11
$begingroup$
Sigma for improvement of the solution, see "tol" here: scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier. The data set size is ~13k, I'm sampling 5k at a time to learn from. It seems like most of them perform similarly, but their learning times can vary quite a bit, (9s-100s). It seems like I'd still want to perform a test to determine which is the best in this regard. Does 4 way ANOVA make sense?
$endgroup$
– Braaedy
Nov 28 '17 at 4:22
add a comment |
$begingroup$
I'm optimizing the parameters for a single layer MLP. I've chosen to vary 4 parameters: hidden layer size, tolerance, activation, and regularization weights. Each of these has 4 possible values it can take (4^4 = 256 combinations).
So the question is, how does one determine that a set of parameters are statistically significantly better than another?
My stats is a little rusty, but my first thought was n-way ANOVA with 4 factors and 4 degrees of freedom in each factor. Is there something better?
python neural-network
asked Nov 28 '17 at 3:36
BraaedyBraaedy
61
$endgroup$
I'm optimizing the parameters for a single layer MLP. I've chosen to vary 4 parameters: hidden layer size, tolerance, activation, and regularization weights. Each of these has 4 possible values it can take (4^4 = 256 combinations).
So the question is, how does one determine that a set of parameters are statistically significantly better than another?
My stats is a little rusty, but my first thought was n-way ANOVA with 4 factors and 4 degrees of freedom in each factor. Is there something better?
python neural-network
python neural-network
asked Nov 28 '17 at 3:36
BraaedyBraaedy
61
asked Nov 28 '17 at 3:36
BraaedyBraaedy
61
asked Nov 28 '17 at 3:36
BraaedyBraaedy
61
asked Nov 28 '17 at 3:36
BraaedyBraaedy
61
asked Nov 28 '17 at 3:36
BraaedyBraaedy
61
61
bumped to the homepage by Community♦ 1 min ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 1 min ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
4
$begingroup$
Welcome to the site! People usually just choose the one that performs the best on the validation set, assuming that the sample is large enough to achieve statistical significance. However, I can't imagine testing 256 hyperparameter combinations on a realistic data set so yours might be small, in which case statistical significance may indeed be a consideration... which in turn calls into question the choice of using a neural network. Just out of curiosity, what is the "tolerance"?
$endgroup$
– Emre
Nov 28 '17 at 4:11
$begingroup$
Sigma for improvement of the solution, see "tol" here: scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier. The data set size is ~13k, I'm sampling 5k at a time to learn from. It seems like most of them perform similarly, but their learning times can vary quite a bit, (9s-100s). It seems like I'd still want to perform a test to determine which is the best in this regard. Does 4 way ANOVA make sense?
$endgroup$
– Braaedy
Nov 28 '17 at 4:22
add a comment |
4
$begingroup$
Welcome to the site! People usually just choose the one that performs the best on the validation set, assuming that the sample is large enough to achieve statistical significance. However, I can't imagine testing 256 hyperparameter combinations on a realistic data set so yours might be small, in which case statistical significance may indeed be a consideration... which in turn calls into question the choice of using a neural network. Just out of curiosity, what is the "tolerance"?
$endgroup$
– Emre
Nov 28 '17 at 4:11
$begingroup$
Sigma for improvement of the solution, see "tol" here: scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier. The data set size is ~13k, I'm sampling 5k at a time to learn from. It seems like most of them perform similarly, but their learning times can vary quite a bit, (9s-100s). It seems like I'd still want to perform a test to determine which is the best in this regard. Does 4 way ANOVA make sense?
$endgroup$
– Braaedy
Nov 28 '17 at 4:22
4
4
$begingroup$
Welcome to the site! People usually just choose the one that performs the best on the validation set, assuming that the sample is large enough to achieve statistical significance. However, I can't imagine testing 256 hyperparameter combinations on a realistic data set so yours might be small, in which case statistical significance may indeed be a consideration... which in turn calls into question the choice of using a neural network. Just out of curiosity, what is the "tolerance"?
$endgroup$
– Emre
Nov 28 '17 at 4:11
$begingroup$
Welcome to the site! People usually just choose the one that performs the best on the validation set, assuming that the sample is large enough to achieve statistical significance. However, I can't imagine testing 256 hyperparameter combinations on a realistic data set so yours might be small, in which case statistical significance may indeed be a consideration... which in turn calls into question the choice of using a neural network. Just out of curiosity, what is the "tolerance"?
$endgroup$
– Emre
Nov 28 '17 at 4:11
$begingroup$
Sigma for improvement of the solution, see "tol" here: scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier. The data set size is ~13k, I'm sampling 5k at a time to learn from. It seems like most of them perform similarly, but their learning times can vary quite a bit, (9s-100s). It seems like I'd still want to perform a test to determine which is the best in this regard. Does 4 way ANOVA make sense?
$endgroup$
– Braaedy
Nov 28 '17 at 4:22
$begingroup$
Sigma for improvement of the solution, see "tol" here: scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier. The data set size is ~13k, I'm sampling 5k at a time to learn from. It seems like most of them perform similarly, but their learning times can vary quite a bit, (9s-100s). It seems like I'd still want to perform a test to determine which is the best in this regard. Does 4 way ANOVA make sense?
$endgroup$
– Braaedy
Nov 28 '17 at 4:22
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
I too was in your place when I started using Neural networks. There are so many hyper-parameters to choose, and each take on many values. Like Emre said, you need to check the model which is giving best metric score on your data (Cross validation set). The parameter values of that model will be your optimized values. You can also check this link-
https://www.youtube.com/watch?v=Gol_qOgRqfA&list=PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A&index=8
answered Jan 23 '18 at 11:53
Ankit SethAnkit Seth
1,024218
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f25160%2fhow-to-determine-best-parameters-after-grid-search%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
I too was in your place when I started using Neural networks. There are so many hyper-parameters to choose, and each take on many values. Like Emre said, you need to check the model which is giving best metric score on your data (Cross validation set). The parameter values of that model will be your optimized values. You can also check this link-
https://www.youtube.com/watch?v=Gol_qOgRqfA&list=PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A&index=8
answered Jan 23 '18 at 11:53
Ankit SethAnkit Seth
1,024218
$endgroup$
add a comment |
$begingroup$
I too was in your place when I started using Neural networks. There are so many hyper-parameters to choose, and each take on many values. Like Emre said, you need to check the model which is giving best metric score on your data (Cross validation set). The parameter values of that model will be your optimized values. You can also check this link-
https://www.youtube.com/watch?v=Gol_qOgRqfA&list=PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A&index=8
answered Jan 23 '18 at 11:53
Ankit SethAnkit Seth
1,024218
$endgroup$
add a comment |
$begingroup$
I too was in your place when I started using Neural networks. There are so many hyper-parameters to choose, and each take on many values. Like Emre said, you need to check the model which is giving best metric score on your data (Cross validation set). The parameter values of that model will be your optimized values. You can also check this link-
https://www.youtube.com/watch?v=Gol_qOgRqfA&list=PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A&index=8
answered Jan 23 '18 at 11:53
Ankit SethAnkit Seth
1,024218
$endgroup$
I too was in your place when I started using Neural networks. There are so many hyper-parameters to choose, and each take on many values. Like Emre said, you need to check the model which is giving best metric score on your data (Cross validation set). The parameter values of that model will be your optimized values. You can also check this link-
https://www.youtube.com/watch?v=Gol_qOgRqfA&list=PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A&index=8
answered Jan 23 '18 at 11:53
Ankit SethAnkit Seth
1,024218
answered Jan 23 '18 at 11:53
Ankit SethAnkit Seth
1,024218
answered Jan 23 '18 at 11:53
Ankit SethAnkit Seth
1,024218
answered Jan 23 '18 at 11:53
Ankit SethAnkit Seth
1,024218
1,024218
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f25160%2fhow-to-determine-best-parameters-after-grid-search%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
4
$begingroup$
Welcome to the site! People usually just choose the one that performs the best on the validation set, assuming that the sample is large enough to achieve statistical significance. However, I can't imagine testing 256 hyperparameter combinations on a realistic data set so yours might be small, in which case statistical significance may indeed be a consideration... which in turn calls into question the choice of using a neural network. Just out of curiosity, what is the "tolerance"?
$endgroup$
– Emre
Nov 28 '17 at 4:11
$begingroup$
Sigma for improvement of the solution, see "tol" here: scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier. The data set size is ~13k, I'm sampling 5k at a time to learn from. It seems like most of them perform similarly, but their learning times can vary quite a bit, (9s-100s). It seems like I'd still want to perform a test to determine which is the best in this regard. Does 4 way ANOVA make sense?
$endgroup$
– Braaedy
Nov 28 '17 at 4:22