review: gradient descent, epochs, validation in neural network training
$begingroup$
These days, training data aren't put in gradient descent all at once. Rather, they are put in batch after batch. Gradient descent is run once for each batch of training data. When all batches are traversed, 1 epoch is done. However, gradient descent hasn't minimized the loss function yet, the loss function still being on a slope. The loss function now and weights serve as initial values for the next epoch. Weights are initialized only at the beginning of the 1st batch of the 1st epoch.
During each epoch, the validation set is used to calculate the error (evaluated using trained weights) on the validation set. If the error is dropping, finish this epoch and go to the next epoch. Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
If the above is all right, seems that the hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?
deep-learning cnn training hyperparameter-tuning
asked 4 mins ago
feynmanfeynman
558
$endgroup$
add a comment |
$begingroup$
These days, training data aren't put in gradient descent all at once. Rather, they are put in batch after batch. Gradient descent is run once for each batch of training data. When all batches are traversed, 1 epoch is done. However, gradient descent hasn't minimized the loss function yet, the loss function still being on a slope. The loss function now and weights serve as initial values for the next epoch. Weights are initialized only at the beginning of the 1st batch of the 1st epoch.
During each epoch, the validation set is used to calculate the error (evaluated using trained weights) on the validation set. If the error is dropping, finish this epoch and go to the next epoch. Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
If the above is all right, seems that the hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?
deep-learning cnn training hyperparameter-tuning
asked 4 mins ago
feynmanfeynman
558
$endgroup$
add a comment |
$begingroup$
These days, training data aren't put in gradient descent all at once. Rather, they are put in batch after batch. Gradient descent is run once for each batch of training data. When all batches are traversed, 1 epoch is done. However, gradient descent hasn't minimized the loss function yet, the loss function still being on a slope. The loss function now and weights serve as initial values for the next epoch. Weights are initialized only at the beginning of the 1st batch of the 1st epoch.
During each epoch, the validation set is used to calculate the error (evaluated using trained weights) on the validation set. If the error is dropping, finish this epoch and go to the next epoch. Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
If the above is all right, seems that the hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?
deep-learning cnn training hyperparameter-tuning
asked 4 mins ago
feynmanfeynman
558
$endgroup$
These days, training data aren't put in gradient descent all at once. Rather, they are put in batch after batch. Gradient descent is run once for each batch of training data. When all batches are traversed, 1 epoch is done. However, gradient descent hasn't minimized the loss function yet, the loss function still being on a slope. The loss function now and weights serve as initial values for the next epoch. Weights are initialized only at the beginning of the 1st batch of the 1st epoch.
During each epoch, the validation set is used to calculate the error (evaluated using trained weights) on the validation set. If the error is dropping, finish this epoch and go to the next epoch. Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
If the above is all right, seems that the hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?
deep-learning cnn training hyperparameter-tuning
deep-learning cnn training hyperparameter-tuning
asked 4 mins ago
feynmanfeynman
558
asked 4 mins ago
feynmanfeynman
558
asked 4 mins ago
feynmanfeynman
558
asked 4 mins ago
feynmanfeynman
558
asked 4 mins ago
feynmanfeynman
558
558
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47276%2freview-gradient-descent-epochs-validation-in-neural-network-training%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47276%2freview-gradient-descent-epochs-validation-in-neural-network-training%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
