Training the parameters of a Restricted Boltzman machine
$begingroup$
Why are the parameters of a Restricted Boltzmann machine trained for a fixed number of iterations (epochs) in many papers instead of choosing the ones corresponding to a stationary point of the likelihood?
Denote the observable data by $x$, hidden data by $h$, the energy function by $E$ and the normalizing constant by $Z$. The probability of $x$ is:
begin{equation}
P(x) = sum_h P(x,h) = sum_h frac{e^{-E(x,h)}}{Z}.
end{equation}
The goal is to maximize the probability of $x$ conditional on the parameters of the model $theta$. Suppose one has access to a sample of $N$ observations of $x$ with typical element $x_i$. As estimator, one could find the roots of the derivative of the average sample log-likelihood:
begin{equation}
leftlbrace hat{theta} in hat{Theta} : N^{-1} sum_{x_i} frac{partial log p(x_i)} {partial theta} = 0 rightrbrace
end{equation}
and chose the one $theta^star in hat{Theta} $ maximizing the empirical likelihood. There exists many different ways to approximate the derivative of the log-likelihood to facilitate (maybe even permit) its computation. For example, Contrastive Divergence and Persistent Contrastive Divergence are used often. I wonder whether it makes sense to estimate the parameters $theta$ recursively until convergence while continuing to approximate the derivative of the log-likelihood. One could update the parameters after seeing each data point $x_i$ as:
begin{equation}
theta_{i+1} = theta_{i} - eta_i frac{partial log p(x_i)}{partial theta_i}
end{equation}
The practice I learned in Hinton et al. (2006) and in Tieleman (2008) is different though: Both papers define a number of fixed iterations a priori. Could somebody kindly educate me why recursively updating the parameters until convergence is not a good idea? In particular, I'm interested whether there's a theoretical flaw in my reasoning or whether computational capacities dictate sticking to a fixed number of iterations. I am grateful for any help!
neural-network optimization rbm
asked Apr 5 '16 at 6:52
fabianfabian
1765
$endgroup$
bumped to the homepage by Community♦ 3 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
Why are the parameters of a Restricted Boltzmann machine trained for a fixed number of iterations (epochs) in many papers instead of choosing the ones corresponding to a stationary point of the likelihood?
Denote the observable data by $x$, hidden data by $h$, the energy function by $E$ and the normalizing constant by $Z$. The probability of $x$ is:
begin{equation}
P(x) = sum_h P(x,h) = sum_h frac{e^{-E(x,h)}}{Z}.
end{equation}
The goal is to maximize the probability of $x$ conditional on the parameters of the model $theta$. Suppose one has access to a sample of $N$ observations of $x$ with typical element $x_i$. As estimator, one could find the roots of the derivative of the average sample log-likelihood:
begin{equation}
leftlbrace hat{theta} in hat{Theta} : N^{-1} sum_{x_i} frac{partial log p(x_i)} {partial theta} = 0 rightrbrace
end{equation}
and chose the one $theta^star in hat{Theta} $ maximizing the empirical likelihood. There exists many different ways to approximate the derivative of the log-likelihood to facilitate (maybe even permit) its computation. For example, Contrastive Divergence and Persistent Contrastive Divergence are used often. I wonder whether it makes sense to estimate the parameters $theta$ recursively until convergence while continuing to approximate the derivative of the log-likelihood. One could update the parameters after seeing each data point $x_i$ as:
begin{equation}
theta_{i+1} = theta_{i} - eta_i frac{partial log p(x_i)}{partial theta_i}
end{equation}
The practice I learned in Hinton et al. (2006) and in Tieleman (2008) is different though: Both papers define a number of fixed iterations a priori. Could somebody kindly educate me why recursively updating the parameters until convergence is not a good idea? In particular, I'm interested whether there's a theoretical flaw in my reasoning or whether computational capacities dictate sticking to a fixed number of iterations. I am grateful for any help!
neural-network optimization rbm
asked Apr 5 '16 at 6:52
fabianfabian
1765
$endgroup$
bumped to the homepage by Community♦ 3 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
Why are the parameters of a Restricted Boltzmann machine trained for a fixed number of iterations (epochs) in many papers instead of choosing the ones corresponding to a stationary point of the likelihood?
Denote the observable data by $x$, hidden data by $h$, the energy function by $E$ and the normalizing constant by $Z$. The probability of $x$ is:
begin{equation}
P(x) = sum_h P(x,h) = sum_h frac{e^{-E(x,h)}}{Z}.
end{equation}
The goal is to maximize the probability of $x$ conditional on the parameters of the model $theta$. Suppose one has access to a sample of $N$ observations of $x$ with typical element $x_i$. As estimator, one could find the roots of the derivative of the average sample log-likelihood:
begin{equation}
leftlbrace hat{theta} in hat{Theta} : N^{-1} sum_{x_i} frac{partial log p(x_i)} {partial theta} = 0 rightrbrace
end{equation}
and chose the one $theta^star in hat{Theta} $ maximizing the empirical likelihood. There exists many different ways to approximate the derivative of the log-likelihood to facilitate (maybe even permit) its computation. For example, Contrastive Divergence and Persistent Contrastive Divergence are used often. I wonder whether it makes sense to estimate the parameters $theta$ recursively until convergence while continuing to approximate the derivative of the log-likelihood. One could update the parameters after seeing each data point $x_i$ as:
begin{equation}
theta_{i+1} = theta_{i} - eta_i frac{partial log p(x_i)}{partial theta_i}
end{equation}
The practice I learned in Hinton et al. (2006) and in Tieleman (2008) is different though: Both papers define a number of fixed iterations a priori. Could somebody kindly educate me why recursively updating the parameters until convergence is not a good idea? In particular, I'm interested whether there's a theoretical flaw in my reasoning or whether computational capacities dictate sticking to a fixed number of iterations. I am grateful for any help!
neural-network optimization rbm
asked Apr 5 '16 at 6:52
fabianfabian
1765
$endgroup$
Why are the parameters of a Restricted Boltzmann machine trained for a fixed number of iterations (epochs) in many papers instead of choosing the ones corresponding to a stationary point of the likelihood?
Denote the observable data by $x$, hidden data by $h$, the energy function by $E$ and the normalizing constant by $Z$. The probability of $x$ is:
begin{equation}
P(x) = sum_h P(x,h) = sum_h frac{e^{-E(x,h)}}{Z}.
end{equation}
The goal is to maximize the probability of $x$ conditional on the parameters of the model $theta$. Suppose one has access to a sample of $N$ observations of $x$ with typical element $x_i$. As estimator, one could find the roots of the derivative of the average sample log-likelihood:
begin{equation}
leftlbrace hat{theta} in hat{Theta} : N^{-1} sum_{x_i} frac{partial log p(x_i)} {partial theta} = 0 rightrbrace
end{equation}
and chose the one $theta^star in hat{Theta} $ maximizing the empirical likelihood. There exists many different ways to approximate the derivative of the log-likelihood to facilitate (maybe even permit) its computation. For example, Contrastive Divergence and Persistent Contrastive Divergence are used often. I wonder whether it makes sense to estimate the parameters $theta$ recursively until convergence while continuing to approximate the derivative of the log-likelihood. One could update the parameters after seeing each data point $x_i$ as:
begin{equation}
theta_{i+1} = theta_{i} - eta_i frac{partial log p(x_i)}{partial theta_i}
end{equation}
The practice I learned in Hinton et al. (2006) and in Tieleman (2008) is different though: Both papers define a number of fixed iterations a priori. Could somebody kindly educate me why recursively updating the parameters until convergence is not a good idea? In particular, I'm interested whether there's a theoretical flaw in my reasoning or whether computational capacities dictate sticking to a fixed number of iterations. I am grateful for any help!
neural-network optimization rbm
neural-network optimization rbm
asked Apr 5 '16 at 6:52
fabianfabian
1765
asked Apr 5 '16 at 6:52
fabianfabian
1765
edited Apr 5 '16 at 8:31
fabian
asked Apr 5 '16 at 6:52
fabianfabian
1765
asked Apr 5 '16 at 6:52
fabianfabian
1765
asked Apr 5 '16 at 6:52
fabianfabian
1765
1765
bumped to the homepage by Community♦ 3 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 3 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
I believe the problem is that log-likelihood is not directly computable, because of exponential in number of units complexity. There exist different proxies to the true log-likelihood, for instance pseudo log-likelihood (more here), and in principle you can train RBM until PLL is not changed too much.
Hovewer, there is a lot of randomness involved in the training process, so PLL most likely will be quite noisy (even true log-likelihood would be I believe).
answered Sep 17 '17 at 19:29
yellyell
665
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f11042%2ftraining-the-parameters-of-a-restricted-boltzman-machine%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
I believe the problem is that log-likelihood is not directly computable, because of exponential in number of units complexity. There exist different proxies to the true log-likelihood, for instance pseudo log-likelihood (more here), and in principle you can train RBM until PLL is not changed too much.
Hovewer, there is a lot of randomness involved in the training process, so PLL most likely will be quite noisy (even true log-likelihood would be I believe).
answered Sep 17 '17 at 19:29
yellyell
665
$endgroup$
add a comment |
$begingroup$
I believe the problem is that log-likelihood is not directly computable, because of exponential in number of units complexity. There exist different proxies to the true log-likelihood, for instance pseudo log-likelihood (more here), and in principle you can train RBM until PLL is not changed too much.
Hovewer, there is a lot of randomness involved in the training process, so PLL most likely will be quite noisy (even true log-likelihood would be I believe).
answered Sep 17 '17 at 19:29
yellyell
665
$endgroup$
add a comment |
$begingroup$
I believe the problem is that log-likelihood is not directly computable, because of exponential in number of units complexity. There exist different proxies to the true log-likelihood, for instance pseudo log-likelihood (more here), and in principle you can train RBM until PLL is not changed too much.
Hovewer, there is a lot of randomness involved in the training process, so PLL most likely will be quite noisy (even true log-likelihood would be I believe).
answered Sep 17 '17 at 19:29
yellyell
665
$endgroup$
I believe the problem is that log-likelihood is not directly computable, because of exponential in number of units complexity. There exist different proxies to the true log-likelihood, for instance pseudo log-likelihood (more here), and in principle you can train RBM until PLL is not changed too much.
Hovewer, there is a lot of randomness involved in the training process, so PLL most likely will be quite noisy (even true log-likelihood would be I believe).
answered Sep 17 '17 at 19:29
yellyell
665
edited Sep 17 '17 at 19:54
answered Sep 17 '17 at 19:29
yellyell
665
answered Sep 17 '17 at 19:29
yellyell
665
answered Sep 17 '17 at 19:29
yellyell
665
665
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f11042%2ftraining-the-parameters-of-a-restricted-boltzman-machine%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
