What is the use of torch.no_grad in pytorch?
$begingroup$
I am new to pytorch and started with this github code. I do not understand the comment in line 60-61 in the code "because weights have requires_grad=True, but we don't need to track this in autograd". I understood that we mention requires_grad=True to the variables which we need to calculate the gradients for using autograd but what does it mean to be "tracked by autograd" ?
pytorch
asked Jun 5 '18 at 8:21
flyingDopeflyingDope
413128
$endgroup$
add a comment |
$begingroup$
I am new to pytorch and started with this github code. I do not understand the comment in line 60-61 in the code "because weights have requires_grad=True, but we don't need to track this in autograd". I understood that we mention requires_grad=True to the variables which we need to calculate the gradients for using autograd but what does it mean to be "tracked by autograd" ?
pytorch
asked Jun 5 '18 at 8:21
flyingDopeflyingDope
413128
$endgroup$
add a comment |
$begingroup$
I am new to pytorch and started with this github code. I do not understand the comment in line 60-61 in the code "because weights have requires_grad=True, but we don't need to track this in autograd". I understood that we mention requires_grad=True to the variables which we need to calculate the gradients for using autograd but what does it mean to be "tracked by autograd" ?
pytorch
asked Jun 5 '18 at 8:21
flyingDopeflyingDope
413128
$endgroup$
I am new to pytorch and started with this github code. I do not understand the comment in line 60-61 in the code "because weights have requires_grad=True, but we don't need to track this in autograd". I understood that we mention requires_grad=True to the variables which we need to calculate the gradients for using autograd but what does it mean to be "tracked by autograd" ?
pytorch
pytorch
asked Jun 5 '18 at 8:21
flyingDopeflyingDope
413128
asked Jun 5 '18 at 8:21
flyingDopeflyingDope
413128
asked Jun 5 '18 at 8:21
flyingDopeflyingDope
413128
asked Jun 5 '18 at 8:21
flyingDopeflyingDope
413128
asked Jun 5 '18 at 8:21
flyingDopeflyingDope
413128
413128
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
The wrapper "with torch.no_grad()" temporarily set all the requires_grad flag to false. An example from the official PyTorch tutorial (https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients) :
x = torch.randn(3, requires_grad=True)
print(x.requires_grad)
print((x ** 2).requires_grad)
with torch.no_grad():
print((x ** 2).requires_grad)
Out:
True
True
False
I recommend you to read all the tutorials from the website above.
In your example : I guess the author don't want PyTorch to calculate the gradients of the new defined variables w1 and w2 since he just want to update their values.
edited Aug 5 '18 at 23:36
Stephen Rauch♦
1,52551330
answered Jun 5 '18 at 9:13
Adrien DAdrien D
56317
$endgroup$
add a comment |
$begingroup$
"with torch.no_grad()" will make all the operations in the block have no gradients.
In pytorch, you can't do inplacement changing of w1 and w2, which are two variables with require_grad = True. I think that avoiding the inplacement changing of w1 and w2 is because it will cause error in back propagation calculation. Since inplacement change will totally change w1 and w2.
However, if you use this no_grad(), you can control the new w1 and new w2 have no gradients since they are generated by operations, which means you only change the value of w1 and w2, not gradient part, they still have previous defined variable gradient information and back propagation can continue.
answered 13 mins ago
Jianing LuJianing Lu
1
New contributor
Jianing Lu is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f32651%2fwhat-is-the-use-of-torch-no-grad-in-pytorch%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The wrapper "with torch.no_grad()" temporarily set all the requires_grad flag to false. An example from the official PyTorch tutorial (https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients) :
x = torch.randn(3, requires_grad=True)
print(x.requires_grad)
print((x ** 2).requires_grad)
with torch.no_grad():
print((x ** 2).requires_grad)
Out:
True
True
False
I recommend you to read all the tutorials from the website above.
In your example : I guess the author don't want PyTorch to calculate the gradients of the new defined variables w1 and w2 since he just want to update their values.
edited Aug 5 '18 at 23:36
Stephen Rauch♦
1,52551330
answered Jun 5 '18 at 9:13
Adrien DAdrien D
56317
$endgroup$
add a comment |
$begingroup$
The wrapper "with torch.no_grad()" temporarily set all the requires_grad flag to false. An example from the official PyTorch tutorial (https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients) :
x = torch.randn(3, requires_grad=True)
print(x.requires_grad)
print((x ** 2).requires_grad)
with torch.no_grad():
print((x ** 2).requires_grad)
Out:
True
True
False
I recommend you to read all the tutorials from the website above.
In your example : I guess the author don't want PyTorch to calculate the gradients of the new defined variables w1 and w2 since he just want to update their values.
edited Aug 5 '18 at 23:36
Stephen Rauch♦
1,52551330
answered Jun 5 '18 at 9:13
Adrien DAdrien D
56317
$endgroup$
add a comment |
$begingroup$
The wrapper "with torch.no_grad()" temporarily set all the requires_grad flag to false. An example from the official PyTorch tutorial (https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients) :
x = torch.randn(3, requires_grad=True)
print(x.requires_grad)
print((x ** 2).requires_grad)
with torch.no_grad():
print((x ** 2).requires_grad)
Out:
True
True
False
I recommend you to read all the tutorials from the website above.
In your example : I guess the author don't want PyTorch to calculate the gradients of the new defined variables w1 and w2 since he just want to update their values.
edited Aug 5 '18 at 23:36
Stephen Rauch♦
1,52551330
answered Jun 5 '18 at 9:13
Adrien DAdrien D
56317
$endgroup$
The wrapper "with torch.no_grad()" temporarily set all the requires_grad flag to false. An example from the official PyTorch tutorial (https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients) :
x = torch.randn(3, requires_grad=True)
print(x.requires_grad)
print((x ** 2).requires_grad)
with torch.no_grad():
print((x ** 2).requires_grad)
Out:
True
True
False
I recommend you to read all the tutorials from the website above.
In your example : I guess the author don't want PyTorch to calculate the gradients of the new defined variables w1 and w2 since he just want to update their values.
edited Aug 5 '18 at 23:36
Stephen Rauch♦
1,52551330
answered Jun 5 '18 at 9:13
Adrien DAdrien D
56317
edited Aug 5 '18 at 23:36
Stephen Rauch♦
1,52551330
edited Aug 5 '18 at 23:36
Stephen Rauch♦
1,52551330
edited Aug 5 '18 at 23:36
Stephen Rauch♦
1,52551330
1,52551330
answered Jun 5 '18 at 9:13
Adrien DAdrien D
56317
answered Jun 5 '18 at 9:13
Adrien DAdrien D
56317
answered Jun 5 '18 at 9:13
Adrien DAdrien D
56317
56317
add a comment |
add a comment |
$begingroup$
"with torch.no_grad()" will make all the operations in the block have no gradients.
In pytorch, you can't do inplacement changing of w1 and w2, which are two variables with require_grad = True. I think that avoiding the inplacement changing of w1 and w2 is because it will cause error in back propagation calculation. Since inplacement change will totally change w1 and w2.
However, if you use this no_grad(), you can control the new w1 and new w2 have no gradients since they are generated by operations, which means you only change the value of w1 and w2, not gradient part, they still have previous defined variable gradient information and back propagation can continue.
answered 13 mins ago
Jianing LuJianing Lu
1
New contributor
Jianing Lu is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
"with torch.no_grad()" will make all the operations in the block have no gradients.
In pytorch, you can't do inplacement changing of w1 and w2, which are two variables with require_grad = True. I think that avoiding the inplacement changing of w1 and w2 is because it will cause error in back propagation calculation. Since inplacement change will totally change w1 and w2.
However, if you use this no_grad(), you can control the new w1 and new w2 have no gradients since they are generated by operations, which means you only change the value of w1 and w2, not gradient part, they still have previous defined variable gradient information and back propagation can continue.
answered 13 mins ago
Jianing LuJianing Lu
1
New contributor
Jianing Lu is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
"with torch.no_grad()" will make all the operations in the block have no gradients.
In pytorch, you can't do inplacement changing of w1 and w2, which are two variables with require_grad = True. I think that avoiding the inplacement changing of w1 and w2 is because it will cause error in back propagation calculation. Since inplacement change will totally change w1 and w2.
However, if you use this no_grad(), you can control the new w1 and new w2 have no gradients since they are generated by operations, which means you only change the value of w1 and w2, not gradient part, they still have previous defined variable gradient information and back propagation can continue.
answered 13 mins ago
Jianing LuJianing Lu
1
New contributor
Jianing Lu is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
"with torch.no_grad()" will make all the operations in the block have no gradients.
In pytorch, you can't do inplacement changing of w1 and w2, which are two variables with require_grad = True. I think that avoiding the inplacement changing of w1 and w2 is because it will cause error in back propagation calculation. Since inplacement change will totally change w1 and w2.
However, if you use this no_grad(), you can control the new w1 and new w2 have no gradients since they are generated by operations, which means you only change the value of w1 and w2, not gradient part, they still have previous defined variable gradient information and back propagation can continue.
answered 13 mins ago
Jianing LuJianing Lu
1
New contributor
Jianing Lu is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered 13 mins ago
Jianing LuJianing Lu
1
New contributor
Jianing Lu is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered 13 mins ago
Jianing LuJianing Lu
1
answered 13 mins ago
Jianing LuJianing Lu
1
1
New contributor
Jianing Lu is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Jianing Lu is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Jianing Lu is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f32651%2fwhat-is-the-use-of-torch-no-grad-in-pytorch%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
