How to train non image data in batches from disk?
$begingroup$
I am working on a project where I have 50 .npy files with each of shape (77156, 30, 50, 1) representing 77156 IDs and 1 label file of shape (77156,2) (one column for ID, one for label). The data is such that the first row of all files (label and 50 data files) represent one ID, the second row represents another ID and so on. The model takes all 50 matrices of each ID at once along with the label to train.
Being said that, I am facing the common problem of training on multiple large files in Keras which are combinely too large to fit on GPU memory.
I have tried many examples and come to the conclusion of making a generator for fit_generator which would pick the data in batches and feed it to fit_generator. This is the function I need to design myself.
And I have no idea how to implement it. Any suggestions?
neural-network keras training
asked 6 mins ago
yamini goelyamini goel
11
$endgroup$
add a comment |
$begingroup$
I am working on a project where I have 50 .npy files with each of shape (77156, 30, 50, 1) representing 77156 IDs and 1 label file of shape (77156,2) (one column for ID, one for label). The data is such that the first row of all files (label and 50 data files) represent one ID, the second row represents another ID and so on. The model takes all 50 matrices of each ID at once along with the label to train.
Being said that, I am facing the common problem of training on multiple large files in Keras which are combinely too large to fit on GPU memory.
I have tried many examples and come to the conclusion of making a generator for fit_generator which would pick the data in batches and feed it to fit_generator. This is the function I need to design myself.
And I have no idea how to implement it. Any suggestions?
neural-network keras training
asked 6 mins ago
yamini goelyamini goel
11
$endgroup$
add a comment |
$begingroup$
I am working on a project where I have 50 .npy files with each of shape (77156, 30, 50, 1) representing 77156 IDs and 1 label file of shape (77156,2) (one column for ID, one for label). The data is such that the first row of all files (label and 50 data files) represent one ID, the second row represents another ID and so on. The model takes all 50 matrices of each ID at once along with the label to train.
Being said that, I am facing the common problem of training on multiple large files in Keras which are combinely too large to fit on GPU memory.
I have tried many examples and come to the conclusion of making a generator for fit_generator which would pick the data in batches and feed it to fit_generator. This is the function I need to design myself.
And I have no idea how to implement it. Any suggestions?
neural-network keras training
asked 6 mins ago
yamini goelyamini goel
11
$endgroup$
I am working on a project where I have 50 .npy files with each of shape (77156, 30, 50, 1) representing 77156 IDs and 1 label file of shape (77156,2) (one column for ID, one for label). The data is such that the first row of all files (label and 50 data files) represent one ID, the second row represents another ID and so on. The model takes all 50 matrices of each ID at once along with the label to train.
Being said that, I am facing the common problem of training on multiple large files in Keras which are combinely too large to fit on GPU memory.
I have tried many examples and come to the conclusion of making a generator for fit_generator which would pick the data in batches and feed it to fit_generator. This is the function I need to design myself.
And I have no idea how to implement it. Any suggestions?
neural-network keras training
neural-network keras training
asked 6 mins ago
yamini goelyamini goel
11
asked 6 mins ago
yamini goelyamini goel
11
asked 6 mins ago
yamini goelyamini goel
11
asked 6 mins ago
yamini goelyamini goel
11
asked 6 mins ago
yamini goelyamini goel
11
11
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f46738%2fhow-to-train-non-image-data-in-batches-from-disk%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f46738%2fhow-to-train-non-image-data-in-batches-from-disk%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
