Performing Named Entity Recognition (NER) with Unreliable Spacing
$begingroup$
I have a large corpus of physical places I would like to apply named entity recognition to. The places are typically short strings, but due to human typos often have critical spaces missing, e.g., "bob's hoteltoronto ontario". I should also add that all of the strings are entirely lowercase, hence my inclination to use a neural network here.
To take the example above, I would like the model to recognize "bob's hotel", "toronto" and "ontario" as separate nouns. However, when I look up how to structure training data for NER models, they always seem to rely on the presence of spaces, e.g.:
U.N. NNP I-NP I-ORG
official NN I-NP O
Ekeus NNP I-NP I-PER
heads VBZ I-VP O
(Source: CoNLL 2003 dataset)
So, in short, I would like to train my model with some examples that do not contain the correct spaces, such as in the example above. How can I do this?
Notes:
One solution, of course, is to use an algorithm to split the data ahead of time (e.g., wordnija). However, I would prefer to leave this for the net to solve.
I should add that I have loads of labeled data.
deep-learning nlp named-entity-recognition
asked 3 mins ago
thpwthpw
1
New contributor
thpw is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
I have a large corpus of physical places I would like to apply named entity recognition to. The places are typically short strings, but due to human typos often have critical spaces missing, e.g., "bob's hoteltoronto ontario". I should also add that all of the strings are entirely lowercase, hence my inclination to use a neural network here.
To take the example above, I would like the model to recognize "bob's hotel", "toronto" and "ontario" as separate nouns. However, when I look up how to structure training data for NER models, they always seem to rely on the presence of spaces, e.g.:
U.N. NNP I-NP I-ORG
official NN I-NP O
Ekeus NNP I-NP I-PER
heads VBZ I-VP O
(Source: CoNLL 2003 dataset)
So, in short, I would like to train my model with some examples that do not contain the correct spaces, such as in the example above. How can I do this?
Notes:
One solution, of course, is to use an algorithm to split the data ahead of time (e.g., wordnija). However, I would prefer to leave this for the net to solve.
I should add that I have loads of labeled data.
deep-learning nlp named-entity-recognition
asked 3 mins ago
thpwthpw
1
New contributor
thpw is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
I have a large corpus of physical places I would like to apply named entity recognition to. The places are typically short strings, but due to human typos often have critical spaces missing, e.g., "bob's hoteltoronto ontario". I should also add that all of the strings are entirely lowercase, hence my inclination to use a neural network here.
To take the example above, I would like the model to recognize "bob's hotel", "toronto" and "ontario" as separate nouns. However, when I look up how to structure training data for NER models, they always seem to rely on the presence of spaces, e.g.:
U.N. NNP I-NP I-ORG
official NN I-NP O
Ekeus NNP I-NP I-PER
heads VBZ I-VP O
(Source: CoNLL 2003 dataset)
So, in short, I would like to train my model with some examples that do not contain the correct spaces, such as in the example above. How can I do this?
Notes:
One solution, of course, is to use an algorithm to split the data ahead of time (e.g., wordnija). However, I would prefer to leave this for the net to solve.
I should add that I have loads of labeled data.
deep-learning nlp named-entity-recognition
asked 3 mins ago
thpwthpw
1
New contributor
thpw is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
I have a large corpus of physical places I would like to apply named entity recognition to. The places are typically short strings, but due to human typos often have critical spaces missing, e.g., "bob's hoteltoronto ontario". I should also add that all of the strings are entirely lowercase, hence my inclination to use a neural network here.
To take the example above, I would like the model to recognize "bob's hotel", "toronto" and "ontario" as separate nouns. However, when I look up how to structure training data for NER models, they always seem to rely on the presence of spaces, e.g.:
U.N. NNP I-NP I-ORG
official NN I-NP O
Ekeus NNP I-NP I-PER
heads VBZ I-VP O
(Source: CoNLL 2003 dataset)
So, in short, I would like to train my model with some examples that do not contain the correct spaces, such as in the example above. How can I do this?
Notes:
One solution, of course, is to use an algorithm to split the data ahead of time (e.g., wordnija). However, I would prefer to leave this for the net to solve.
I should add that I have loads of labeled data.
deep-learning nlp named-entity-recognition
deep-learning nlp named-entity-recognition
asked 3 mins ago
thpwthpw
1
New contributor
thpw is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 3 mins ago
thpwthpw
1
New contributor
thpw is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 3 mins ago
thpwthpw
1
New contributor
thpw is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 3 mins ago
thpwthpw
1
asked 3 mins ago
thpwthpw
1
1
New contributor
thpw is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
thpw is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
thpw is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
thpw is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f44502%2fperforming-named-entity-recognition-ner-with-unreliable-spacing%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
thpw is a new contributor. Be nice, and check out our Code of Conduct.
thpw is a new contributor. Be nice, and check out our Code of Conduct.
thpw is a new contributor. Be nice, and check out our Code of Conduct.
thpw is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f44502%2fperforming-named-entity-recognition-ner-with-unreliable-spacing%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
