Estimating class prevalence in unlabelled data after predicting labels with a binary classifier
$begingroup$
I'm looking to get an estimate of the prevalence of 1's (i.e. the rate of positive labels) in a very large dataset that I have. However, I am hoping to report this percentage as a 95% credible interval instead of as an exact estimate of rate, taking into account the model uncertainties.
These are the steps I'm hoping to perform:
- Train a binary classifier on labelled training data.
- Use a labelled test set to estimate the specificity and sensitivity of the classifier.
- Use the classifier to predict the label for the unlabelled records in the dataset.
- Obviously I could get an exact prevalence estimate by simply calculating the mean of the predicted outputs. But this is where I'm hoping to implement an approach for reporting the prevalence estimate as an interval.
So my question is: Is there a best-practice approach to doing this? I found this study which trains a binary classifier and then uses a Bayesian prevalence model to report the prevalence as a 95% confidence interval by incorporating the uncertainty associated with the model specificity and sensitivity. However, I'm having trouble understanding exactly what they did here. I'm also not finding many others who have done something similar. So, any suggestions for a reliable approach I could take to do this would be greatly appreciated.
Thanks in advance!
machine-learning classification statistics bayesian
asked Oct 19 '18 at 18:26
CadPat CadPat
12
$endgroup$
bumped to the homepage by Community♦ 6 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I'm looking to get an estimate of the prevalence of 1's (i.e. the rate of positive labels) in a very large dataset that I have. However, I am hoping to report this percentage as a 95% credible interval instead of as an exact estimate of rate, taking into account the model uncertainties.
These are the steps I'm hoping to perform:
- Train a binary classifier on labelled training data.
- Use a labelled test set to estimate the specificity and sensitivity of the classifier.
- Use the classifier to predict the label for the unlabelled records in the dataset.
- Obviously I could get an exact prevalence estimate by simply calculating the mean of the predicted outputs. But this is where I'm hoping to implement an approach for reporting the prevalence estimate as an interval.
So my question is: Is there a best-practice approach to doing this? I found this study which trains a binary classifier and then uses a Bayesian prevalence model to report the prevalence as a 95% confidence interval by incorporating the uncertainty associated with the model specificity and sensitivity. However, I'm having trouble understanding exactly what they did here. I'm also not finding many others who have done something similar. So, any suggestions for a reliable approach I could take to do this would be greatly appreciated.
Thanks in advance!
machine-learning classification statistics bayesian
asked Oct 19 '18 at 18:26
CadPat CadPat
12
$endgroup$
bumped to the homepage by Community♦ 6 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I'm looking to get an estimate of the prevalence of 1's (i.e. the rate of positive labels) in a very large dataset that I have. However, I am hoping to report this percentage as a 95% credible interval instead of as an exact estimate of rate, taking into account the model uncertainties.
These are the steps I'm hoping to perform:
- Train a binary classifier on labelled training data.
- Use a labelled test set to estimate the specificity and sensitivity of the classifier.
- Use the classifier to predict the label for the unlabelled records in the dataset.
- Obviously I could get an exact prevalence estimate by simply calculating the mean of the predicted outputs. But this is where I'm hoping to implement an approach for reporting the prevalence estimate as an interval.
So my question is: Is there a best-practice approach to doing this? I found this study which trains a binary classifier and then uses a Bayesian prevalence model to report the prevalence as a 95% confidence interval by incorporating the uncertainty associated with the model specificity and sensitivity. However, I'm having trouble understanding exactly what they did here. I'm also not finding many others who have done something similar. So, any suggestions for a reliable approach I could take to do this would be greatly appreciated.
Thanks in advance!
machine-learning classification statistics bayesian
asked Oct 19 '18 at 18:26
CadPat CadPat
12
$endgroup$
I'm looking to get an estimate of the prevalence of 1's (i.e. the rate of positive labels) in a very large dataset that I have. However, I am hoping to report this percentage as a 95% credible interval instead of as an exact estimate of rate, taking into account the model uncertainties.
These are the steps I'm hoping to perform:
- Train a binary classifier on labelled training data.
- Use a labelled test set to estimate the specificity and sensitivity of the classifier.
- Use the classifier to predict the label for the unlabelled records in the dataset.
- Obviously I could get an exact prevalence estimate by simply calculating the mean of the predicted outputs. But this is where I'm hoping to implement an approach for reporting the prevalence estimate as an interval.
So my question is: Is there a best-practice approach to doing this? I found this study which trains a binary classifier and then uses a Bayesian prevalence model to report the prevalence as a 95% confidence interval by incorporating the uncertainty associated with the model specificity and sensitivity. However, I'm having trouble understanding exactly what they did here. I'm also not finding many others who have done something similar. So, any suggestions for a reliable approach I could take to do this would be greatly appreciated.
Thanks in advance!
machine-learning classification statistics bayesian
machine-learning classification statistics bayesian
asked Oct 19 '18 at 18:26
CadPat CadPat
12
asked Oct 19 '18 at 18:26
CadPat CadPat
12
edited Oct 24 '18 at 16:01
CadPat
asked Oct 19 '18 at 18:26
CadPat CadPat
12
asked Oct 19 '18 at 18:26
CadPat CadPat
12
asked Oct 19 '18 at 18:26
CadPat CadPat
12
12
bumped to the homepage by Community♦ 6 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 6 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
There's a domain named quantification that deals with this kind of problem. It aims to create "quantifiers" (instead of classifiers) that will focus more on estimating the prevalence of a class in a population rather than on individual classifications.
An easy approach is "adjusted count" (AC), but there are other (potentially better) approaches. You can find more in this paper or this one.
Basically, the idea of AC is:
1.1) Learn a binary classifier from the train dataset
1.2) Estimate the False Positive Rate (fpr) and True Positive Rate (tpr) from the training set, using cross-validation
2) Estimate prevalence of the test set based on the observed prevalence in the test set, corrected by estimated fpr and tpr (I guess the ideal is to have different test sets with different prevalences)
That way, you can estimate the prevalence of your sample based on the fraction of predicted positives that are actually positive, and the fraction of predicted negatives that are actually positive (and I guess you can easily compute confidence intervals).
The good thing with this is that your model will be way more robust to a change of prevalence in a population, rather than if you just count the positive and negative instances. (All this is explained better in the link papers)
answered Jan 5 at 2:58
KjianKjian
11
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f39936%2festimating-class-prevalence-in-unlabelled-data-after-predicting-labels-with-a-bi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
There's a domain named quantification that deals with this kind of problem. It aims to create "quantifiers" (instead of classifiers) that will focus more on estimating the prevalence of a class in a population rather than on individual classifications.
An easy approach is "adjusted count" (AC), but there are other (potentially better) approaches. You can find more in this paper or this one.
Basically, the idea of AC is:
1.1) Learn a binary classifier from the train dataset
1.2) Estimate the False Positive Rate (fpr) and True Positive Rate (tpr) from the training set, using cross-validation
2) Estimate prevalence of the test set based on the observed prevalence in the test set, corrected by estimated fpr and tpr (I guess the ideal is to have different test sets with different prevalences)
That way, you can estimate the prevalence of your sample based on the fraction of predicted positives that are actually positive, and the fraction of predicted negatives that are actually positive (and I guess you can easily compute confidence intervals).
The good thing with this is that your model will be way more robust to a change of prevalence in a population, rather than if you just count the positive and negative instances. (All this is explained better in the link papers)
answered Jan 5 at 2:58
KjianKjian
11
$endgroup$
add a comment |
$begingroup$
There's a domain named quantification that deals with this kind of problem. It aims to create "quantifiers" (instead of classifiers) that will focus more on estimating the prevalence of a class in a population rather than on individual classifications.
An easy approach is "adjusted count" (AC), but there are other (potentially better) approaches. You can find more in this paper or this one.
Basically, the idea of AC is:
1.1) Learn a binary classifier from the train dataset
1.2) Estimate the False Positive Rate (fpr) and True Positive Rate (tpr) from the training set, using cross-validation
2) Estimate prevalence of the test set based on the observed prevalence in the test set, corrected by estimated fpr and tpr (I guess the ideal is to have different test sets with different prevalences)
That way, you can estimate the prevalence of your sample based on the fraction of predicted positives that are actually positive, and the fraction of predicted negatives that are actually positive (and I guess you can easily compute confidence intervals).
The good thing with this is that your model will be way more robust to a change of prevalence in a population, rather than if you just count the positive and negative instances. (All this is explained better in the link papers)
answered Jan 5 at 2:58
KjianKjian
11
$endgroup$
add a comment |
$begingroup$
There's a domain named quantification that deals with this kind of problem. It aims to create "quantifiers" (instead of classifiers) that will focus more on estimating the prevalence of a class in a population rather than on individual classifications.
An easy approach is "adjusted count" (AC), but there are other (potentially better) approaches. You can find more in this paper or this one.
Basically, the idea of AC is:
1.1) Learn a binary classifier from the train dataset
1.2) Estimate the False Positive Rate (fpr) and True Positive Rate (tpr) from the training set, using cross-validation
2) Estimate prevalence of the test set based on the observed prevalence in the test set, corrected by estimated fpr and tpr (I guess the ideal is to have different test sets with different prevalences)
That way, you can estimate the prevalence of your sample based on the fraction of predicted positives that are actually positive, and the fraction of predicted negatives that are actually positive (and I guess you can easily compute confidence intervals).
The good thing with this is that your model will be way more robust to a change of prevalence in a population, rather than if you just count the positive and negative instances. (All this is explained better in the link papers)
answered Jan 5 at 2:58
KjianKjian
11
$endgroup$
There's a domain named quantification that deals with this kind of problem. It aims to create "quantifiers" (instead of classifiers) that will focus more on estimating the prevalence of a class in a population rather than on individual classifications.
An easy approach is "adjusted count" (AC), but there are other (potentially better) approaches. You can find more in this paper or this one.
Basically, the idea of AC is:
1.1) Learn a binary classifier from the train dataset
1.2) Estimate the False Positive Rate (fpr) and True Positive Rate (tpr) from the training set, using cross-validation
2) Estimate prevalence of the test set based on the observed prevalence in the test set, corrected by estimated fpr and tpr (I guess the ideal is to have different test sets with different prevalences)
That way, you can estimate the prevalence of your sample based on the fraction of predicted positives that are actually positive, and the fraction of predicted negatives that are actually positive (and I guess you can easily compute confidence intervals).
The good thing with this is that your model will be way more robust to a change of prevalence in a population, rather than if you just count the positive and negative instances. (All this is explained better in the link papers)
answered Jan 5 at 2:58
KjianKjian
11
answered Jan 5 at 2:58
KjianKjian
11
answered Jan 5 at 2:58
KjianKjian
11
answered Jan 5 at 2:58
KjianKjian
11
11
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f39936%2festimating-class-prevalence-in-unlabelled-data-after-predicting-labels-with-a-bi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
