Anomaly detection using clustering of highly correlated Categorical data
$begingroup$
My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else its Anomaly. Likewise there are thousands of combination. I already tried KModes clustering where number of clusters = unique values in column1. However each cluster do not have equal density hence some bad data with high density is classified as normal and good data with low density is marked anomalous.
I want to have unsupervised algo where I can force it to use column1 as primary criteria for clustering. One with the highest frequency of column2 data for each unique value of column1 is good data. Rest is anomalous. Kindly suggest what would be the best algo and how to approach this problem.
scikit-learn clustering categorical-data anomaly-detection
asked Jul 30 '18 at 15:18
viral kapadiaviral kapadia
111
$endgroup$
bumped to the homepage by Community♦ 5 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else its Anomaly. Likewise there are thousands of combination. I already tried KModes clustering where number of clusters = unique values in column1. However each cluster do not have equal density hence some bad data with high density is classified as normal and good data with low density is marked anomalous.
I want to have unsupervised algo where I can force it to use column1 as primary criteria for clustering. One with the highest frequency of column2 data for each unique value of column1 is good data. Rest is anomalous. Kindly suggest what would be the best algo and how to approach this problem.
scikit-learn clustering categorical-data anomaly-detection
asked Jul 30 '18 at 15:18
viral kapadiaviral kapadia
111
$endgroup$
bumped to the homepage by Community♦ 5 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Clustering is likely to fail. There is little use in relying on heuristics like kmeans when the patterns are that obvious. Juat identify the common values (ABC and XYZ are supposedly frequent) by counting not by clustering, and label everything else as anomalous.
$endgroup$
– Anony-Mousse
Jul 30 '18 at 19:49
$begingroup$
Although I am using KModes (not KMeans), your idea of simple counting and label low frequency pattern as outliers definitely makes sense. I will try it out
$endgroup$
– viral kapadia
Jul 31 '18 at 4:17
add a comment |
$begingroup$
My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else its Anomaly. Likewise there are thousands of combination. I already tried KModes clustering where number of clusters = unique values in column1. However each cluster do not have equal density hence some bad data with high density is classified as normal and good data with low density is marked anomalous.
I want to have unsupervised algo where I can force it to use column1 as primary criteria for clustering. One with the highest frequency of column2 data for each unique value of column1 is good data. Rest is anomalous. Kindly suggest what would be the best algo and how to approach this problem.
scikit-learn clustering categorical-data anomaly-detection
asked Jul 30 '18 at 15:18
viral kapadiaviral kapadia
111
$endgroup$
My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else its Anomaly. Likewise there are thousands of combination. I already tried KModes clustering where number of clusters = unique values in column1. However each cluster do not have equal density hence some bad data with high density is classified as normal and good data with low density is marked anomalous.
I want to have unsupervised algo where I can force it to use column1 as primary criteria for clustering. One with the highest frequency of column2 data for each unique value of column1 is good data. Rest is anomalous. Kindly suggest what would be the best algo and how to approach this problem.
scikit-learn clustering categorical-data anomaly-detection
scikit-learn clustering categorical-data anomaly-detection
asked Jul 30 '18 at 15:18
viral kapadiaviral kapadia
111
asked Jul 30 '18 at 15:18
viral kapadiaviral kapadia
111
asked Jul 30 '18 at 15:18
viral kapadiaviral kapadia
111
asked Jul 30 '18 at 15:18
viral kapadiaviral kapadia
111
asked Jul 30 '18 at 15:18
viral kapadiaviral kapadia
111
111
bumped to the homepage by Community♦ 5 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 5 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Clustering is likely to fail. There is little use in relying on heuristics like kmeans when the patterns are that obvious. Juat identify the common values (ABC and XYZ are supposedly frequent) by counting not by clustering, and label everything else as anomalous.
$endgroup$
– Anony-Mousse
Jul 30 '18 at 19:49
$begingroup$
Although I am using KModes (not KMeans), your idea of simple counting and label low frequency pattern as outliers definitely makes sense. I will try it out
$endgroup$
– viral kapadia
Jul 31 '18 at 4:17
add a comment |
$begingroup$
Clustering is likely to fail. There is little use in relying on heuristics like kmeans when the patterns are that obvious. Juat identify the common values (ABC and XYZ are supposedly frequent) by counting not by clustering, and label everything else as anomalous.
$endgroup$
– Anony-Mousse
Jul 30 '18 at 19:49
$begingroup$
Although I am using KModes (not KMeans), your idea of simple counting and label low frequency pattern as outliers definitely makes sense. I will try it out
$endgroup$
– viral kapadia
Jul 31 '18 at 4:17
$begingroup$
Clustering is likely to fail. There is little use in relying on heuristics like kmeans when the patterns are that obvious. Juat identify the common values (ABC and XYZ are supposedly frequent) by counting not by clustering, and label everything else as anomalous.
$endgroup$
– Anony-Mousse
Jul 30 '18 at 19:49
$begingroup$
Clustering is likely to fail. There is little use in relying on heuristics like kmeans when the patterns are that obvious. Juat identify the common values (ABC and XYZ are supposedly frequent) by counting not by clustering, and label everything else as anomalous.
$endgroup$
– Anony-Mousse
Jul 30 '18 at 19:49
$begingroup$
Although I am using KModes (not KMeans), your idea of simple counting and label low frequency pattern as outliers definitely makes sense. I will try it out
$endgroup$
– viral kapadia
Jul 31 '18 at 4:17
$begingroup$
Although I am using KModes (not KMeans), your idea of simple counting and label low frequency pattern as outliers definitely makes sense. I will try it out
$endgroup$
– viral kapadia
Jul 31 '18 at 4:17
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
Your problem is actually a regression problem rather than general clustering, you look for the values far away from the regression line, outliers in the sense of regression. Therefore fit the regression line and filter the values with the largest residual errors which are your "bad" values in the sense of not following the correlation structure given by your two variables.
answered Jul 30 '18 at 15:29
Alex2006Alex2006
25129
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f36205%2fanomaly-detection-using-clustering-of-highly-correlated-categorical-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Your problem is actually a regression problem rather than general clustering, you look for the values far away from the regression line, outliers in the sense of regression. Therefore fit the regression line and filter the values with the largest residual errors which are your "bad" values in the sense of not following the correlation structure given by your two variables.
answered Jul 30 '18 at 15:29
Alex2006Alex2006
25129
$endgroup$
add a comment |
$begingroup$
Your problem is actually a regression problem rather than general clustering, you look for the values far away from the regression line, outliers in the sense of regression. Therefore fit the regression line and filter the values with the largest residual errors which are your "bad" values in the sense of not following the correlation structure given by your two variables.
answered Jul 30 '18 at 15:29
Alex2006Alex2006
25129
$endgroup$
add a comment |
$begingroup$
Your problem is actually a regression problem rather than general clustering, you look for the values far away from the regression line, outliers in the sense of regression. Therefore fit the regression line and filter the values with the largest residual errors which are your "bad" values in the sense of not following the correlation structure given by your two variables.
answered Jul 30 '18 at 15:29
Alex2006Alex2006
25129
$endgroup$
Your problem is actually a regression problem rather than general clustering, you look for the values far away from the regression line, outliers in the sense of regression. Therefore fit the regression line and filter the values with the largest residual errors which are your "bad" values in the sense of not following the correlation structure given by your two variables.
answered Jul 30 '18 at 15:29
Alex2006Alex2006
25129
edited Jul 30 '18 at 15:36
answered Jul 30 '18 at 15:29
Alex2006Alex2006
25129
answered Jul 30 '18 at 15:29
Alex2006Alex2006
25129
answered Jul 30 '18 at 15:29
Alex2006Alex2006
25129
25129
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f36205%2fanomaly-detection-using-clustering-of-highly-correlated-categorical-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Clustering is likely to fail. There is little use in relying on heuristics like kmeans when the patterns are that obvious. Juat identify the common values (ABC and XYZ are supposedly frequent) by counting not by clustering, and label everything else as anomalous.
$endgroup$
– Anony-Mousse
Jul 30 '18 at 19:49
$begingroup$
Although I am using KModes (not KMeans), your idea of simple counting and label low frequency pattern as outliers definitely makes sense. I will try it out
$endgroup$
– viral kapadia
Jul 31 '18 at 4:17