what could this mean if your “elbow curve” looks like this?
$begingroup$
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.
Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?
I'm computing the silhouette distance using a 80-20 train test split.
machine-learning k-means
asked 7 hours ago
MrLMrL
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.
Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?
I'm computing the silhouette distance using a 80-20 train test split.
machine-learning k-means
asked 7 hours ago
MrLMrL
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
2 hours ago
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
54 mins ago
add a comment |
$begingroup$
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.
Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?
I'm computing the silhouette distance using a 80-20 train test split.
machine-learning k-means
asked 7 hours ago
MrLMrL
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.
Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?
I'm computing the silhouette distance using a 80-20 train test split.
machine-learning k-means
machine-learning k-means
asked 7 hours ago
MrLMrL
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 7 hours ago
MrLMrL
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 7 hours ago
MrLMrL
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 7 hours ago
MrLMrL
62
asked 7 hours ago
MrLMrL
62
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
2 hours ago
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
54 mins ago
add a comment |
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
2 hours ago
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
54 mins ago
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
2 hours ago
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
2 hours ago
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
54 mins ago
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
54 mins ago
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered 42 mins ago
Djib2011Djib2011
2,60231125
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
MrL is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48883%2fwhat-could-this-mean-if-your-elbow-curve-looks-like-this%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered 42 mins ago
Djib2011Djib2011
2,60231125
$endgroup$
add a comment |
$begingroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered 42 mins ago
Djib2011Djib2011
2,60231125
$endgroup$
add a comment |
$begingroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered 42 mins ago
Djib2011Djib2011
2,60231125
$endgroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered 42 mins ago
Djib2011Djib2011
2,60231125
answered 42 mins ago
Djib2011Djib2011
2,60231125
answered 42 mins ago
Djib2011Djib2011
2,60231125
answered 42 mins ago
Djib2011Djib2011
2,60231125
2,60231125
add a comment |
add a comment |
MrL is a new contributor. Be nice, and check out our Code of Conduct.
MrL is a new contributor. Be nice, and check out our Code of Conduct.
MrL is a new contributor. Be nice, and check out our Code of Conduct.
MrL is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48883%2fwhat-could-this-mean-if-your-elbow-curve-looks-like-this%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
2 hours ago
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
54 mins ago