what could this mean if your “elbow curve” looks like this?2019 Community Moderator ElectionWhat does 'contextual' mean in 'contextual bandits'?What does “zero-meaned vector” meanWhat did Geoffrey Hinton mean when he said this?What do mean and variance mean for high dimensional data?Is removing poorly predicted data points a valid approach?What does Logits in machine learning mean?What approach other than Tf-Idf could I use for text-clustering using K-Means?What algorithm could be used to fuzzy merge multiple datasets?What is wrong with my Precision-Recall curve?What should the neural network output look like for this implementation?
Multi tool use
Is Social Media Science Fiction?
How did the USSR manage to innovate in an environment characterized by government censorship and high bureaucracy?
Does bootstrapped regression allow for inference?
Could a US political party gain complete control over the government by removing checks & balances?
What is the command to reset a PC without deleting any files
Finding files for which a command fails
Can I find out the caloric content of bread by dehydrating it?
Domain expired, GoDaddy holds it and is asking more money
"My colleague's body is amazing"
When blogging recipes, how can I support both readers who want the narrative/journey and ones who want the printer-friendly recipe?
Copycat chess is back
Is this food a bread or a loaf?
What are the advantages and disadvantages of running one shots compared to campaigns?
Was there ever an axiom rendered a theorem?
Is ipsum/ipsa/ipse a third person pronoun, or can it serve other functions?
Is there any use for defining additional entity types in a SOQL FROM clause?
Landing in very high winds
Typesetting a double Over Dot on top of a symbol
Why doesn't a const reference extend the life of a temporary object passed via a function?
Email Account under attack (really) - anything I can do?
Is a vector space a subspace of itself?
What do the Banks children have against barley water?
Where to refill my bottle in India?
What does "enim et" mean?
what could this mean if your “elbow curve” looks like this?
2019 Community Moderator ElectionWhat does 'contextual' mean in 'contextual bandits'?What does “zero-meaned vector” meanWhat did Geoffrey Hinton mean when he said this?What do mean and variance mean for high dimensional data?Is removing poorly predicted data points a valid approach?What does Logits in machine learning mean?What approach other than Tf-Idf could I use for text-clustering using K-Means?What algorithm could be used to fuzzy merge multiple datasets?What is wrong with my Precision-Recall curve?What should the neural network output look like for this implementation?
$begingroup$
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.
Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?
I'm computing the silhouette distance using a 80-20 train test split.
machine-learning k-means
asked 14 hours ago
MrLMrL
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.
Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?
I'm computing the silhouette distance using a 80-20 train test split.
machine-learning k-means
asked 14 hours ago
MrLMrL
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
8 hours ago
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
7 hours ago
add a comment |
$begingroup$
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.
Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?
I'm computing the silhouette distance using a 80-20 train test split.
machine-learning k-means
asked 14 hours ago
MrLMrL
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.
Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?
I'm computing the silhouette distance using a 80-20 train test split.
machine-learning k-means
machine-learning k-means
asked 14 hours ago
MrLMrL
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 14 hours ago
MrLMrL
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 14 hours ago
MrLMrL
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 14 hours ago
MrLMrL
62
asked 14 hours ago
MrLMrL
62
62
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
8 hours ago
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
7 hours ago
add a comment |
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
8 hours ago
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
7 hours ago
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
8 hours ago
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
8 hours ago
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
7 hours ago
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
7 hours ago
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered 7 hours ago
Djib2011Djib2011
2,61231125
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
MrL is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48883%2fwhat-could-this-mean-if-your-elbow-curve-looks-like-this%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered 7 hours ago
Djib2011Djib2011
2,61231125
$endgroup$
add a comment |
$begingroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered 7 hours ago
Djib2011Djib2011
2,61231125
$endgroup$
add a comment |
$begingroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered 7 hours ago
Djib2011Djib2011
2,61231125
$endgroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered 7 hours ago
Djib2011Djib2011
2,61231125
answered 7 hours ago
Djib2011Djib2011
2,61231125
answered 7 hours ago
Djib2011Djib2011
2,61231125
answered 7 hours ago
Djib2011Djib2011
2,61231125
2,61231125
add a comment |
add a comment |
MrL is a new contributor. Be nice, and check out our Code of Conduct.
MrL is a new contributor. Be nice, and check out our Code of Conduct.
MrL is a new contributor. Be nice, and check out our Code of Conduct.
MrL is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48883%2fwhat-could-this-mean-if-your-elbow-curve-looks-like-this%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
LjPmlimkn g4C hrPF1,F3,i7elMHORaeQQ8CphNkjJVT9Cfc V,yA,32dOk3Lm889shtP9LGhc97FS,CpV5,cETnOZLW,jCrg t3SJz
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
8 hours ago
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
7 hours ago