what could this mean if your “elbow curve” looks like this?2019 Community Moderator ElectionWhat does 'contextual' mean in 'contextual bandits'?What does “zero-meaned vector” meanWhat did Geoffrey Hinton mean when he said this?What do mean and variance mean for high dimensional data?Is removing poorly predicted data points a valid approach?What does Logits in machine learning mean?What approach other than Tf-Idf could I use for text-clustering using K-Means?What algorithm could be used to fuzzy merge multiple datasets?What is wrong with my Precision-Recall curve?What should the neural network output look like for this implementation?

Multi tool use
Multi tool use

Is Social Media Science Fiction?

How did the USSR manage to innovate in an environment characterized by government censorship and high bureaucracy?

Does bootstrapped regression allow for inference?

Could a US political party gain complete control over the government by removing checks & balances?

What is the command to reset a PC without deleting any files

Finding files for which a command fails

Can I find out the caloric content of bread by dehydrating it?

Domain expired, GoDaddy holds it and is asking more money

"My colleague's body is amazing"

When blogging recipes, how can I support both readers who want the narrative/journey and ones who want the printer-friendly recipe?

Copycat chess is back

Is this food a bread or a loaf?

What are the advantages and disadvantages of running one shots compared to campaigns?

Was there ever an axiom rendered a theorem?

Is ipsum/ipsa/ipse a third person pronoun, or can it serve other functions?

Is there any use for defining additional entity types in a SOQL FROM clause?

Landing in very high winds

Typesetting a double Over Dot on top of a symbol

Why doesn't a const reference extend the life of a temporary object passed via a function?

Email Account under attack (really) - anything I can do?

Is a vector space a subspace of itself?

What do the Banks children have against barley water?

Where to refill my bottle in India?

What does "enim et" mean?



what could this mean if your “elbow curve” looks like this?



2019 Community Moderator ElectionWhat does 'contextual' mean in 'contextual bandits'?What does “zero-meaned vector” meanWhat did Geoffrey Hinton mean when he said this?What do mean and variance mean for high dimensional data?Is removing poorly predicted data points a valid approach?What does Logits in machine learning mean?What approach other than Tf-Idf could I use for text-clustering using K-Means?What algorithm could be used to fuzzy merge multiple datasets?What is wrong with my Precision-Recall curve?What should the neural network output look like for this implementation?










1












$begingroup$


enter image description here



This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.



Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?



I'm computing the silhouette distance using a 80-20 train test split.










share|improve this question







New contributor




MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.







$endgroup$











  • $begingroup$
    So, what’s the size of your data?
    $endgroup$
    – pythinker
    8 hours ago










  • $begingroup$
    few thousand rows , TFIDF based clustering ~ 50 000 features
    $endgroup$
    – MrL
    7 hours ago















1












$begingroup$


enter image description here



This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.



Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?



I'm computing the silhouette distance using a 80-20 train test split.










share|improve this question







New contributor




MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.







$endgroup$











  • $begingroup$
    So, what’s the size of your data?
    $endgroup$
    – pythinker
    8 hours ago










  • $begingroup$
    few thousand rows , TFIDF based clustering ~ 50 000 features
    $endgroup$
    – MrL
    7 hours ago













1












1








1





$begingroup$


enter image description here



This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.



Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?



I'm computing the silhouette distance using a 80-20 train test split.










share|improve this question







New contributor




MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.







$endgroup$




enter image description here



This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.



Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?



I'm computing the silhouette distance using a 80-20 train test split.







machine-learning k-means






share|improve this question







New contributor




MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











share|improve this question







New contributor




MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question






New contributor




MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked 14 hours ago









MrLMrL

62




62




New contributor




MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






MrL is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











  • $begingroup$
    So, what’s the size of your data?
    $endgroup$
    – pythinker
    8 hours ago










  • $begingroup$
    few thousand rows , TFIDF based clustering ~ 50 000 features
    $endgroup$
    – MrL
    7 hours ago
















  • $begingroup$
    So, what’s the size of your data?
    $endgroup$
    – pythinker
    8 hours ago










  • $begingroup$
    few thousand rows , TFIDF based clustering ~ 50 000 features
    $endgroup$
    – MrL
    7 hours ago















$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
8 hours ago




$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
8 hours ago












$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
7 hours ago




$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
7 hours ago










1 Answer
1






active

oldest

votes


















1












$begingroup$

First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:





Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).



Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).



The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).






share|improve this answer









$endgroup$













    Your Answer





    StackExchange.ifUsing("editor", function ()
    return StackExchange.using("mathjaxEditing", function ()
    StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
    StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
    );
    );
    , "mathjax-editing");

    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "557"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader:
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    ,
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );






    MrL is a new contributor. Be nice, and check out our Code of Conduct.









    draft saved

    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48883%2fwhat-could-this-mean-if-your-elbow-curve-looks-like-this%23new-answer', 'question_page');

    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    1












    $begingroup$

    First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:





    Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).



    Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).



    The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).






    share|improve this answer









    $endgroup$

















      1












      $begingroup$

      First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:





      Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).



      Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).



      The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).






      share|improve this answer









      $endgroup$















        1












        1








        1





        $begingroup$

        First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:





        Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).



        Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).



        The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).






        share|improve this answer









        $endgroup$



        First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:





        Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).



        Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).



        The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered 7 hours ago









        Djib2011Djib2011

        2,61231125




        2,61231125




















            MrL is a new contributor. Be nice, and check out our Code of Conduct.









            draft saved

            draft discarded


















            MrL is a new contributor. Be nice, and check out our Code of Conduct.












            MrL is a new contributor. Be nice, and check out our Code of Conduct.











            MrL is a new contributor. Be nice, and check out our Code of Conduct.














            Thanks for contributing an answer to Data Science Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid


            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.

            Use MathJax to format equations. MathJax reference.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48883%2fwhat-could-this-mean-if-your-elbow-curve-looks-like-this%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            LjPmlimkn g4C hrPF1,F3,i7elMHORaeQQ8CphNkjJVT9Cfc V,yA,32dOk3Lm889shtP9LGhc97FS,CpV5,cETnOZLW,jCrg t3SJz
            JORmZREdvxaJTm,gxFGa9d,01ddFQ,I1 VoKiW2

            Popular posts from this blog

            Францішак Багушэвіч Змест Сям'я | Біяграфія | Творчасць | Мова Багушэвіча | Ацэнкі дзейнасці | Цікавыя факты | Спадчына | Выбраная бібліяграфія | Ушанаванне памяці | У філатэліі | Зноскі | Літаратура | Спасылкі | НавігацыяЛяхоўскі У. Рупіўся дзеля Бога і людзей: Жыццёвы шлях Лявона Вітан-Дубейкаўскага // Вольскі і Памідораў з песняй пра немца Адвакат, паэт, народны заступнік Ашмянскі веснікВ Минске появится площадь Богушевича и улица Сырокомли, Белорусская деловая газета, 19 июля 2001 г.Айцец беларускай нацыянальнай ідэі паўстаў у бронзе Сяргей Аляксандравіч Адашкевіч (1918, Мінск). 80-я гады. Бюст «Францішак Багушэвіч».Яўген Мікалаевіч Ціхановіч. «Партрэт Францішка Багушэвіча»Мікола Мікалаевіч Купава. «Партрэт зачынальніка новай беларускай літаратуры Францішка Багушэвіча»Уладзімір Іванавіч Мелехаў. На помніку «Змагарам за родную мову» Барэльеф «Францішак Багушэвіч»Памяць пра Багушэвіча на Віленшчыне Страчаная сталіца. Беларускія шыльды на вуліцах Вільні«Krynica». Ideologia i przywódcy białoruskiego katolicyzmuФранцішак БагушэвічТворы на knihi.comТворы Францішка Багушэвіча на bellib.byСодаль Уладзімір. Францішак Багушэвіч на Лідчыне;Луцкевіч Антон. Жыцьцё і творчасьць Фр. Багушэвіча ў успамінах ягоных сучасьнікаў // Запісы Беларускага Навуковага таварыства. Вільня, 1938. Сшытак 1. С. 16-34.Большая российская1188761710000 0000 5537 633Xn9209310021619551927869394п

            Partai Komunis Tiongkok Daftar isi Kepemimpinan | Pranala luar | Referensi | Menu navigasidiperiksa1 perubahan tertundacpc.people.com.cnSitus resmiSurat kabar resmi"Why the Communist Party is alive, well and flourishing in China"0307-1235"Full text of Constitution of Communist Party of China"smengembangkannyas

            На ростанях Змест Гісторыя напісання | Месца дзеяння | Час дзеяння | Назва | Праблематыка трылогіі | Аўтабіяграфічнасць | Трылогія ў тэатры і кіно | Пераклады | У культуры | Зноскі Літаратура | Спасылкі | НавігацыяДагледжаная версіяправерана1 зменаДагледжаная версіяправерана1 зменаАкадэмік МІЦКЕВІЧ Канстанцін Міхайлавіч (Якуб Колас) Прадмова М. І. Мушынскага, доктара філалагічных навук, члена-карэспандэнта Нацыянальнай акадэміі навук Рэспублікі Беларусь, прафесараНашаніўцы ў трылогіі Якуба Коласа «На ростанях»: вобразы і прататыпы125 лет Янке МавруКнижно-документальная выставка к 125-летию со дня рождения Якуба Коласа (1882—1956)Колас Якуб. Новая зямля (паэма), На ростанях (трылогія). Сулкоўскі Уладзімір. Радзіма Якуба Коласа (серыял жывапісных палотнаў)Вокладка кнігіІлюстрацыя М. С. БасалыгіНа ростаняхАўдыёверсія трылогііВ. Жолтак У Люсiнскай школе 1959