Hfrxdjt

I have categorical data and I'm trying to implement k-modes using the GitHub package available here. I am trying to create clusters in my (large) dataset of say, 5-7 records, each of most similar records.

However, as of now I have no means to select the optimal 'k' which would result in maximum silhouette score, ideally. This would be ideal as k-modes works on dissimilarity/similarity measure as a distance. So I would assume that silhouette distance would then measure how close/far the clusters are based on the distance metric defined by this dissimilarity and thus, establish the silhouette score. I'm not able to find an implementation of this.

Can I perhaps use the elbow method here? But then, I'm not able to understand how to programmatically determine this, without looking at a graph as I have to do this process repeatedly a large number of times. Currently, an idea is - find k where cost drops substantially. See if the next few values introduce a very less drop in cost or not. If yes, choose this as k, if no.. then what? I'm a little confused at this point.

I was looking online and also found this, which I'm not able to interpret in terms of k modes. I'm looking for any code/suggestions to start me off on the right path.

Instead of trying to find a place to download some source code, why don't you just implement, e.g., Silhouette yourself?

Plenty of the code you find online in blogs and repos is broken.

I've seen so many github repositories with bad code, and people like you wondering why it doesn't work. Relying on anonymous others to not have made mistakes is a bad idea. At some point you are better off writing the code yourself!

Of course it is okay to rely on large open-source projects like sklearn, R, ELKI, Weka. These have code-reviews, discuss pull requests, and dozens of people look at the code, use it, try to find and fix bugs (but even there are errors in the code).

From my understanding of silhouette score from the wikipedia page, here is an implementation:

def matching_similarity(a, b):
 return np.sum(a == b, axis=1)

distinct_cluster_label_predictions = [...]
silhouette_dict = dict()

for i in m_array:
 other_records_in_cluster = m_array_(with cluster_prediction == cluster_prediction of i)
 other_records_outside_cluster = m_array_(with cluster_prediction != cluster_prediction of i)
 sum_a = 0
 sum_b = 0
 sum_cluster_dist = dict()
 avg_cluster_dist = dict()

 for c in distinct_cluster_label_predictions:
 avg_cluster_dist[c] = 0

 # finding a(i) by taking avg. of intra-cluster distance
 for j in other_records_in_cluster:
 sum_a += matching_similarity(i, j)
 a = sum_a/len(other_records_in_cluster)

 dict_b = dict()
 # find average of inter-cluster distance with nearest neighbour
 for j in other_records_in_cluster:
 dist_i_to_j = matching_similarity(i,j)
 dict_b[j] = (cluster[j], dist_i_to_j)
 sum_till_now = avg_cluster_dist[cluster[j]]
 sum_cluster_dist[cluster[j]] = sum_till_now+dist_i_to_j

 for c in distinct_cluster_label_predictions:
 avg_cluster_dist[c] = sum_cluster_dist[c]/len(elements_belonging_to_c)

 # nearest_neighbour is the with smallest average distance
 nearest_cluster_label = key of minimum avg_cluster_dist value

 # for more than one nearest neighbour? Break randomly?

 neighbouring_cluster_records = list of records with cluster_prediction == nearest_cluster_label

 for k in neighbouring_cluster_records:
 sum_b += dict_b[j][1]

 b = sum_b/len(neighbouring_cluster_records)

 if (a<b):
 sil = 1 - (a/b)
 elif(a==b):
 sil = 0
 else:
 sil = b/a - 1

 silhouette_dict[i] = sil

average_silhouette_score = avg(all values in silhouette_dict)