Clustering time series based on monotonic similarityR: Comparing dissimilarity between metabolic models with discrete wavelet transformationClassify Customers based on 2 features AND a Time series of eventsClustering users based on buying behaviourHow to cluster multiple time-series from one data frameAgglomerative Hierarchial Clustering in python using DTW distancehow to compare different sets of time series dataDifference between Time series clustering and Time series SegmentationDoes hierarchical agglomerative clustering with centroid-linkage suffer from chain-effect?Code or Package to cluster sequences (or time series) of different lengths based on HMM?Any cluster algo can cluster time series datasets based on variation ratio(or quantity)?
Why does the Persian emissary display a string of crowned skulls?
Proving an identity involving cross products and coplanar vectors
How do I prevent inappropriate ads from appearing in my game?
Storage of electrolytic capacitors - how long?
If Captain Marvel (MCU) were to have a child with a human male, would the child be human or Kree?
What's the name of the logical fallacy where a debater extends a statement far beyond the original statement to make it true?
Sigmoid with a slope but no asymptotes?
How to make money from a browser who sees 5 seconds into the future of any web page?
What happens if I try to grapple mirror image?
Why can't the Brexit deadlock in the UK parliament be solved with a plurality vote?
Quoting Keynes in a lecture
Determining multivariate least squares with constraint
How do I fix the group tension caused by my character stealing and possibly killing without provocation?
Ways of geometrical multiplication
How do I tell my boss that I'm quitting in 15 days (a colleague left this week)
Review your own paper in Mathematics
How to leave product feedback on macOS?
Animation: customize bounce interpolation
Why the "ls" command is showing the permissions of files in a FAT32 partition?
Possible Eco thriller, man invents a device to remove rain from glass
Should I warn a new PhD Student?
How to make a list of partial sums using forEach
SOQL query causes internal Salesforce error
What is the meaning of "You've never met a graph you didn't like?"
Clustering time series based on monotonic similarity
R: Comparing dissimilarity between metabolic models with discrete wavelet transformationClassify Customers based on 2 features AND a Time series of eventsClustering users based on buying behaviourHow to cluster multiple time-series from one data frameAgglomerative Hierarchial Clustering in python using DTW distancehow to compare different sets of time series dataDifference between Time series clustering and Time series SegmentationDoes hierarchical agglomerative clustering with centroid-linkage suffer from chain-effect?Code or Package to cluster sequences (or time series) of different lengths based on HMM?Any cluster algo can cluster time series datasets based on variation ratio(or quantity)?
$begingroup$
Context
I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, but responding to the same exogenous variables. However, for each time series, the magnitude of the response is very different. For a time series of reference $X$, I would like to be grouped in the same cluster series that are alike $X^a$ for all $a > 0$.
Tryouts
So far, my interpretation of the problem is that I want to cluster time series sharing a strong monotonic relationship. My first tryouts used hierarchical agglomerative clustering by defining a distance based Kendall's tau rank coefficient since it measures the strength of a monotonic relationship. By visual interpretation, best results were obtained using Ward's linkage method. However, this approach seems unorthodox, non-robust, or doubtful for several reasons.
First, Scipy documentation mentions here that Ward's method is only correct when the Euclidean distance is used. Secondly, I couldn't find any detailed application of time series clustering based either on Spearman or Kendall's tau coefficient. Furthermore, I was very surprised that I couldn't find any paper or reference aiming at clustering based on a monotonic criterion.
I am willing to consider other approaches, though I cannot measure their benefits. For instance, rescaling all time-series to map them to a standardized gaussian distribution (e.g. Box-Cox) and then using the Euclidean distance. Another possibility is to turn the first difference of time series into a boolean vector (1 if $Delta X >0$, $0$ otherwise) and then use the Euclidean distance or another distance metric.
Questions
Since I am new to time series clustering, I have some troubles to picture by myself what would be the best approach(es) (or the worse) for this specific purpose. Hence, I have two related questions:
- Specifically, is using Hierarchical Clustering based on Kendall's tau and Ward's linkage method a wrong way to go and why?
- Generally, what is the best way to cluster time series based on monotonic association?
Some references on the topic are also welcome.
time-series clustering preprocessing distance
asked Feb 13 at 14:33
DelforgeDelforge
1114
$endgroup$
bumped to the homepage by Community♦ 31 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
Context
I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, but responding to the same exogenous variables. However, for each time series, the magnitude of the response is very different. For a time series of reference $X$, I would like to be grouped in the same cluster series that are alike $X^a$ for all $a > 0$.
Tryouts
So far, my interpretation of the problem is that I want to cluster time series sharing a strong monotonic relationship. My first tryouts used hierarchical agglomerative clustering by defining a distance based Kendall's tau rank coefficient since it measures the strength of a monotonic relationship. By visual interpretation, best results were obtained using Ward's linkage method. However, this approach seems unorthodox, non-robust, or doubtful for several reasons.
First, Scipy documentation mentions here that Ward's method is only correct when the Euclidean distance is used. Secondly, I couldn't find any detailed application of time series clustering based either on Spearman or Kendall's tau coefficient. Furthermore, I was very surprised that I couldn't find any paper or reference aiming at clustering based on a monotonic criterion.
I am willing to consider other approaches, though I cannot measure their benefits. For instance, rescaling all time-series to map them to a standardized gaussian distribution (e.g. Box-Cox) and then using the Euclidean distance. Another possibility is to turn the first difference of time series into a boolean vector (1 if $Delta X >0$, $0$ otherwise) and then use the Euclidean distance or another distance metric.
Questions
Since I am new to time series clustering, I have some troubles to picture by myself what would be the best approach(es) (or the worse) for this specific purpose. Hence, I have two related questions:
- Specifically, is using Hierarchical Clustering based on Kendall's tau and Ward's linkage method a wrong way to go and why?
- Generally, what is the best way to cluster time series based on monotonic association?
Some references on the topic are also welcome.
time-series clustering preprocessing distance
asked Feb 13 at 14:33
DelforgeDelforge
1114
$endgroup$
bumped to the homepage by Community♦ 31 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
Context
I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, but responding to the same exogenous variables. However, for each time series, the magnitude of the response is very different. For a time series of reference $X$, I would like to be grouped in the same cluster series that are alike $X^a$ for all $a > 0$.
Tryouts
So far, my interpretation of the problem is that I want to cluster time series sharing a strong monotonic relationship. My first tryouts used hierarchical agglomerative clustering by defining a distance based Kendall's tau rank coefficient since it measures the strength of a monotonic relationship. By visual interpretation, best results were obtained using Ward's linkage method. However, this approach seems unorthodox, non-robust, or doubtful for several reasons.
First, Scipy documentation mentions here that Ward's method is only correct when the Euclidean distance is used. Secondly, I couldn't find any detailed application of time series clustering based either on Spearman or Kendall's tau coefficient. Furthermore, I was very surprised that I couldn't find any paper or reference aiming at clustering based on a monotonic criterion.
I am willing to consider other approaches, though I cannot measure their benefits. For instance, rescaling all time-series to map them to a standardized gaussian distribution (e.g. Box-Cox) and then using the Euclidean distance. Another possibility is to turn the first difference of time series into a boolean vector (1 if $Delta X >0$, $0$ otherwise) and then use the Euclidean distance or another distance metric.
Questions
Since I am new to time series clustering, I have some troubles to picture by myself what would be the best approach(es) (or the worse) for this specific purpose. Hence, I have two related questions:
- Specifically, is using Hierarchical Clustering based on Kendall's tau and Ward's linkage method a wrong way to go and why?
- Generally, what is the best way to cluster time series based on monotonic association?
Some references on the topic are also welcome.
time-series clustering preprocessing distance
asked Feb 13 at 14:33
DelforgeDelforge
1114
$endgroup$
Context
I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, but responding to the same exogenous variables. However, for each time series, the magnitude of the response is very different. For a time series of reference $X$, I would like to be grouped in the same cluster series that are alike $X^a$ for all $a > 0$.
Tryouts
So far, my interpretation of the problem is that I want to cluster time series sharing a strong monotonic relationship. My first tryouts used hierarchical agglomerative clustering by defining a distance based Kendall's tau rank coefficient since it measures the strength of a monotonic relationship. By visual interpretation, best results were obtained using Ward's linkage method. However, this approach seems unorthodox, non-robust, or doubtful for several reasons.
First, Scipy documentation mentions here that Ward's method is only correct when the Euclidean distance is used. Secondly, I couldn't find any detailed application of time series clustering based either on Spearman or Kendall's tau coefficient. Furthermore, I was very surprised that I couldn't find any paper or reference aiming at clustering based on a monotonic criterion.
I am willing to consider other approaches, though I cannot measure their benefits. For instance, rescaling all time-series to map them to a standardized gaussian distribution (e.g. Box-Cox) and then using the Euclidean distance. Another possibility is to turn the first difference of time series into a boolean vector (1 if $Delta X >0$, $0$ otherwise) and then use the Euclidean distance or another distance metric.
Questions
Since I am new to time series clustering, I have some troubles to picture by myself what would be the best approach(es) (or the worse) for this specific purpose. Hence, I have two related questions:
- Specifically, is using Hierarchical Clustering based on Kendall's tau and Ward's linkage method a wrong way to go and why?
- Generally, what is the best way to cluster time series based on monotonic association?
Some references on the topic are also welcome.
time-series clustering preprocessing distance
time-series clustering preprocessing distance
asked Feb 13 at 14:33
DelforgeDelforge
1114
asked Feb 13 at 14:33
DelforgeDelforge
1114
asked Feb 13 at 14:33
DelforgeDelforge
1114
asked Feb 13 at 14:33
DelforgeDelforge
1114
asked Feb 13 at 14:33
DelforgeDelforge
1114
1114
bumped to the homepage by Community♦ 31 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 31 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
The way Ward's linkage is computed really only makes sense with squared Euclidean type of measures. Only then the Konig-Huygens theorem can be used.
Why don't you consider average linkage? Why Ward?
answered Feb 18 at 20:07
Anony-MousseAnony-Mousse
5,010624
$endgroup$
$begingroup$
As it is a spatiotemporal dataset, I am able to check how clusters are spatially distributed. Kendall + Average linkage yields to one dominant cluster while the others are small and spatially split. Ward + Kendall yield to more consistent clusters with respect to the spatial structure of the system. However, I am able to get a similar spatial distribution using Pearson's correlation on Z standardized data with Ward linkage. This is even better if I apply a quantile mapping to a normal distribution on my time series. I guess it might be more be "mathematical" than relying on Kendall's tau.
$endgroup$
– Delforge
Feb 19 at 10:26
$begingroup$
Pearson does its own normalization... And it's easy to prove that it is related to Euclidean on appropriately preprocessed data. So if you do this math, you may be able to set it up so that you can use Euclidean (and hence Ward) to get Pearson results. You just need to show that you can pull out the normalization, and that it won't yield a division by 0 on your data.
$endgroup$
– Anony-Mousse
Feb 19 at 11:50
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f45510%2fclustering-time-series-based-on-monotonic-similarity%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The way Ward's linkage is computed really only makes sense with squared Euclidean type of measures. Only then the Konig-Huygens theorem can be used.
Why don't you consider average linkage? Why Ward?
answered Feb 18 at 20:07
Anony-MousseAnony-Mousse
5,010624
$endgroup$
$begingroup$
As it is a spatiotemporal dataset, I am able to check how clusters are spatially distributed. Kendall + Average linkage yields to one dominant cluster while the others are small and spatially split. Ward + Kendall yield to more consistent clusters with respect to the spatial structure of the system. However, I am able to get a similar spatial distribution using Pearson's correlation on Z standardized data with Ward linkage. This is even better if I apply a quantile mapping to a normal distribution on my time series. I guess it might be more be "mathematical" than relying on Kendall's tau.
$endgroup$
– Delforge
Feb 19 at 10:26
$begingroup$
Pearson does its own normalization... And it's easy to prove that it is related to Euclidean on appropriately preprocessed data. So if you do this math, you may be able to set it up so that you can use Euclidean (and hence Ward) to get Pearson results. You just need to show that you can pull out the normalization, and that it won't yield a division by 0 on your data.
$endgroup$
– Anony-Mousse
Feb 19 at 11:50
add a comment |
$begingroup$
The way Ward's linkage is computed really only makes sense with squared Euclidean type of measures. Only then the Konig-Huygens theorem can be used.
Why don't you consider average linkage? Why Ward?
answered Feb 18 at 20:07
Anony-MousseAnony-Mousse
5,010624
$endgroup$
$begingroup$
As it is a spatiotemporal dataset, I am able to check how clusters are spatially distributed. Kendall + Average linkage yields to one dominant cluster while the others are small and spatially split. Ward + Kendall yield to more consistent clusters with respect to the spatial structure of the system. However, I am able to get a similar spatial distribution using Pearson's correlation on Z standardized data with Ward linkage. This is even better if I apply a quantile mapping to a normal distribution on my time series. I guess it might be more be "mathematical" than relying on Kendall's tau.
$endgroup$
– Delforge
Feb 19 at 10:26
$begingroup$
Pearson does its own normalization... And it's easy to prove that it is related to Euclidean on appropriately preprocessed data. So if you do this math, you may be able to set it up so that you can use Euclidean (and hence Ward) to get Pearson results. You just need to show that you can pull out the normalization, and that it won't yield a division by 0 on your data.
$endgroup$
– Anony-Mousse
Feb 19 at 11:50
add a comment |
$begingroup$
The way Ward's linkage is computed really only makes sense with squared Euclidean type of measures. Only then the Konig-Huygens theorem can be used.
Why don't you consider average linkage? Why Ward?
answered Feb 18 at 20:07
Anony-MousseAnony-Mousse
5,010624
$endgroup$
The way Ward's linkage is computed really only makes sense with squared Euclidean type of measures. Only then the Konig-Huygens theorem can be used.
Why don't you consider average linkage? Why Ward?
answered Feb 18 at 20:07
Anony-MousseAnony-Mousse
5,010624
answered Feb 18 at 20:07
Anony-MousseAnony-Mousse
5,010624
answered Feb 18 at 20:07
Anony-MousseAnony-Mousse
5,010624
answered Feb 18 at 20:07
Anony-MousseAnony-Mousse
5,010624
5,010624
$begingroup$
As it is a spatiotemporal dataset, I am able to check how clusters are spatially distributed. Kendall + Average linkage yields to one dominant cluster while the others are small and spatially split. Ward + Kendall yield to more consistent clusters with respect to the spatial structure of the system. However, I am able to get a similar spatial distribution using Pearson's correlation on Z standardized data with Ward linkage. This is even better if I apply a quantile mapping to a normal distribution on my time series. I guess it might be more be "mathematical" than relying on Kendall's tau.
$endgroup$
– Delforge
Feb 19 at 10:26
$begingroup$
Pearson does its own normalization... And it's easy to prove that it is related to Euclidean on appropriately preprocessed data. So if you do this math, you may be able to set it up so that you can use Euclidean (and hence Ward) to get Pearson results. You just need to show that you can pull out the normalization, and that it won't yield a division by 0 on your data.
$endgroup$
– Anony-Mousse
Feb 19 at 11:50
add a comment |
$begingroup$
As it is a spatiotemporal dataset, I am able to check how clusters are spatially distributed. Kendall + Average linkage yields to one dominant cluster while the others are small and spatially split. Ward + Kendall yield to more consistent clusters with respect to the spatial structure of the system. However, I am able to get a similar spatial distribution using Pearson's correlation on Z standardized data with Ward linkage. This is even better if I apply a quantile mapping to a normal distribution on my time series. I guess it might be more be "mathematical" than relying on Kendall's tau.
$endgroup$
– Delforge
Feb 19 at 10:26
$begingroup$
Pearson does its own normalization... And it's easy to prove that it is related to Euclidean on appropriately preprocessed data. So if you do this math, you may be able to set it up so that you can use Euclidean (and hence Ward) to get Pearson results. You just need to show that you can pull out the normalization, and that it won't yield a division by 0 on your data.
$endgroup$
– Anony-Mousse
Feb 19 at 11:50
$begingroup$
As it is a spatiotemporal dataset, I am able to check how clusters are spatially distributed. Kendall + Average linkage yields to one dominant cluster while the others are small and spatially split. Ward + Kendall yield to more consistent clusters with respect to the spatial structure of the system. However, I am able to get a similar spatial distribution using Pearson's correlation on Z standardized data with Ward linkage. This is even better if I apply a quantile mapping to a normal distribution on my time series. I guess it might be more be "mathematical" than relying on Kendall's tau.
$endgroup$
– Delforge
Feb 19 at 10:26
$begingroup$
As it is a spatiotemporal dataset, I am able to check how clusters are spatially distributed. Kendall + Average linkage yields to one dominant cluster while the others are small and spatially split. Ward + Kendall yield to more consistent clusters with respect to the spatial structure of the system. However, I am able to get a similar spatial distribution using Pearson's correlation on Z standardized data with Ward linkage. This is even better if I apply a quantile mapping to a normal distribution on my time series. I guess it might be more be "mathematical" than relying on Kendall's tau.
$endgroup$
– Delforge
Feb 19 at 10:26
$begingroup$
Pearson does its own normalization... And it's easy to prove that it is related to Euclidean on appropriately preprocessed data. So if you do this math, you may be able to set it up so that you can use Euclidean (and hence Ward) to get Pearson results. You just need to show that you can pull out the normalization, and that it won't yield a division by 0 on your data.
$endgroup$
– Anony-Mousse
Feb 19 at 11:50
$begingroup$
Pearson does its own normalization... And it's easy to prove that it is related to Euclidean on appropriately preprocessed data. So if you do this math, you may be able to set it up so that you can use Euclidean (and hence Ward) to get Pearson results. You just need to show that you can pull out the normalization, and that it won't yield a division by 0 on your data.
$endgroup$
– Anony-Mousse
Feb 19 at 11:50
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f45510%2fclustering-time-series-based-on-monotonic-similarity%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
