Train an LSTM neural network with time series containing seasonal and trend Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) 2019 Moderator Election Q&A - Questionnaire 2019 Community Moderator Election ResultsHow to deal with time series which change in seasonality or other patterns?Time series prediction using ARIMA vs LSTMWhat is a better approach for cross-validation with time-related predictorsPrediction interval around LSTM time series forecastLSTM for time series forecasting with H20.aiKeras LSTM with 1D time seriesTime series binary classificaiton with labelling issuesLSTM Model for predicting the minutely seasonal data of the dayMultivariate time series forecasting with LSTMTrain LSTM model with multiple time series
Random body shuffle every night—can we still function?
How do living politicians protect their readily obtainable signatures from misuse?
Understanding p-Values using an example
Does the Mueller report show a conspiracy between Russia and the Trump Campaign?
How could we fake a moon landing now?
Does the Black Tentacles spell do damage twice at the start of turn to an already restrained creature?
Short story about a child who is a miniature, living Earth
Why does electrolysis of aqueous concentrated sodium bromide produce bromine at the anode?
How can I prevent/balance waiting and turtling as a response to cooldown mechanics
How to force a browser when connecting to a specific domain to be https only using only the client machine?
Is there more forest in the Northern Hemisphere now than 100 years ago?
Omitting the following parentheses
Nose gear failure in single prop aircraft: belly landing or nose-gear up landing?
What is the "studentd" process?
As a dual citizen, my US passport will expire one day after traveling to the US. Will this work?
Universal covering space of the real projective line?
Why weren't discrete x86 CPUs ever used in game hardware?
Tips to organize LaTeX presentations for a semester
Select every other edge (they share a common vertex)
I can't produce songs
Why is it faster to reheat something than it is to cook it?
Why is std::move not [[nodiscard]] in C++20?
Can two person see the same photon?
License to disallow distribution in closed source software, but allow exceptions made by owner?
Train an LSTM neural network with time series containing seasonal and trend
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern)
2019 Moderator Election Q&A - Questionnaire
2019 Community Moderator Election ResultsHow to deal with time series which change in seasonality or other patterns?Time series prediction using ARIMA vs LSTMWhat is a better approach for cross-validation with time-related predictorsPrediction interval around LSTM time series forecastLSTM for time series forecasting with H20.aiKeras LSTM with 1D time seriesTime series binary classificaiton with labelling issuesLSTM Model for predicting the minutely seasonal data of the dayMultivariate time series forecasting with LSTMTrain LSTM model with multiple time series
$begingroup$
I am working on a project for predicting the number of DNS queries from the site:
DNS queries statistics. The data I use is minutely data, which means the number of DNS queries of every minute.
If you look at the number of DNS queries from South Korea or any other countries, it has the seasonality and trend characteristics: increase then decrease day by day.
The requirement for me is: given the number of DNS queries of every minute, then for the given data of 20 previous minutes, predict the number of DNS queries in the next 20 minutes.
My problem is: my trained LSTM could not detect these factors, it always predicts my data to decrease everytime.
I have employed some hand-defined features like the minute of the day, period of the day like morning/afternoon... But it still keeps the same problem.
So I want to ask if there is any possible improvement to make my LSTM to understand when it should decrease and when it should increase with the seasonal data?
predictive-modeling time-series rnn lstm prediction
asked Aug 18 '18 at 7:23
Truong NguyenTruong Nguyen
152
$endgroup$
bumped to the homepage by Community♦ 33 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I am working on a project for predicting the number of DNS queries from the site:
DNS queries statistics. The data I use is minutely data, which means the number of DNS queries of every minute.
If you look at the number of DNS queries from South Korea or any other countries, it has the seasonality and trend characteristics: increase then decrease day by day.
The requirement for me is: given the number of DNS queries of every minute, then for the given data of 20 previous minutes, predict the number of DNS queries in the next 20 minutes.
My problem is: my trained LSTM could not detect these factors, it always predicts my data to decrease everytime.
I have employed some hand-defined features like the minute of the day, period of the day like morning/afternoon... But it still keeps the same problem.
So I want to ask if there is any possible improvement to make my LSTM to understand when it should decrease and when it should increase with the seasonal data?
predictive-modeling time-series rnn lstm prediction
asked Aug 18 '18 at 7:23
Truong NguyenTruong Nguyen
152
$endgroup$
bumped to the homepage by Community♦ 33 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
What about using Prophet?
$endgroup$
– Aditya
Aug 18 '18 at 9:53
$begingroup$
I think you might get more specific suggestions if you described how much data you have (number of days) and what platform you are using (Keras?, other?) I ask about number of days because I noticed that hte linked site that their plots were labeled as smoothed even with durations as short as a week.
$endgroup$
– 42-
Nov 20 '18 at 16:14
add a comment |
$begingroup$
I am working on a project for predicting the number of DNS queries from the site:
DNS queries statistics. The data I use is minutely data, which means the number of DNS queries of every minute.
If you look at the number of DNS queries from South Korea or any other countries, it has the seasonality and trend characteristics: increase then decrease day by day.
The requirement for me is: given the number of DNS queries of every minute, then for the given data of 20 previous minutes, predict the number of DNS queries in the next 20 minutes.
My problem is: my trained LSTM could not detect these factors, it always predicts my data to decrease everytime.
I have employed some hand-defined features like the minute of the day, period of the day like morning/afternoon... But it still keeps the same problem.
So I want to ask if there is any possible improvement to make my LSTM to understand when it should decrease and when it should increase with the seasonal data?
predictive-modeling time-series rnn lstm prediction
asked Aug 18 '18 at 7:23
Truong NguyenTruong Nguyen
152
$endgroup$
I am working on a project for predicting the number of DNS queries from the site:
DNS queries statistics. The data I use is minutely data, which means the number of DNS queries of every minute.
If you look at the number of DNS queries from South Korea or any other countries, it has the seasonality and trend characteristics: increase then decrease day by day.
The requirement for me is: given the number of DNS queries of every minute, then for the given data of 20 previous minutes, predict the number of DNS queries in the next 20 minutes.
My problem is: my trained LSTM could not detect these factors, it always predicts my data to decrease everytime.
I have employed some hand-defined features like the minute of the day, period of the day like morning/afternoon... But it still keeps the same problem.
So I want to ask if there is any possible improvement to make my LSTM to understand when it should decrease and when it should increase with the seasonal data?
predictive-modeling time-series rnn lstm prediction
predictive-modeling time-series rnn lstm prediction
asked Aug 18 '18 at 7:23
Truong NguyenTruong Nguyen
152
asked Aug 18 '18 at 7:23
Truong NguyenTruong Nguyen
152
asked Aug 18 '18 at 7:23
Truong NguyenTruong Nguyen
152
asked Aug 18 '18 at 7:23
Truong NguyenTruong Nguyen
152
asked Aug 18 '18 at 7:23
Truong NguyenTruong Nguyen
152
152
bumped to the homepage by Community♦ 33 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 33 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
What about using Prophet?
$endgroup$
– Aditya
Aug 18 '18 at 9:53
$begingroup$
I think you might get more specific suggestions if you described how much data you have (number of days) and what platform you are using (Keras?, other?) I ask about number of days because I noticed that hte linked site that their plots were labeled as smoothed even with durations as short as a week.
$endgroup$
– 42-
Nov 20 '18 at 16:14
add a comment |
$begingroup$
What about using Prophet?
$endgroup$
– Aditya
Aug 18 '18 at 9:53
$begingroup$
I think you might get more specific suggestions if you described how much data you have (number of days) and what platform you are using (Keras?, other?) I ask about number of days because I noticed that hte linked site that their plots were labeled as smoothed even with durations as short as a week.
$endgroup$
– 42-
Nov 20 '18 at 16:14
$begingroup$
What about using Prophet?
$endgroup$
– Aditya
Aug 18 '18 at 9:53
$begingroup$
What about using Prophet?
$endgroup$
– Aditya
Aug 18 '18 at 9:53
$begingroup$
I think you might get more specific suggestions if you described how much data you have (number of days) and what platform you are using (Keras?, other?) I ask about number of days because I noticed that hte linked site that their plots were labeled as smoothed even with durations as short as a week.
$endgroup$
– 42-
Nov 20 '18 at 16:14
$begingroup$
I think you might get more specific suggestions if you described how much data you have (number of days) and what platform you are using (Keras?, other?) I ask about number of days because I noticed that hte linked site that their plots were labeled as smoothed even with durations as short as a week.
$endgroup$
– 42-
Nov 20 '18 at 16:14
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
I don’t know if the seasonality will be achievable in your predictions, due simply to the timeframes you we using. If you see a daily up—down movement, but only provide 20 minutes for a prediction, how will the Model know whether or not it is at a turning point? You would perhaps need to include other features that contain that information - perhaps even the time stamp would suffice.
It is odd, that the model always predicts a downward movement — I would have expected it to simply continue on the current path (up or down), assuming you have both directoins in your training data...?
Perhaps you could look into some ideas used in common timeseries analysis methods, like separating the seasonality, trend and noise and feeding them separately to the model. Search for terms SARIMA, ARIMA, seasonality and cycles. (S)ARIMA stands for “seasonal autoregressive integrated moving average”, and represents a common way to look at data over time using previous values (autoregressivej, the differences of current value to previous values (integrated) and a moving average of past time steps (moving average).
The terms generated would of course catch the phases where the value changes direction and so be able to model seasonality fairly well.
answered Aug 18 '18 at 10:17
n1k31t4n1k31t4
6,6212421
$endgroup$
$begingroup$
Thank you for your reply, I have tried to define the turning point by my hand for training and testing the data, which i define the increasing period and decreasing period based on my observation. But the result seems to be still like before :( Our aim at this time is trying if LSTM is suitable for our job or not. Maybe I shouldtry other model as well :)
$endgroup$
– Truong Nguyen
Aug 20 '18 at 19:19
add a comment |
$begingroup$
Sound to me as a perfect candidate for a Holt-Winters model with a 24 hours seasonality. The trend should not be really sensitive on the time scale you use.
An other way to go would be to compute the seasonality by averaging a great number of 24-hours cycle. Then study the de-seasonlized data.
Once you understand what is happening, you have a chance to teach it to your LSTM.
answered Aug 21 '18 at 8:12
AlainDAlainD
2366
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f37113%2ftrain-an-lstm-neural-network-with-time-series-containing-seasonal-and-trend%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
I don’t know if the seasonality will be achievable in your predictions, due simply to the timeframes you we using. If you see a daily up—down movement, but only provide 20 minutes for a prediction, how will the Model know whether or not it is at a turning point? You would perhaps need to include other features that contain that information - perhaps even the time stamp would suffice.
It is odd, that the model always predicts a downward movement — I would have expected it to simply continue on the current path (up or down), assuming you have both directoins in your training data...?
Perhaps you could look into some ideas used in common timeseries analysis methods, like separating the seasonality, trend and noise and feeding them separately to the model. Search for terms SARIMA, ARIMA, seasonality and cycles. (S)ARIMA stands for “seasonal autoregressive integrated moving average”, and represents a common way to look at data over time using previous values (autoregressivej, the differences of current value to previous values (integrated) and a moving average of past time steps (moving average).
The terms generated would of course catch the phases where the value changes direction and so be able to model seasonality fairly well.
answered Aug 18 '18 at 10:17
n1k31t4n1k31t4
6,6212421
$endgroup$
$begingroup$
Thank you for your reply, I have tried to define the turning point by my hand for training and testing the data, which i define the increasing period and decreasing period based on my observation. But the result seems to be still like before :( Our aim at this time is trying if LSTM is suitable for our job or not. Maybe I shouldtry other model as well :)
$endgroup$
– Truong Nguyen
Aug 20 '18 at 19:19
add a comment |
$begingroup$
I don’t know if the seasonality will be achievable in your predictions, due simply to the timeframes you we using. If you see a daily up—down movement, but only provide 20 minutes for a prediction, how will the Model know whether or not it is at a turning point? You would perhaps need to include other features that contain that information - perhaps even the time stamp would suffice.
It is odd, that the model always predicts a downward movement — I would have expected it to simply continue on the current path (up or down), assuming you have both directoins in your training data...?
Perhaps you could look into some ideas used in common timeseries analysis methods, like separating the seasonality, trend and noise and feeding them separately to the model. Search for terms SARIMA, ARIMA, seasonality and cycles. (S)ARIMA stands for “seasonal autoregressive integrated moving average”, and represents a common way to look at data over time using previous values (autoregressivej, the differences of current value to previous values (integrated) and a moving average of past time steps (moving average).
The terms generated would of course catch the phases where the value changes direction and so be able to model seasonality fairly well.
answered Aug 18 '18 at 10:17
n1k31t4n1k31t4
6,6212421
$endgroup$
$begingroup$
Thank you for your reply, I have tried to define the turning point by my hand for training and testing the data, which i define the increasing period and decreasing period based on my observation. But the result seems to be still like before :( Our aim at this time is trying if LSTM is suitable for our job or not. Maybe I shouldtry other model as well :)
$endgroup$
– Truong Nguyen
Aug 20 '18 at 19:19
add a comment |
$begingroup$
I don’t know if the seasonality will be achievable in your predictions, due simply to the timeframes you we using. If you see a daily up—down movement, but only provide 20 minutes for a prediction, how will the Model know whether or not it is at a turning point? You would perhaps need to include other features that contain that information - perhaps even the time stamp would suffice.
It is odd, that the model always predicts a downward movement — I would have expected it to simply continue on the current path (up or down), assuming you have both directoins in your training data...?
Perhaps you could look into some ideas used in common timeseries analysis methods, like separating the seasonality, trend and noise and feeding them separately to the model. Search for terms SARIMA, ARIMA, seasonality and cycles. (S)ARIMA stands for “seasonal autoregressive integrated moving average”, and represents a common way to look at data over time using previous values (autoregressivej, the differences of current value to previous values (integrated) and a moving average of past time steps (moving average).
The terms generated would of course catch the phases where the value changes direction and so be able to model seasonality fairly well.
answered Aug 18 '18 at 10:17
n1k31t4n1k31t4
6,6212421
$endgroup$
I don’t know if the seasonality will be achievable in your predictions, due simply to the timeframes you we using. If you see a daily up—down movement, but only provide 20 minutes for a prediction, how will the Model know whether or not it is at a turning point? You would perhaps need to include other features that contain that information - perhaps even the time stamp would suffice.
It is odd, that the model always predicts a downward movement — I would have expected it to simply continue on the current path (up or down), assuming you have both directoins in your training data...?
Perhaps you could look into some ideas used in common timeseries analysis methods, like separating the seasonality, trend and noise and feeding them separately to the model. Search for terms SARIMA, ARIMA, seasonality and cycles. (S)ARIMA stands for “seasonal autoregressive integrated moving average”, and represents a common way to look at data over time using previous values (autoregressivej, the differences of current value to previous values (integrated) and a moving average of past time steps (moving average).
The terms generated would of course catch the phases where the value changes direction and so be able to model seasonality fairly well.
answered Aug 18 '18 at 10:17
n1k31t4n1k31t4
6,6212421
answered Aug 18 '18 at 10:17
n1k31t4n1k31t4
6,6212421
answered Aug 18 '18 at 10:17
n1k31t4n1k31t4
6,6212421
answered Aug 18 '18 at 10:17
n1k31t4n1k31t4
6,6212421
6,6212421
$begingroup$
Thank you for your reply, I have tried to define the turning point by my hand for training and testing the data, which i define the increasing period and decreasing period based on my observation. But the result seems to be still like before :( Our aim at this time is trying if LSTM is suitable for our job or not. Maybe I shouldtry other model as well :)
$endgroup$
– Truong Nguyen
Aug 20 '18 at 19:19
add a comment |
$begingroup$
Thank you for your reply, I have tried to define the turning point by my hand for training and testing the data, which i define the increasing period and decreasing period based on my observation. But the result seems to be still like before :( Our aim at this time is trying if LSTM is suitable for our job or not. Maybe I shouldtry other model as well :)
$endgroup$
– Truong Nguyen
Aug 20 '18 at 19:19
$begingroup$
Thank you for your reply, I have tried to define the turning point by my hand for training and testing the data, which i define the increasing period and decreasing period based on my observation. But the result seems to be still like before :( Our aim at this time is trying if LSTM is suitable for our job or not. Maybe I shouldtry other model as well :)
$endgroup$
– Truong Nguyen
Aug 20 '18 at 19:19
$begingroup$
Thank you for your reply, I have tried to define the turning point by my hand for training and testing the data, which i define the increasing period and decreasing period based on my observation. But the result seems to be still like before :( Our aim at this time is trying if LSTM is suitable for our job or not. Maybe I shouldtry other model as well :)
$endgroup$
– Truong Nguyen
Aug 20 '18 at 19:19
add a comment |
$begingroup$
Sound to me as a perfect candidate for a Holt-Winters model with a 24 hours seasonality. The trend should not be really sensitive on the time scale you use.
An other way to go would be to compute the seasonality by averaging a great number of 24-hours cycle. Then study the de-seasonlized data.
Once you understand what is happening, you have a chance to teach it to your LSTM.
answered Aug 21 '18 at 8:12
AlainDAlainD
2366
$endgroup$
add a comment |
$begingroup$
Sound to me as a perfect candidate for a Holt-Winters model with a 24 hours seasonality. The trend should not be really sensitive on the time scale you use.
An other way to go would be to compute the seasonality by averaging a great number of 24-hours cycle. Then study the de-seasonlized data.
Once you understand what is happening, you have a chance to teach it to your LSTM.
answered Aug 21 '18 at 8:12
AlainDAlainD
2366
$endgroup$
add a comment |
$begingroup$
Sound to me as a perfect candidate for a Holt-Winters model with a 24 hours seasonality. The trend should not be really sensitive on the time scale you use.
An other way to go would be to compute the seasonality by averaging a great number of 24-hours cycle. Then study the de-seasonlized data.
Once you understand what is happening, you have a chance to teach it to your LSTM.
answered Aug 21 '18 at 8:12
AlainDAlainD
2366
$endgroup$
Sound to me as a perfect candidate for a Holt-Winters model with a 24 hours seasonality. The trend should not be really sensitive on the time scale you use.
An other way to go would be to compute the seasonality by averaging a great number of 24-hours cycle. Then study the de-seasonlized data.
Once you understand what is happening, you have a chance to teach it to your LSTM.
answered Aug 21 '18 at 8:12
AlainDAlainD
2366
answered Aug 21 '18 at 8:12
AlainDAlainD
2366
answered Aug 21 '18 at 8:12
AlainDAlainD
2366
answered Aug 21 '18 at 8:12
AlainDAlainD
2366
2366
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f37113%2ftrain-an-lstm-neural-network-with-time-series-containing-seasonal-and-trend%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
What about using Prophet?
$endgroup$
– Aditya
Aug 18 '18 at 9:53
$begingroup$
I think you might get more specific suggestions if you described how much data you have (number of days) and what platform you are using (Keras?, other?) I ask about number of days because I noticed that hte linked site that their plots were labeled as smoothed even with durations as short as a week.
$endgroup$
– 42-
Nov 20 '18 at 16:14