Using SMAPE as a loss function for an LSTMLoss function for sparse taggingRNN for classification giving vastly different results (Keras)Classifier that optimizes performance on only a subset of the data?Understanding LSTM behaviour: Validation loss smaller than training loss throughout training for regression problemExpected behaviour of loss and accuracy when using data augmentationLSTM - divide gradients by number of timesteps IMMEDIATELY or in the end?Using deep learning to classify similar imagesTuning a sequence to sequence modelIs there any standard or normal range for the amount of LSTM loss function?
Identifying "long and narrow" polygons in with Postgis
Is there a distance limit for minecart tracks?
Extracting patterns from a text
Purpose of creating non root user
Proving a complicated language is not a CFL
Why is participating in the European Parliamentary elections used as a threat?
I'm just a whisper. Who am I?
How can I, as DM, avoid the Conga Line of Death occurring when implementing some form of flanking rule?
Can I run 125khz RF circuit on a breadboard?
Can anyone precisely describe what it means (or feels like) to play exactly what your "inner ear" is hearing?
What happens if a creature's ETB would bounce Thalia, Heretic Cathar?
Asserting that Atheism and Theism are both faith based positions
Why does a 97 / 92 key piano exist by Bösendorfer?
Air travel with refrigerated insulin
Storage of electrolytic capacitors - how long?
El Dorado Word Puzzle II: Videogame Edition
Overlapping circles covering polygon
The Digit Triangles
Why the "ls" command is showing the permissions of files in a FAT32 partition?
Review your own paper in Mathematics
Why does the Persian emissary display a string of crowned skulls?
Should a narrator ever describe things based on a character's view instead of facts?
Limit max CPU usage SQL SERVER with WSRM
How to write Quadratic equation with negative coefficient
Using SMAPE as a loss function for an LSTM
Loss function for sparse taggingRNN for classification giving vastly different results (Keras)Classifier that optimizes performance on only a subset of the data?Understanding LSTM behaviour: Validation loss smaller than training loss throughout training for regression problemExpected behaviour of loss and accuracy when using data augmentationLSTM - divide gradients by number of timesteps IMMEDIATELY or in the end?Using deep learning to classify similar imagesTuning a sequence to sequence modelIs there any standard or normal range for the amount of LSTM loss function?
$begingroup$
I am currently working on a time series forecasting problem and am looking into using an LSTM.
My final accuracy metric that I use to determine whether or not the forecast is good or not is defined as follows:
$$textSMAPE-3 = fracsum^n_t=1sum^n_t=1left(y_t+haty_tright),$$
where $y_t$ is the actual value and $haty_t$ is the forecast value. So this formula will produce a score between $[0, 1]$. This score can then be subtracted from 1 to produce a percentage accuracy of the forecast over the specified forecast horizon
$$textAccuracy = 1-fracsum^n_t=1sum^n_t=1left(y_t+haty_tright).$$
I am using keras and have seen that I can specify a custom loss function and metric. As my forecast accuracy will be measured using sMAPE 3 (as defined above) it made sense to me to use this as my loss function. This is because my networks success will be measured by how well it scores using this. A score of 0 indicates 100% accuracy and a score of 1 indicates 0% accuracy.
So this raises two questions, can a metric such as smape be used as loss function at all, if so why / why not? Are the standard loss functions within keras considered better? Perhaps because they allow quicker convergence? Does my choice of using SMAPE3 as a loss function have an impact on what kind of optimiser I should use?
I ask because I haven't seen many people using loss functions other than RMSE, MSE and MAE and there must be a reason for this.
keras time-series lstm rnn loss-function
asked Nov 12 '18 at 11:38
AesirAesir
1498
$endgroup$
add a comment |
$begingroup$
I am currently working on a time series forecasting problem and am looking into using an LSTM.
My final accuracy metric that I use to determine whether or not the forecast is good or not is defined as follows:
$$textSMAPE-3 = fracsum^n_t=1sum^n_t=1left(y_t+haty_tright),$$
where $y_t$ is the actual value and $haty_t$ is the forecast value. So this formula will produce a score between $[0, 1]$. This score can then be subtracted from 1 to produce a percentage accuracy of the forecast over the specified forecast horizon
$$textAccuracy = 1-fracsum^n_t=1sum^n_t=1left(y_t+haty_tright).$$
I am using keras and have seen that I can specify a custom loss function and metric. As my forecast accuracy will be measured using sMAPE 3 (as defined above) it made sense to me to use this as my loss function. This is because my networks success will be measured by how well it scores using this. A score of 0 indicates 100% accuracy and a score of 1 indicates 0% accuracy.
So this raises two questions, can a metric such as smape be used as loss function at all, if so why / why not? Are the standard loss functions within keras considered better? Perhaps because they allow quicker convergence? Does my choice of using SMAPE3 as a loss function have an impact on what kind of optimiser I should use?
I ask because I haven't seen many people using loss functions other than RMSE, MSE and MAE and there must be a reason for this.
keras time-series lstm rnn loss-function
asked Nov 12 '18 at 11:38
AesirAesir
1498
$endgroup$
add a comment |
$begingroup$
I am currently working on a time series forecasting problem and am looking into using an LSTM.
My final accuracy metric that I use to determine whether or not the forecast is good or not is defined as follows:
$$textSMAPE-3 = fracsum^n_t=1sum^n_t=1left(y_t+haty_tright),$$
where $y_t$ is the actual value and $haty_t$ is the forecast value. So this formula will produce a score between $[0, 1]$. This score can then be subtracted from 1 to produce a percentage accuracy of the forecast over the specified forecast horizon
$$textAccuracy = 1-fracsum^n_t=1sum^n_t=1left(y_t+haty_tright).$$
I am using keras and have seen that I can specify a custom loss function and metric. As my forecast accuracy will be measured using sMAPE 3 (as defined above) it made sense to me to use this as my loss function. This is because my networks success will be measured by how well it scores using this. A score of 0 indicates 100% accuracy and a score of 1 indicates 0% accuracy.
So this raises two questions, can a metric such as smape be used as loss function at all, if so why / why not? Are the standard loss functions within keras considered better? Perhaps because they allow quicker convergence? Does my choice of using SMAPE3 as a loss function have an impact on what kind of optimiser I should use?
I ask because I haven't seen many people using loss functions other than RMSE, MSE and MAE and there must be a reason for this.
keras time-series lstm rnn loss-function
asked Nov 12 '18 at 11:38
AesirAesir
1498
$endgroup$
I am currently working on a time series forecasting problem and am looking into using an LSTM.
My final accuracy metric that I use to determine whether or not the forecast is good or not is defined as follows:
$$textSMAPE-3 = fracsum^n_t=1sum^n_t=1left(y_t+haty_tright),$$
where $y_t$ is the actual value and $haty_t$ is the forecast value. So this formula will produce a score between $[0, 1]$. This score can then be subtracted from 1 to produce a percentage accuracy of the forecast over the specified forecast horizon
$$textAccuracy = 1-fracsum^n_t=1sum^n_t=1left(y_t+haty_tright).$$
I am using keras and have seen that I can specify a custom loss function and metric. As my forecast accuracy will be measured using sMAPE 3 (as defined above) it made sense to me to use this as my loss function. This is because my networks success will be measured by how well it scores using this. A score of 0 indicates 100% accuracy and a score of 1 indicates 0% accuracy.
So this raises two questions, can a metric such as smape be used as loss function at all, if so why / why not? Are the standard loss functions within keras considered better? Perhaps because they allow quicker convergence? Does my choice of using SMAPE3 as a loss function have an impact on what kind of optimiser I should use?
I ask because I haven't seen many people using loss functions other than RMSE, MSE and MAE and there must be a reason for this.
keras time-series lstm rnn loss-function
keras time-series lstm rnn loss-function
asked Nov 12 '18 at 11:38
AesirAesir
1498
asked Nov 12 '18 at 11:38
AesirAesir
1498
edited Nov 12 '18 at 21:33
Aesir
asked Nov 12 '18 at 11:38
AesirAesir
1498
asked Nov 12 '18 at 11:38
AesirAesir
1498
asked Nov 12 '18 at 11:38
AesirAesir
1498
1498
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
This is a great question. I too have been wondering about using sMAPE.
Was reading a paper on "Modeling approaches for time series forecasting and anomaly detection" (S Du, 2017) . and it mentions using sMAPE as "This metric is more robust towards outliers and it has a unified scale across different time series with different scale." (ibid., 4)
But most docs I see are using the standard RMSE, MSE, and MAE metrics.
Would love to have this answered.
answered 1 hour ago
JohnnyDenimJohnnyDenim
1
New contributor
JohnnyDenim is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f41093%2fusing-smape-as-a-loss-function-for-an-lstm%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
This is a great question. I too have been wondering about using sMAPE.
Was reading a paper on "Modeling approaches for time series forecasting and anomaly detection" (S Du, 2017) . and it mentions using sMAPE as "This metric is more robust towards outliers and it has a unified scale across different time series with different scale." (ibid., 4)
But most docs I see are using the standard RMSE, MSE, and MAE metrics.
Would love to have this answered.
answered 1 hour ago
JohnnyDenimJohnnyDenim
1
New contributor
JohnnyDenim is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
This is a great question. I too have been wondering about using sMAPE.
Was reading a paper on "Modeling approaches for time series forecasting and anomaly detection" (S Du, 2017) . and it mentions using sMAPE as "This metric is more robust towards outliers and it has a unified scale across different time series with different scale." (ibid., 4)
But most docs I see are using the standard RMSE, MSE, and MAE metrics.
Would love to have this answered.
answered 1 hour ago
JohnnyDenimJohnnyDenim
1
New contributor
JohnnyDenim is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
This is a great question. I too have been wondering about using sMAPE.
Was reading a paper on "Modeling approaches for time series forecasting and anomaly detection" (S Du, 2017) . and it mentions using sMAPE as "This metric is more robust towards outliers and it has a unified scale across different time series with different scale." (ibid., 4)
But most docs I see are using the standard RMSE, MSE, and MAE metrics.
Would love to have this answered.
answered 1 hour ago
JohnnyDenimJohnnyDenim
1
New contributor
JohnnyDenim is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
This is a great question. I too have been wondering about using sMAPE.
Was reading a paper on "Modeling approaches for time series forecasting and anomaly detection" (S Du, 2017) . and it mentions using sMAPE as "This metric is more robust towards outliers and it has a unified scale across different time series with different scale." (ibid., 4)
But most docs I see are using the standard RMSE, MSE, and MAE metrics.
Would love to have this answered.
answered 1 hour ago
JohnnyDenimJohnnyDenim
1
New contributor
JohnnyDenim is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered 1 hour ago
JohnnyDenimJohnnyDenim
1
New contributor
JohnnyDenim is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered 1 hour ago
JohnnyDenimJohnnyDenim
1
answered 1 hour ago
JohnnyDenimJohnnyDenim
1
1
New contributor
JohnnyDenim is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
JohnnyDenim is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
JohnnyDenim is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f41093%2fusing-smape-as-a-loss-function-for-an-lstm%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
