Reducing noisy data from non normal distribution of data with std deviation? The 2019 Stack Overflow Developer Survey Results Are InWhen to remove outlier in preparing features for machine learning algorithmWhat is the loss function defined by Mnih and Hinton in their paper “Learning to Label Aerial Images from Noisy Data”?Paramaeter estimation in noisy conditions with Machine Learning, possible?Training deep CNN with noisy dataset
Why can Shazam do this?
CiviEvent: Public link for events of a specific type
What does "sndry explns" mean in one of the Hitchhiker's guide books?
What is the motivation for a law requiring 2 parties to consent for recording a conversation
Carnot-Caratheodory metric
In microwave frequencies, do you use a circulator when you need a (near) perfect diode?
What is the best strategy for white in this position?
What does Linus Torvalds mean when he says that Git "never ever" tracks a file?
Why is it "Tumoren" and not "Tumore"?
Should I use my personal or workplace e-mail when registering to external websites for work purpose?
How to change the limits of integration
How come people say “Would of”?
Why did Howard Stark use all the Vibranium they had on a prototype shield?
I see my dog run
"What time...?" or "At what time...?" - what is more grammatically correct?
What do the Banks children have against barley water?
aging parents with no investments
Inversion Puzzle
Does duplicating a spell with Wish count as casting that spell?
Why is the maximum length of OpenWrt’s root password 8 characters?
What is the steepest angle that a canal can be traversable without locks?
Limit the amount of RAM Mathematica may access?
Pristine Bit Checking
How can I create a character who can assume the widest possible range of creature sizes?
Reducing noisy data from non normal distribution of data with std deviation?
The 2019 Stack Overflow Developer Survey Results Are InWhen to remove outlier in preparing features for machine learning algorithmWhat is the loss function defined by Mnih and Hinton in their paper “Learning to Label Aerial Images from Noisy Data”?Paramaeter estimation in noisy conditions with Machine Learning, possible?Training deep CNN with noisy dataset
$begingroup$
I have used MATLAB code and get the two different row vectors A=1×18 and B=1×350. From both row vectors separately I need to remove the noisy data by using standard deviation. But the problem is that data in both row vectors are NOT normally distributed. Is there any way that I used standard deviation for reducing noise from non normally distributed data. Any guidance will be appreciated. Thanks
noise
asked Aug 12 '18 at 8:49
user57546user57546
1
$endgroup$
bumped to the homepage by Community♦ 10 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I have used MATLAB code and get the two different row vectors A=1×18 and B=1×350. From both row vectors separately I need to remove the noisy data by using standard deviation. But the problem is that data in both row vectors are NOT normally distributed. Is there any way that I used standard deviation for reducing noise from non normally distributed data. Any guidance will be appreciated. Thanks
noise
asked Aug 12 '18 at 8:49
user57546user57546
1
$endgroup$
bumped to the homepage by Community♦ 10 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I have used MATLAB code and get the two different row vectors A=1×18 and B=1×350. From both row vectors separately I need to remove the noisy data by using standard deviation. But the problem is that data in both row vectors are NOT normally distributed. Is there any way that I used standard deviation for reducing noise from non normally distributed data. Any guidance will be appreciated. Thanks
noise
asked Aug 12 '18 at 8:49
user57546user57546
1
$endgroup$
I have used MATLAB code and get the two different row vectors A=1×18 and B=1×350. From both row vectors separately I need to remove the noisy data by using standard deviation. But the problem is that data in both row vectors are NOT normally distributed. Is there any way that I used standard deviation for reducing noise from non normally distributed data. Any guidance will be appreciated. Thanks
noise
noise
asked Aug 12 '18 at 8:49
user57546user57546
1
asked Aug 12 '18 at 8:49
user57546user57546
1
asked Aug 12 '18 at 8:49
user57546user57546
1
asked Aug 12 '18 at 8:49
user57546user57546
1
asked Aug 12 '18 at 8:49
user57546user57546
1
1
bumped to the homepage by Community♦ 10 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 10 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
First, good practice to raise validity concerns here when removing outliers and/or filtering data. This may have strong affects of validity of results. An intro is here: When to remove outlier in preparing features for machine learning algorithm .
Second, is it possible to address the small dataset problem -- Can you collect more data? Redefine the population to produce more data? Use another data set that is similar in developing the model?
Lastly, this seems to be a filter problem, to get started on a solution, check MATLAB documentation for filters.
If the results are going to be used anywhere, probably a good idea to document all of your decisions and the first two concerns. Absent much more detail, experience says there is a pretty high risk to any conclusions based on this model.
answered Aug 12 '18 at 14:36
davmordavmor
914
$endgroup$
$begingroup$
@ davmor thanks for your guidance. First I will try what you have mentioned in your answer, then will discuss. thanks
$endgroup$
– user57546
Sep 10 '18 at 14:30
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f36814%2freducing-noisy-data-from-non-normal-distribution-of-data-with-std-deviation%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
First, good practice to raise validity concerns here when removing outliers and/or filtering data. This may have strong affects of validity of results. An intro is here: When to remove outlier in preparing features for machine learning algorithm .
Second, is it possible to address the small dataset problem -- Can you collect more data? Redefine the population to produce more data? Use another data set that is similar in developing the model?
Lastly, this seems to be a filter problem, to get started on a solution, check MATLAB documentation for filters.
If the results are going to be used anywhere, probably a good idea to document all of your decisions and the first two concerns. Absent much more detail, experience says there is a pretty high risk to any conclusions based on this model.
answered Aug 12 '18 at 14:36
davmordavmor
914
$endgroup$
$begingroup$
@ davmor thanks for your guidance. First I will try what you have mentioned in your answer, then will discuss. thanks
$endgroup$
– user57546
Sep 10 '18 at 14:30
add a comment |
$begingroup$
First, good practice to raise validity concerns here when removing outliers and/or filtering data. This may have strong affects of validity of results. An intro is here: When to remove outlier in preparing features for machine learning algorithm .
Second, is it possible to address the small dataset problem -- Can you collect more data? Redefine the population to produce more data? Use another data set that is similar in developing the model?
Lastly, this seems to be a filter problem, to get started on a solution, check MATLAB documentation for filters.
If the results are going to be used anywhere, probably a good idea to document all of your decisions and the first two concerns. Absent much more detail, experience says there is a pretty high risk to any conclusions based on this model.
answered Aug 12 '18 at 14:36
davmordavmor
914
$endgroup$
$begingroup$
@ davmor thanks for your guidance. First I will try what you have mentioned in your answer, then will discuss. thanks
$endgroup$
– user57546
Sep 10 '18 at 14:30
add a comment |
$begingroup$
First, good practice to raise validity concerns here when removing outliers and/or filtering data. This may have strong affects of validity of results. An intro is here: When to remove outlier in preparing features for machine learning algorithm .
Second, is it possible to address the small dataset problem -- Can you collect more data? Redefine the population to produce more data? Use another data set that is similar in developing the model?
Lastly, this seems to be a filter problem, to get started on a solution, check MATLAB documentation for filters.
If the results are going to be used anywhere, probably a good idea to document all of your decisions and the first two concerns. Absent much more detail, experience says there is a pretty high risk to any conclusions based on this model.
answered Aug 12 '18 at 14:36
davmordavmor
914
$endgroup$
First, good practice to raise validity concerns here when removing outliers and/or filtering data. This may have strong affects of validity of results. An intro is here: When to remove outlier in preparing features for machine learning algorithm .
Second, is it possible to address the small dataset problem -- Can you collect more data? Redefine the population to produce more data? Use another data set that is similar in developing the model?
Lastly, this seems to be a filter problem, to get started on a solution, check MATLAB documentation for filters.
If the results are going to be used anywhere, probably a good idea to document all of your decisions and the first two concerns. Absent much more detail, experience says there is a pretty high risk to any conclusions based on this model.
answered Aug 12 '18 at 14:36
davmordavmor
914
answered Aug 12 '18 at 14:36
davmordavmor
914
answered Aug 12 '18 at 14:36
davmordavmor
914
answered Aug 12 '18 at 14:36
davmordavmor
914
914
$begingroup$
@ davmor thanks for your guidance. First I will try what you have mentioned in your answer, then will discuss. thanks
$endgroup$
– user57546
Sep 10 '18 at 14:30
add a comment |
$begingroup$
@ davmor thanks for your guidance. First I will try what you have mentioned in your answer, then will discuss. thanks
$endgroup$
– user57546
Sep 10 '18 at 14:30
$begingroup$
@ davmor thanks for your guidance. First I will try what you have mentioned in your answer, then will discuss. thanks
$endgroup$
– user57546
Sep 10 '18 at 14:30
$begingroup$
@ davmor thanks for your guidance. First I will try what you have mentioned in your answer, then will discuss. thanks
$endgroup$
– user57546
Sep 10 '18 at 14:30
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f36814%2freducing-noisy-data-from-non-normal-distribution-of-data-with-std-deviation%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
