Why does not log transformation make the data normalized?How to approach the numer.ai competition with anonymous scaled numerical predictors?What does normalizing and mean centering data do?TFLearn Does Not Load ModelWhy `max_features=n_features` does not make the Random Forest independent of number of trees?Am I doing a log transformation of data correctly?Why this model does not converge in keras?How standardizing and/or log transformation affect prediction result in machine learning modelsWhat is the reason behind taking log transformation of few continuous variables?why tsne plot can not show all the labels
Writing in a Christian voice
Is there a distance limit for minecart tracks?
Is divisi notation needed for brass or woodwind in an orchestra?
is this saw blade faulty?
Get top 1 row value from third table while joining 3 tables mssql
Output visual diagram of picture
What do the positive and negative (+/-) transmit and receive pins mean on Ethernet cables?
Mortal danger in mid-grade literature
Do people actually use the word "kaputt" in conversation?
Weird lines in Microsoft Word
Exposing a company lying about themselves in a tightly knit industry (videogames) : Is my career at risk on the long run?
Offset in split text content
Strange behavior of Out[]
Matching feature by ID from CSV
Why doesn't the fusion process of the sun speed up?
Should I be concerned about student access to a test bank?
Why is indicated airspeed rather than ground speed used during the takeoff roll?
Friend wants my recommendation but I don't want to give it to him
Is Crypto.generateMac method work exactly the same as Mac and SecretKeySpec java classes
Is this Pascal's Matrix?
Capacitor electron flow
Cedent or Cedant
Why is "la Gestapo" feminine?
What happens when the centripetal force is equal and opposite to the centrifugal force?
Why does not log transformation make the data normalized?
How to approach the numer.ai competition with anonymous scaled numerical predictors?What does normalizing and mean centering data do?TFLearn Does Not Load ModelWhy `max_features=n_features` does not make the Random Forest independent of number of trees?Am I doing a log transformation of data correctly?Why this model does not converge in keras?How standardizing and/or log transformation affect prediction result in machine learning modelsWhat is the reason behind taking log transformation of few continuous variables?why tsne plot can not show all the labels
$begingroup$
Having some skewed features as shown in the following figure. I am trying to imply log transformation to the feature called vBMD(mgHA/cm3). I run the following codes
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
df=pd.read_csv("Data.csv")
scaler=MinMaxScaler(feature_range=(0,1))
df['vBMD (mgHA/cm3)']=scaler.fit_transform(np.array(df['vBMD
(mgHA/cm3)']).reshape(-1,1))
df['vBMD (mgHA/cm3)']=np.log(np.array(df['vBMD (mgHA/cm3)']))
After the transfromation, I have got the following result.
While I am waiting that the feature will be normalized, its skewness increased. Thus, what am I doing wrong?
python preprocessing
asked Mar 6 at 8:39
RamRam
193
$endgroup$
add a comment |
$begingroup$
Having some skewed features as shown in the following figure. I am trying to imply log transformation to the feature called vBMD(mgHA/cm3). I run the following codes
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
df=pd.read_csv("Data.csv")
scaler=MinMaxScaler(feature_range=(0,1))
df['vBMD (mgHA/cm3)']=scaler.fit_transform(np.array(df['vBMD
(mgHA/cm3)']).reshape(-1,1))
df['vBMD (mgHA/cm3)']=np.log(np.array(df['vBMD (mgHA/cm3)']))
After the transfromation, I have got the following result.
While I am waiting that the feature will be normalized, its skewness increased. Thus, what am I doing wrong?
python preprocessing
asked Mar 6 at 8:39
RamRam
193
$endgroup$
add a comment |
$begingroup$
Having some skewed features as shown in the following figure. I am trying to imply log transformation to the feature called vBMD(mgHA/cm3). I run the following codes
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
df=pd.read_csv("Data.csv")
scaler=MinMaxScaler(feature_range=(0,1))
df['vBMD (mgHA/cm3)']=scaler.fit_transform(np.array(df['vBMD
(mgHA/cm3)']).reshape(-1,1))
df['vBMD (mgHA/cm3)']=np.log(np.array(df['vBMD (mgHA/cm3)']))
After the transfromation, I have got the following result.
While I am waiting that the feature will be normalized, its skewness increased. Thus, what am I doing wrong?
python preprocessing
asked Mar 6 at 8:39
RamRam
193
$endgroup$
Having some skewed features as shown in the following figure. I am trying to imply log transformation to the feature called vBMD(mgHA/cm3). I run the following codes
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
df=pd.read_csv("Data.csv")
scaler=MinMaxScaler(feature_range=(0,1))
df['vBMD (mgHA/cm3)']=scaler.fit_transform(np.array(df['vBMD
(mgHA/cm3)']).reshape(-1,1))
df['vBMD (mgHA/cm3)']=np.log(np.array(df['vBMD (mgHA/cm3)']))
After the transfromation, I have got the following result.
While I am waiting that the feature will be normalized, its skewness increased. Thus, what am I doing wrong?
python preprocessing
python preprocessing
asked Mar 6 at 8:39
RamRam
193
asked Mar 6 at 8:39
RamRam
193
asked Mar 6 at 8:39
RamRam
193
asked Mar 6 at 8:39
RamRam
193
asked Mar 6 at 8:39
RamRam
193
193
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
Log transformation leads to a normal distribution only for log-normal distributions. Not all distributions are log-normal, meaning they will not become normal after the log transformation.
EDIT:
As you have commented, if you are trying to convert an arbitrary distribution to normal, methods like QuantileTransformer can be used. But note that these transformations make a distribution normal by changing (destroying) some information from the original data.
answered Mar 6 at 10:23
EsmailianEsmailian
1,491113
$endgroup$
1
$begingroup$
Thanks a lot. I figure out the main idea. In my case, to make mentioned feature normalize, can I use QuantileTransformer such asquantile_transformer = QuantileTransformer(output_distribution='normal', random_state=0)df['vBMD (mgHA/cm3)'] = quantile_transformer.fit_transform(np.array(df['vBMD (mgHA/cm3)']).reshape(-1,1))
$endgroup$
– Ram
Mar 6 at 11:57
$begingroup$
@Ram Please accept the answer if it answers the original question. Thanks.
$endgroup$
– Esmailian
1 hour ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f46763%2fwhy-does-not-log-transformation-make-the-data-normalized%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Log transformation leads to a normal distribution only for log-normal distributions. Not all distributions are log-normal, meaning they will not become normal after the log transformation.
EDIT:
As you have commented, if you are trying to convert an arbitrary distribution to normal, methods like QuantileTransformer can be used. But note that these transformations make a distribution normal by changing (destroying) some information from the original data.
answered Mar 6 at 10:23
EsmailianEsmailian
1,491113
$endgroup$
1
$begingroup$
Thanks a lot. I figure out the main idea. In my case, to make mentioned feature normalize, can I use QuantileTransformer such asquantile_transformer = QuantileTransformer(output_distribution='normal', random_state=0)df['vBMD (mgHA/cm3)'] = quantile_transformer.fit_transform(np.array(df['vBMD (mgHA/cm3)']).reshape(-1,1))
$endgroup$
– Ram
Mar 6 at 11:57
$begingroup$
@Ram Please accept the answer if it answers the original question. Thanks.
$endgroup$
– Esmailian
1 hour ago
add a comment |
$begingroup$
Log transformation leads to a normal distribution only for log-normal distributions. Not all distributions are log-normal, meaning they will not become normal after the log transformation.
EDIT:
As you have commented, if you are trying to convert an arbitrary distribution to normal, methods like QuantileTransformer can be used. But note that these transformations make a distribution normal by changing (destroying) some information from the original data.
answered Mar 6 at 10:23
EsmailianEsmailian
1,491113
$endgroup$
1
$begingroup$
Thanks a lot. I figure out the main idea. In my case, to make mentioned feature normalize, can I use QuantileTransformer such asquantile_transformer = QuantileTransformer(output_distribution='normal', random_state=0)df['vBMD (mgHA/cm3)'] = quantile_transformer.fit_transform(np.array(df['vBMD (mgHA/cm3)']).reshape(-1,1))
$endgroup$
– Ram
Mar 6 at 11:57
$begingroup$
@Ram Please accept the answer if it answers the original question. Thanks.
$endgroup$
– Esmailian
1 hour ago
add a comment |
$begingroup$
Log transformation leads to a normal distribution only for log-normal distributions. Not all distributions are log-normal, meaning they will not become normal after the log transformation.
EDIT:
As you have commented, if you are trying to convert an arbitrary distribution to normal, methods like QuantileTransformer can be used. But note that these transformations make a distribution normal by changing (destroying) some information from the original data.
answered Mar 6 at 10:23
EsmailianEsmailian
1,491113
$endgroup$
Log transformation leads to a normal distribution only for log-normal distributions. Not all distributions are log-normal, meaning they will not become normal after the log transformation.
EDIT:
As you have commented, if you are trying to convert an arbitrary distribution to normal, methods like QuantileTransformer can be used. But note that these transformations make a distribution normal by changing (destroying) some information from the original data.
answered Mar 6 at 10:23
EsmailianEsmailian
1,491113
edited 1 hour ago
answered Mar 6 at 10:23
EsmailianEsmailian
1,491113
answered Mar 6 at 10:23
EsmailianEsmailian
1,491113
answered Mar 6 at 10:23
EsmailianEsmailian
1,491113
1,491113
1
$begingroup$
Thanks a lot. I figure out the main idea. In my case, to make mentioned feature normalize, can I use QuantileTransformer such asquantile_transformer = QuantileTransformer(output_distribution='normal', random_state=0)df['vBMD (mgHA/cm3)'] = quantile_transformer.fit_transform(np.array(df['vBMD (mgHA/cm3)']).reshape(-1,1))
$endgroup$
– Ram
Mar 6 at 11:57
$begingroup$
@Ram Please accept the answer if it answers the original question. Thanks.
$endgroup$
– Esmailian
1 hour ago
add a comment |
1
$begingroup$
Thanks a lot. I figure out the main idea. In my case, to make mentioned feature normalize, can I use QuantileTransformer such asquantile_transformer = QuantileTransformer(output_distribution='normal', random_state=0)df['vBMD (mgHA/cm3)'] = quantile_transformer.fit_transform(np.array(df['vBMD (mgHA/cm3)']).reshape(-1,1))
$endgroup$
– Ram
Mar 6 at 11:57
$begingroup$
@Ram Please accept the answer if it answers the original question. Thanks.
$endgroup$
– Esmailian
1 hour ago
1
1
$begingroup$
Thanks a lot. I figure out the main idea. In my case, to make mentioned feature normalize, can I use QuantileTransformer such as
quantile_transformer = QuantileTransformer(output_distribution='normal', random_state=0) df['vBMD (mgHA/cm3)'] = quantile_transformer.fit_transform(np.array(df['vBMD (mgHA/cm3)']).reshape(-1,1))$endgroup$
– Ram
Mar 6 at 11:57
$begingroup$
Thanks a lot. I figure out the main idea. In my case, to make mentioned feature normalize, can I use QuantileTransformer such as
quantile_transformer = QuantileTransformer(output_distribution='normal', random_state=0) df['vBMD (mgHA/cm3)'] = quantile_transformer.fit_transform(np.array(df['vBMD (mgHA/cm3)']).reshape(-1,1))$endgroup$
– Ram
Mar 6 at 11:57
$begingroup$
@Ram Please accept the answer if it answers the original question. Thanks.
$endgroup$
– Esmailian
1 hour ago
$begingroup$
@Ram Please accept the answer if it answers the original question. Thanks.
$endgroup$
– Esmailian
1 hour ago
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f46763%2fwhy-does-not-log-transformation-make-the-data-normalized%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
