Dealing with multiple distinct-value categorical variables Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern) 2019 Moderator Election Q&A - Questionnaire 2019 Community Moderator Election ResultsChoosing the right data mining method to find the effect of each parameter over the targetHow to visualise multidimensional categorical data with additional time dimensionHow can I dynamically distinguish between categorical data and numerical data?Imputation of missing values and dealing with categorical valuesOutlier detection on categorical network log dataPreparing, Scaling and Selecting from a combination of numerical and categorical featureshow does XGBoost's exact greedy split finding algorithm determine candidate split values for different feature types?ML Models: How to handle categorical feature with over 1000 unique valuesProblem with important feature having a lot of missing valueTraining NLP with multiple text input features
Are my PIs rude or am I just being too sensitive?
Complexity of many constant time steps with occasional logarithmic steps
Why is there no army of Iron-Mans in the MCU?
Fishing simulator
How should I respond to a player wanting to catch a sword between their hands?
What items from the Roman-age tech-level could be used to deter all creatures from entering a small area?
How do I automatically answer y in bash script?
How does modal jazz use chord progressions?
When is phishing education going too far?
Using "nakedly" instead of "with nothing on"
Antler Helmet: Can it work?
What do you call a plan that's an alternative plan in case your initial plan fails?
Can the prologue be the backstory of your main character?
Strange behaviour of Check
Slither Like a Snake
How is simplicity better than precision and clarity in prose?
How can players take actions together that are impossible otherwise?
What's the difference between (size_t)-1 and ~0?
What to do with post with dry rot?
How are presidential pardons supposed to be used?
Can a monk deflect thrown melee weapons?
How do I keep my slimes from escaping their pens?
Was credit for the black hole image misattributed?
Working around an AWS network ACL rule limit
Dealing with multiple distinct-value categorical variables
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)
2019 Moderator Election Q&A - Questionnaire
2019 Community Moderator Election ResultsChoosing the right data mining method to find the effect of each parameter over the targetHow to visualise multidimensional categorical data with additional time dimensionHow can I dynamically distinguish between categorical data and numerical data?Imputation of missing values and dealing with categorical valuesOutlier detection on categorical network log dataPreparing, Scaling and Selecting from a combination of numerical and categorical featureshow does XGBoost's exact greedy split finding algorithm determine candidate split values for different feature types?ML Models: How to handle categorical feature with over 1000 unique valuesProblem with important feature having a lot of missing valueTraining NLP with multiple text input features
$begingroup$
So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values.
For instance, one column have more than one million unique value, it's an IP address column in case anyone is interested. Someone suggested to split it into multiple other columns using domain knowledge, so split it to Network Class type, Host type and so on. However wouldn't that make my dataset lose some information? What if I wanted to deal with IP addresses as is?
Nevertheless, the domain knowledge solution might work on the IP column, however, I've got other columns that have more than 100K distinct values, each value is a constant-length random string.
I did work with Embedding Layers before, I was dealing with max thousands of features, never worked with 10K++ features, so I'm not sure if that would work with millions.
Much Regards
machine-learning neural-network categorical-data word-embeddings
asked Mar 14 at 11:04
Abdullah MohamedAbdullah Mohamed
62
$endgroup$
bumped to the homepage by Community♦ 40 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
|
show 4 more comments
$begingroup$
So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values.
For instance, one column have more than one million unique value, it's an IP address column in case anyone is interested. Someone suggested to split it into multiple other columns using domain knowledge, so split it to Network Class type, Host type and so on. However wouldn't that make my dataset lose some information? What if I wanted to deal with IP addresses as is?
Nevertheless, the domain knowledge solution might work on the IP column, however, I've got other columns that have more than 100K distinct values, each value is a constant-length random string.
I did work with Embedding Layers before, I was dealing with max thousands of features, never worked with 10K++ features, so I'm not sure if that would work with millions.
Much Regards
machine-learning neural-network categorical-data word-embeddings
asked Mar 14 at 11:04
Abdullah MohamedAbdullah Mohamed
62
$endgroup$
bumped to the homepage by Community♦ 40 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Can you explain more about the problem you are trying to solve?
$endgroup$
– Alireza Zolanvari
Mar 14 at 11:07
$begingroup$
Mainly, I'm trying to classify data according to some inputs, the inputs mainly constitute of categorical data, each categorical variable constitutes of so many distinct values. One of the independent variables is the IP address, which is essential for my classification problem. What I'm trying to do is to binary classify based on the (mostly categorical) inputs. Does that help? Let me know if you need more details.
$endgroup$
– Abdullah Mohamed
Mar 14 at 11:51
$begingroup$
Embedding, Domain-based-features are most promising options here. For IP, it would be subnet ID, geo-location etc. Embedding works for large number of value (Such as word embedding for 10 Million+ words)
$endgroup$
– Shamit Verma
Mar 14 at 11:59
$begingroup$
What kind of information you are trying to extract from the IP?
$endgroup$
– Alireza Zolanvari
Mar 14 at 11:59
$begingroup$
@ShamitVerma My dataset already contains countries, however, the country variable might be different than the IP country (usage of VPN's/proxies for instance). I didn't know that Embeddings work for data having millions of features actually, in that case that would be a reasonable solution for my question.
$endgroup$
– Abdullah Mohamed
Mar 14 at 12:04
|
show 4 more comments
$begingroup$
So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values.
For instance, one column have more than one million unique value, it's an IP address column in case anyone is interested. Someone suggested to split it into multiple other columns using domain knowledge, so split it to Network Class type, Host type and so on. However wouldn't that make my dataset lose some information? What if I wanted to deal with IP addresses as is?
Nevertheless, the domain knowledge solution might work on the IP column, however, I've got other columns that have more than 100K distinct values, each value is a constant-length random string.
I did work with Embedding Layers before, I was dealing with max thousands of features, never worked with 10K++ features, so I'm not sure if that would work with millions.
Much Regards
machine-learning neural-network categorical-data word-embeddings
asked Mar 14 at 11:04
Abdullah MohamedAbdullah Mohamed
62
$endgroup$
So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values.
For instance, one column have more than one million unique value, it's an IP address column in case anyone is interested. Someone suggested to split it into multiple other columns using domain knowledge, so split it to Network Class type, Host type and so on. However wouldn't that make my dataset lose some information? What if I wanted to deal with IP addresses as is?
Nevertheless, the domain knowledge solution might work on the IP column, however, I've got other columns that have more than 100K distinct values, each value is a constant-length random string.
I did work with Embedding Layers before, I was dealing with max thousands of features, never worked with 10K++ features, so I'm not sure if that would work with millions.
Much Regards
machine-learning neural-network categorical-data word-embeddings
machine-learning neural-network categorical-data word-embeddings
asked Mar 14 at 11:04
Abdullah MohamedAbdullah Mohamed
62
asked Mar 14 at 11:04
Abdullah MohamedAbdullah Mohamed
62
asked Mar 14 at 11:04
Abdullah MohamedAbdullah Mohamed
62
asked Mar 14 at 11:04
Abdullah MohamedAbdullah Mohamed
62
asked Mar 14 at 11:04
Abdullah MohamedAbdullah Mohamed
62
62
bumped to the homepage by Community♦ 40 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 40 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Can you explain more about the problem you are trying to solve?
$endgroup$
– Alireza Zolanvari
Mar 14 at 11:07
$begingroup$
Mainly, I'm trying to classify data according to some inputs, the inputs mainly constitute of categorical data, each categorical variable constitutes of so many distinct values. One of the independent variables is the IP address, which is essential for my classification problem. What I'm trying to do is to binary classify based on the (mostly categorical) inputs. Does that help? Let me know if you need more details.
$endgroup$
– Abdullah Mohamed
Mar 14 at 11:51
$begingroup$
Embedding, Domain-based-features are most promising options here. For IP, it would be subnet ID, geo-location etc. Embedding works for large number of value (Such as word embedding for 10 Million+ words)
$endgroup$
– Shamit Verma
Mar 14 at 11:59
$begingroup$
What kind of information you are trying to extract from the IP?
$endgroup$
– Alireza Zolanvari
Mar 14 at 11:59
$begingroup$
@ShamitVerma My dataset already contains countries, however, the country variable might be different than the IP country (usage of VPN's/proxies for instance). I didn't know that Embeddings work for data having millions of features actually, in that case that would be a reasonable solution for my question.
$endgroup$
– Abdullah Mohamed
Mar 14 at 12:04
|
show 4 more comments
$begingroup$
Can you explain more about the problem you are trying to solve?
$endgroup$
– Alireza Zolanvari
Mar 14 at 11:07
$begingroup$
Mainly, I'm trying to classify data according to some inputs, the inputs mainly constitute of categorical data, each categorical variable constitutes of so many distinct values. One of the independent variables is the IP address, which is essential for my classification problem. What I'm trying to do is to binary classify based on the (mostly categorical) inputs. Does that help? Let me know if you need more details.
$endgroup$
– Abdullah Mohamed
Mar 14 at 11:51
$begingroup$
Embedding, Domain-based-features are most promising options here. For IP, it would be subnet ID, geo-location etc. Embedding works for large number of value (Such as word embedding for 10 Million+ words)
$endgroup$
– Shamit Verma
Mar 14 at 11:59
$begingroup$
What kind of information you are trying to extract from the IP?
$endgroup$
– Alireza Zolanvari
Mar 14 at 11:59
$begingroup$
@ShamitVerma My dataset already contains countries, however, the country variable might be different than the IP country (usage of VPN's/proxies for instance). I didn't know that Embeddings work for data having millions of features actually, in that case that would be a reasonable solution for my question.
$endgroup$
– Abdullah Mohamed
Mar 14 at 12:04
$begingroup$
Can you explain more about the problem you are trying to solve?
$endgroup$
– Alireza Zolanvari
Mar 14 at 11:07
$begingroup$
Can you explain more about the problem you are trying to solve?
$endgroup$
– Alireza Zolanvari
Mar 14 at 11:07
$begingroup$
Mainly, I'm trying to classify data according to some inputs, the inputs mainly constitute of categorical data, each categorical variable constitutes of so many distinct values. One of the independent variables is the IP address, which is essential for my classification problem. What I'm trying to do is to binary classify based on the (mostly categorical) inputs. Does that help? Let me know if you need more details.
$endgroup$
– Abdullah Mohamed
Mar 14 at 11:51
$begingroup$
Mainly, I'm trying to classify data according to some inputs, the inputs mainly constitute of categorical data, each categorical variable constitutes of so many distinct values. One of the independent variables is the IP address, which is essential for my classification problem. What I'm trying to do is to binary classify based on the (mostly categorical) inputs. Does that help? Let me know if you need more details.
$endgroup$
– Abdullah Mohamed
Mar 14 at 11:51
$begingroup$
Embedding, Domain-based-features are most promising options here. For IP, it would be subnet ID, geo-location etc. Embedding works for large number of value (Such as word embedding for 10 Million+ words)
$endgroup$
– Shamit Verma
Mar 14 at 11:59
$begingroup$
Embedding, Domain-based-features are most promising options here. For IP, it would be subnet ID, geo-location etc. Embedding works for large number of value (Such as word embedding for 10 Million+ words)
$endgroup$
– Shamit Verma
Mar 14 at 11:59
$begingroup$
What kind of information you are trying to extract from the IP?
$endgroup$
– Alireza Zolanvari
Mar 14 at 11:59
$begingroup$
What kind of information you are trying to extract from the IP?
$endgroup$
– Alireza Zolanvari
Mar 14 at 11:59
$begingroup$
@ShamitVerma My dataset already contains countries, however, the country variable might be different than the IP country (usage of VPN's/proxies for instance). I didn't know that Embeddings work for data having millions of features actually, in that case that would be a reasonable solution for my question.
$endgroup$
– Abdullah Mohamed
Mar 14 at 12:04
$begingroup$
@ShamitVerma My dataset already contains countries, however, the country variable might be different than the IP country (usage of VPN's/proxies for instance). I didn't know that Embeddings work for data having millions of features actually, in that case that would be a reasonable solution for my question.
$endgroup$
– Abdullah Mohamed
Mar 14 at 12:04
|
show 4 more comments
1 Answer
1
active
oldest
votes
$begingroup$
Have you heard of CatBoostClassifier?
https://tech.yandex.com/catboost/doc/dg/concepts/python-reference_catboostclassifier-docpage/
It is type of Boosting classifier developed to deal specifically with categorical features. It has achieved state of the art results and the package developed by the authors have excellent support and even GPU portability. Take a look, this can be your solution.
answered Mar 14 at 12:51
Victor OliveiraVictor Oliveira
3657
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47295%2fdealing-with-multiple-distinct-value-categorical-variables%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Have you heard of CatBoostClassifier?
https://tech.yandex.com/catboost/doc/dg/concepts/python-reference_catboostclassifier-docpage/
It is type of Boosting classifier developed to deal specifically with categorical features. It has achieved state of the art results and the package developed by the authors have excellent support and even GPU portability. Take a look, this can be your solution.
answered Mar 14 at 12:51
Victor OliveiraVictor Oliveira
3657
$endgroup$
add a comment |
$begingroup$
Have you heard of CatBoostClassifier?
https://tech.yandex.com/catboost/doc/dg/concepts/python-reference_catboostclassifier-docpage/
It is type of Boosting classifier developed to deal specifically with categorical features. It has achieved state of the art results and the package developed by the authors have excellent support and even GPU portability. Take a look, this can be your solution.
answered Mar 14 at 12:51
Victor OliveiraVictor Oliveira
3657
$endgroup$
add a comment |
$begingroup$
Have you heard of CatBoostClassifier?
https://tech.yandex.com/catboost/doc/dg/concepts/python-reference_catboostclassifier-docpage/
It is type of Boosting classifier developed to deal specifically with categorical features. It has achieved state of the art results and the package developed by the authors have excellent support and even GPU portability. Take a look, this can be your solution.
answered Mar 14 at 12:51
Victor OliveiraVictor Oliveira
3657
$endgroup$
Have you heard of CatBoostClassifier?
https://tech.yandex.com/catboost/doc/dg/concepts/python-reference_catboostclassifier-docpage/
It is type of Boosting classifier developed to deal specifically with categorical features. It has achieved state of the art results and the package developed by the authors have excellent support and even GPU portability. Take a look, this can be your solution.
answered Mar 14 at 12:51
Victor OliveiraVictor Oliveira
3657
answered Mar 14 at 12:51
Victor OliveiraVictor Oliveira
3657
answered Mar 14 at 12:51
Victor OliveiraVictor Oliveira
3657
answered Mar 14 at 12:51
Victor OliveiraVictor Oliveira
3657
3657
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47295%2fdealing-with-multiple-distinct-value-categorical-variables%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Can you explain more about the problem you are trying to solve?
$endgroup$
– Alireza Zolanvari
Mar 14 at 11:07
$begingroup$
Mainly, I'm trying to classify data according to some inputs, the inputs mainly constitute of categorical data, each categorical variable constitutes of so many distinct values. One of the independent variables is the IP address, which is essential for my classification problem. What I'm trying to do is to binary classify based on the (mostly categorical) inputs. Does that help? Let me know if you need more details.
$endgroup$
– Abdullah Mohamed
Mar 14 at 11:51
$begingroup$
Embedding, Domain-based-features are most promising options here. For IP, it would be subnet ID, geo-location etc. Embedding works for large number of value (Such as word embedding for 10 Million+ words)
$endgroup$
– Shamit Verma
Mar 14 at 11:59
$begingroup$
What kind of information you are trying to extract from the IP?
$endgroup$
– Alireza Zolanvari
Mar 14 at 11:59
$begingroup$
@ShamitVerma My dataset already contains countries, however, the country variable might be different than the IP country (usage of VPN's/proxies for instance). I didn't know that Embeddings work for data having millions of features actually, in that case that would be a reasonable solution for my question.
$endgroup$
– Abdullah Mohamed
Mar 14 at 12:04