Resource and useful tips on Transfer Learning in NLP Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 00:00UTC (8:00pm US/Eastern) 2019 Moderator Election Q&A - Questionnaire 2019 Community Moderator Election ResultsConvnet training error does not decreaseChoosing the right model to learntransfer learning with sentiment analysis?Transfer learning by concatenating the last classification layercorrecting conditional and marginal distribution in transfer learningTransfer learning - small databaseCan transfer learning be applied to predict salesHow to input different sized images into transfer learning networkNeural Network Model using Transfer Learning not learningOutput range of BERT model shrinks after fine-tuning on domain specific dataset
Disembodied hand growing fangs
Amount of permutations on an NxNxN Rubik's Cube
How do I use the new nonlinear finite element in Mathematica 12 for this equation?
Can a new player join a group only when a new campaign starts?
How do living politicians protect their readily obtainable signatures from misuse?
Why do we need to use the builder design pattern when we can do the same thing with setters?
Chinese Seal on silk painting - what does it mean?
Can anything be seen from the center of the Boötes void? How dark would it be?
Putting class ranking in CV, but against dept guidelines
What initially awakened the Balrog?
How to react to hostile behavior from a senior developer?
AppleTVs create a chatty alternate WiFi network
Do wooden building fires get hotter than 600°C?
What's the meaning of "fortified infraction restraint"?
How could we fake a moon landing now?
Time to Settle Down!
How does Python know the values already stored in its memory?
Maximum summed subsequences with non-adjacent items
Take 2! Is this homebrew Lady of Pain warlock patron balanced?
How to play a character with a disability or mental disorder without being offensive?
Is there a kind of relay that only consumes power when switching?
What order were files/directories outputted in dir?
Drawing without replacement: why is the order of draw irrelevant?
Do I really need to have a message in a novel to appeal to readers?
Resource and useful tips on Transfer Learning in NLP
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 00:00UTC (8:00pm US/Eastern)
2019 Moderator Election Q&A - Questionnaire
2019 Community Moderator Election ResultsConvnet training error does not decreaseChoosing the right model to learntransfer learning with sentiment analysis?Transfer learning by concatenating the last classification layercorrecting conditional and marginal distribution in transfer learningTransfer learning - small databaseCan transfer learning be applied to predict salesHow to input different sized images into transfer learning networkNeural Network Model using Transfer Learning not learningOutput range of BERT model shrinks after fine-tuning on domain specific dataset
$begingroup$
I have a few label data for training and testing a DNN. Main purpose of my work is to train a model which can do a binary classification of text. And for this purpose, I have around 3000 label data and 60000 unlabeled data available to me. My data type is related to instructions (like- open the door[label-1], give me a cup of water[label-1], give me money[label-0] etc.) In this case, I heard that Transferring knowledge from other models will help me a lot. Can anyone give me some useful resource for transfer learning in NLP domain?
I already did a few experiments. I used GLoVE as a pretrained embeddings. Then test it with my label data. But got around 70% accuracy. Also tried with embedding built using my own data (63k) and then train the model. Got 75% accuracy on the test data. My model architecture is given below-
Q1: I have a quick question will it be referred to as Transfer learning if I use GLOVE embeddings in model?
Any kind of help is welcomed. Even someone has other ideas for building a model without using transfer learning is welcomed.
deep-learning nlp convnet word-embeddings transfer-learning
asked Aug 20 '18 at 5:12
faysalfaysal
61
$endgroup$
bumped to the homepage by Community♦ 25 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I have a few label data for training and testing a DNN. Main purpose of my work is to train a model which can do a binary classification of text. And for this purpose, I have around 3000 label data and 60000 unlabeled data available to me. My data type is related to instructions (like- open the door[label-1], give me a cup of water[label-1], give me money[label-0] etc.) In this case, I heard that Transferring knowledge from other models will help me a lot. Can anyone give me some useful resource for transfer learning in NLP domain?
I already did a few experiments. I used GLoVE as a pretrained embeddings. Then test it with my label data. But got around 70% accuracy. Also tried with embedding built using my own data (63k) and then train the model. Got 75% accuracy on the test data. My model architecture is given below-
Q1: I have a quick question will it be referred to as Transfer learning if I use GLOVE embeddings in model?
Any kind of help is welcomed. Even someone has other ideas for building a model without using transfer learning is welcomed.
deep-learning nlp convnet word-embeddings transfer-learning
asked Aug 20 '18 at 5:12
faysalfaysal
61
$endgroup$
bumped to the homepage by Community♦ 25 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I have a few label data for training and testing a DNN. Main purpose of my work is to train a model which can do a binary classification of text. And for this purpose, I have around 3000 label data and 60000 unlabeled data available to me. My data type is related to instructions (like- open the door[label-1], give me a cup of water[label-1], give me money[label-0] etc.) In this case, I heard that Transferring knowledge from other models will help me a lot. Can anyone give me some useful resource for transfer learning in NLP domain?
I already did a few experiments. I used GLoVE as a pretrained embeddings. Then test it with my label data. But got around 70% accuracy. Also tried with embedding built using my own data (63k) and then train the model. Got 75% accuracy on the test data. My model architecture is given below-
Q1: I have a quick question will it be referred to as Transfer learning if I use GLOVE embeddings in model?
Any kind of help is welcomed. Even someone has other ideas for building a model without using transfer learning is welcomed.
deep-learning nlp convnet word-embeddings transfer-learning
asked Aug 20 '18 at 5:12
faysalfaysal
61
$endgroup$
I have a few label data for training and testing a DNN. Main purpose of my work is to train a model which can do a binary classification of text. And for this purpose, I have around 3000 label data and 60000 unlabeled data available to me. My data type is related to instructions (like- open the door[label-1], give me a cup of water[label-1], give me money[label-0] etc.) In this case, I heard that Transferring knowledge from other models will help me a lot. Can anyone give me some useful resource for transfer learning in NLP domain?
I already did a few experiments. I used GLoVE as a pretrained embeddings. Then test it with my label data. But got around 70% accuracy. Also tried with embedding built using my own data (63k) and then train the model. Got 75% accuracy on the test data. My model architecture is given below-
Q1: I have a quick question will it be referred to as Transfer learning if I use GLOVE embeddings in model?
Any kind of help is welcomed. Even someone has other ideas for building a model without using transfer learning is welcomed.
deep-learning nlp convnet word-embeddings transfer-learning
deep-learning nlp convnet word-embeddings transfer-learning
asked Aug 20 '18 at 5:12
faysalfaysal
61
asked Aug 20 '18 at 5:12
faysalfaysal
61
asked Aug 20 '18 at 5:12
faysalfaysal
61
asked Aug 20 '18 at 5:12
faysalfaysal
61
asked Aug 20 '18 at 5:12
faysalfaysal
61
61
bumped to the homepage by Community♦ 25 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 25 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
If you use pre-trained models on data that are distinct from the data they were originally trained on, it's transfer learning. Your two-class sentence corpus is distinct from the data that the GloVe embeddings were generated on, so this could be considered a form of transfer learning. This might be a helpful explainer for general ideas around pre-training (and why it's a worthy pursuit).
Recent work in the NLP transfer learning space that I'm aware of is ULMFiT by Howard and Ruder of fast.ai, here's the paper if you prefer that. OpenAI also has recent work extending the Transformer model with a unsupervised pre-training, task specific fine-tuning approach.
As for your task, I think it might be helpful to explore research around sentence classification rather than digging deeply into transfer learning. For your purposes, it seems that embeddings are a means to have a reasonable representation of your data rather than prove that Common Crawl (or some other dataset) extends to your corpus.
Hope that helps, good luck!
answered Aug 20 '18 at 15:14
tm1212tm1212
46517
$endgroup$
$begingroup$
Your comments are really helpful. Yes I am aware of fast.ai. I was also thinking of using sentence classification without using transfer learning. Also, getting around 87~90% accuracy without using TL. Do you think it is reasonable accuracy in case of training 673606 params with around 3k label data?
$endgroup$
– faysal
Aug 26 '18 at 20:33
$begingroup$
I don't know enough about the data, difficulty of the task, or the final application of what you're working on to have a well-founded answer for you.
$endgroup$
– tm1212
Aug 27 '18 at 15:10
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f37167%2fresource-and-useful-tips-on-transfer-learning-in-nlp%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
If you use pre-trained models on data that are distinct from the data they were originally trained on, it's transfer learning. Your two-class sentence corpus is distinct from the data that the GloVe embeddings were generated on, so this could be considered a form of transfer learning. This might be a helpful explainer for general ideas around pre-training (and why it's a worthy pursuit).
Recent work in the NLP transfer learning space that I'm aware of is ULMFiT by Howard and Ruder of fast.ai, here's the paper if you prefer that. OpenAI also has recent work extending the Transformer model with a unsupervised pre-training, task specific fine-tuning approach.
As for your task, I think it might be helpful to explore research around sentence classification rather than digging deeply into transfer learning. For your purposes, it seems that embeddings are a means to have a reasonable representation of your data rather than prove that Common Crawl (or some other dataset) extends to your corpus.
Hope that helps, good luck!
answered Aug 20 '18 at 15:14
tm1212tm1212
46517
$endgroup$
$begingroup$
Your comments are really helpful. Yes I am aware of fast.ai. I was also thinking of using sentence classification without using transfer learning. Also, getting around 87~90% accuracy without using TL. Do you think it is reasonable accuracy in case of training 673606 params with around 3k label data?
$endgroup$
– faysal
Aug 26 '18 at 20:33
$begingroup$
I don't know enough about the data, difficulty of the task, or the final application of what you're working on to have a well-founded answer for you.
$endgroup$
– tm1212
Aug 27 '18 at 15:10
add a comment |
$begingroup$
If you use pre-trained models on data that are distinct from the data they were originally trained on, it's transfer learning. Your two-class sentence corpus is distinct from the data that the GloVe embeddings were generated on, so this could be considered a form of transfer learning. This might be a helpful explainer for general ideas around pre-training (and why it's a worthy pursuit).
Recent work in the NLP transfer learning space that I'm aware of is ULMFiT by Howard and Ruder of fast.ai, here's the paper if you prefer that. OpenAI also has recent work extending the Transformer model with a unsupervised pre-training, task specific fine-tuning approach.
As for your task, I think it might be helpful to explore research around sentence classification rather than digging deeply into transfer learning. For your purposes, it seems that embeddings are a means to have a reasonable representation of your data rather than prove that Common Crawl (or some other dataset) extends to your corpus.
Hope that helps, good luck!
answered Aug 20 '18 at 15:14
tm1212tm1212
46517
$endgroup$
$begingroup$
Your comments are really helpful. Yes I am aware of fast.ai. I was also thinking of using sentence classification without using transfer learning. Also, getting around 87~90% accuracy without using TL. Do you think it is reasonable accuracy in case of training 673606 params with around 3k label data?
$endgroup$
– faysal
Aug 26 '18 at 20:33
$begingroup$
I don't know enough about the data, difficulty of the task, or the final application of what you're working on to have a well-founded answer for you.
$endgroup$
– tm1212
Aug 27 '18 at 15:10
add a comment |
$begingroup$
If you use pre-trained models on data that are distinct from the data they were originally trained on, it's transfer learning. Your two-class sentence corpus is distinct from the data that the GloVe embeddings were generated on, so this could be considered a form of transfer learning. This might be a helpful explainer for general ideas around pre-training (and why it's a worthy pursuit).
Recent work in the NLP transfer learning space that I'm aware of is ULMFiT by Howard and Ruder of fast.ai, here's the paper if you prefer that. OpenAI also has recent work extending the Transformer model with a unsupervised pre-training, task specific fine-tuning approach.
As for your task, I think it might be helpful to explore research around sentence classification rather than digging deeply into transfer learning. For your purposes, it seems that embeddings are a means to have a reasonable representation of your data rather than prove that Common Crawl (or some other dataset) extends to your corpus.
Hope that helps, good luck!
answered Aug 20 '18 at 15:14
tm1212tm1212
46517
$endgroup$
If you use pre-trained models on data that are distinct from the data they were originally trained on, it's transfer learning. Your two-class sentence corpus is distinct from the data that the GloVe embeddings were generated on, so this could be considered a form of transfer learning. This might be a helpful explainer for general ideas around pre-training (and why it's a worthy pursuit).
Recent work in the NLP transfer learning space that I'm aware of is ULMFiT by Howard and Ruder of fast.ai, here's the paper if you prefer that. OpenAI also has recent work extending the Transformer model with a unsupervised pre-training, task specific fine-tuning approach.
As for your task, I think it might be helpful to explore research around sentence classification rather than digging deeply into transfer learning. For your purposes, it seems that embeddings are a means to have a reasonable representation of your data rather than prove that Common Crawl (or some other dataset) extends to your corpus.
Hope that helps, good luck!
answered Aug 20 '18 at 15:14
tm1212tm1212
46517
answered Aug 20 '18 at 15:14
tm1212tm1212
46517
answered Aug 20 '18 at 15:14
tm1212tm1212
46517
answered Aug 20 '18 at 15:14
tm1212tm1212
46517
46517
$begingroup$
Your comments are really helpful. Yes I am aware of fast.ai. I was also thinking of using sentence classification without using transfer learning. Also, getting around 87~90% accuracy without using TL. Do you think it is reasonable accuracy in case of training 673606 params with around 3k label data?
$endgroup$
– faysal
Aug 26 '18 at 20:33
$begingroup$
I don't know enough about the data, difficulty of the task, or the final application of what you're working on to have a well-founded answer for you.
$endgroup$
– tm1212
Aug 27 '18 at 15:10
add a comment |
$begingroup$
Your comments are really helpful. Yes I am aware of fast.ai. I was also thinking of using sentence classification without using transfer learning. Also, getting around 87~90% accuracy without using TL. Do you think it is reasonable accuracy in case of training 673606 params with around 3k label data?
$endgroup$
– faysal
Aug 26 '18 at 20:33
$begingroup$
I don't know enough about the data, difficulty of the task, or the final application of what you're working on to have a well-founded answer for you.
$endgroup$
– tm1212
Aug 27 '18 at 15:10
$begingroup$
Your comments are really helpful. Yes I am aware of fast.ai. I was also thinking of using sentence classification without using transfer learning. Also, getting around 87~90% accuracy without using TL. Do you think it is reasonable accuracy in case of training 673606 params with around 3k label data?
$endgroup$
– faysal
Aug 26 '18 at 20:33
$begingroup$
Your comments are really helpful. Yes I am aware of fast.ai. I was also thinking of using sentence classification without using transfer learning. Also, getting around 87~90% accuracy without using TL. Do you think it is reasonable accuracy in case of training 673606 params with around 3k label data?
$endgroup$
– faysal
Aug 26 '18 at 20:33
$begingroup$
I don't know enough about the data, difficulty of the task, or the final application of what you're working on to have a well-founded answer for you.
$endgroup$
– tm1212
Aug 27 '18 at 15:10
$begingroup$
I don't know enough about the data, difficulty of the task, or the final application of what you're working on to have a well-founded answer for you.
$endgroup$
– tm1212
Aug 27 '18 at 15:10
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f37167%2fresource-and-useful-tips-on-transfer-learning-in-nlp%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
