Improve NER label results on Non-English text Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern) 2019 Moderator Election Q&A - Questionnaire 2019 Community Moderator Election ResultsPrediction with non-scalar output (label)Multi-label Text ClassificationHow does MITIE perform named entity recognition?Can training label confidence be used to improve prediction accuracy?Handling Larger number of category label in text classificationDetecting Offensive Text Content in English and GermanExtracting specific data from unstructured text - NERHow to improve accuracy of Named entity recognition (NER) tagger on local data?Training NLP with multiple text input featuresImprove results using user input
What is Arya's weapon design?
Short Story with Cinderella as a Voo-doo Witch
Should I discuss the type of campaign with my players?
How do I keep my slimes from escaping their pens?
Is it ethical to give a final exam after the professor has quit before teaching the remaining chapters of the course?
English words in a non-english sci-fi novel
When do you get frequent flier miles - when you buy, or when you fly?
What would be the ideal power source for a cybernetic eye?
Is the Standard Deduction better than Itemized when both are the same amount?
What is Wonderstone and are there any references to it pre-1982?
Check which numbers satisfy the condition [A*B*C = A! + B! + C!]
When were vectors invented?
String `!23` is replaced with `docker` in command line
Why did the rest of the Eastern Bloc not invade Yugoslavia?
Identify plant with long narrow paired leaves and reddish stems
Can a USB port passively 'listen only'?
The logistics of corpse disposal
What's the meaning of 間時肆拾貳 at a car parking sign
Storing hydrofluoric acid before the invention of plastics
Naming the result of a source block
How to find out what spells would be useless to a blind NPC spellcaster?
How does the particle を relate to the verb 行く in the structure「A を + B に行く」?
Using audio cues to encourage good posture
How can I make names more distinctive without making them longer?
Improve NER label results on Non-English text
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)
2019 Moderator Election Q&A - Questionnaire
2019 Community Moderator Election ResultsPrediction with non-scalar output (label)Multi-label Text ClassificationHow does MITIE perform named entity recognition?Can training label confidence be used to improve prediction accuracy?Handling Larger number of category label in text classificationDetecting Offensive Text Content in English and GermanExtracting specific data from unstructured text - NERHow to improve accuracy of Named entity recognition (NER) tagger on local data?Training NLP with multiple text input featuresImprove results using user input
$begingroup$
I am working on some Medieval Latin text and was using various methods of NER such as CLTK (Latin Model), Spacy (Multilingual, Italian, Spanish Model) and StanfordNER (Spanish Model). When I used the non-Latin models I used the original Latin text as the translated one was not making any sense.
Fortunately Spacy Multilingual model managed to extract all Persons and Places of the sample documents but with additional words that I am not considering them as Entities. Moreover, the labels are incorrect.
Here is an example output:
'LOC': ['Artali', 'Artalis', 'Bruges', 'Unde'],
'MISC': ['Marianum lu Tignusu'],
'PER': ['Simone de Mazara',
'Artalem de Alagona',
'Apoca',
'Coram',
'Pero de Naso',
'Pero Caruana',
'Bartholomeo Xacara',
'Testamur',
'Artalis de Alagona',
'Melite',
'Simonis de Mazara',
'Simonem',
'Simone',
'Mariano',
'Artalis',
'Artalem',
'Simoni',
'Panormi',
'Renunciando']
where the LOCATIONS should be: Panormi, Bruges, Melite and PERSONAL names should be all others except Unde, Apoca, Coram, Testamur, Renunciando which are neither locations nor personal names.
I was thinking of ignoring the labels and do some classification ML algorithm. The problem is that I do not have any training data available and the only possible usable corpus that I think it might be useful is Proiel treebank which labels proper nouns as NE. How would you go with such a problem?
machine-learning classification named-entity-recognition
$endgroup$
bumped to the homepage by Community♦ 27 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I am working on some Medieval Latin text and was using various methods of NER such as CLTK (Latin Model), Spacy (Multilingual, Italian, Spanish Model) and StanfordNER (Spanish Model). When I used the non-Latin models I used the original Latin text as the translated one was not making any sense.
Fortunately Spacy Multilingual model managed to extract all Persons and Places of the sample documents but with additional words that I am not considering them as Entities. Moreover, the labels are incorrect.
Here is an example output:
'LOC': ['Artali', 'Artalis', 'Bruges', 'Unde'],
'MISC': ['Marianum lu Tignusu'],
'PER': ['Simone de Mazara',
'Artalem de Alagona',
'Apoca',
'Coram',
'Pero de Naso',
'Pero Caruana',
'Bartholomeo Xacara',
'Testamur',
'Artalis de Alagona',
'Melite',
'Simonis de Mazara',
'Simonem',
'Simone',
'Mariano',
'Artalis',
'Artalem',
'Simoni',
'Panormi',
'Renunciando']
where the LOCATIONS should be: Panormi, Bruges, Melite and PERSONAL names should be all others except Unde, Apoca, Coram, Testamur, Renunciando which are neither locations nor personal names.
I was thinking of ignoring the labels and do some classification ML algorithm. The problem is that I do not have any training data available and the only possible usable corpus that I think it might be useful is Proiel treebank which labels proper nouns as NE. How would you go with such a problem?
machine-learning classification named-entity-recognition
$endgroup$
bumped to the homepage by Community♦ 27 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
In a situation like this, where you have very little labeled data, it might be worth labeling some manually, either yourself or by hiting someone to do it.
$endgroup$
– Josh Friedlander
Aug 9 '18 at 19:42
add a comment |
$begingroup$
I am working on some Medieval Latin text and was using various methods of NER such as CLTK (Latin Model), Spacy (Multilingual, Italian, Spanish Model) and StanfordNER (Spanish Model). When I used the non-Latin models I used the original Latin text as the translated one was not making any sense.
Fortunately Spacy Multilingual model managed to extract all Persons and Places of the sample documents but with additional words that I am not considering them as Entities. Moreover, the labels are incorrect.
Here is an example output:
'LOC': ['Artali', 'Artalis', 'Bruges', 'Unde'],
'MISC': ['Marianum lu Tignusu'],
'PER': ['Simone de Mazara',
'Artalem de Alagona',
'Apoca',
'Coram',
'Pero de Naso',
'Pero Caruana',
'Bartholomeo Xacara',
'Testamur',
'Artalis de Alagona',
'Melite',
'Simonis de Mazara',
'Simonem',
'Simone',
'Mariano',
'Artalis',
'Artalem',
'Simoni',
'Panormi',
'Renunciando']
where the LOCATIONS should be: Panormi, Bruges, Melite and PERSONAL names should be all others except Unde, Apoca, Coram, Testamur, Renunciando which are neither locations nor personal names.
I was thinking of ignoring the labels and do some classification ML algorithm. The problem is that I do not have any training data available and the only possible usable corpus that I think it might be useful is Proiel treebank which labels proper nouns as NE. How would you go with such a problem?
machine-learning classification named-entity-recognition
$endgroup$
I am working on some Medieval Latin text and was using various methods of NER such as CLTK (Latin Model), Spacy (Multilingual, Italian, Spanish Model) and StanfordNER (Spanish Model). When I used the non-Latin models I used the original Latin text as the translated one was not making any sense.
Fortunately Spacy Multilingual model managed to extract all Persons and Places of the sample documents but with additional words that I am not considering them as Entities. Moreover, the labels are incorrect.
Here is an example output:
'LOC': ['Artali', 'Artalis', 'Bruges', 'Unde'],
'MISC': ['Marianum lu Tignusu'],
'PER': ['Simone de Mazara',
'Artalem de Alagona',
'Apoca',
'Coram',
'Pero de Naso',
'Pero Caruana',
'Bartholomeo Xacara',
'Testamur',
'Artalis de Alagona',
'Melite',
'Simonis de Mazara',
'Simonem',
'Simone',
'Mariano',
'Artalis',
'Artalem',
'Simoni',
'Panormi',
'Renunciando']
where the LOCATIONS should be: Panormi, Bruges, Melite and PERSONAL names should be all others except Unde, Apoca, Coram, Testamur, Renunciando which are neither locations nor personal names.
I was thinking of ignoring the labels and do some classification ML algorithm. The problem is that I do not have any training data available and the only possible usable corpus that I think it might be useful is Proiel treebank which labels proper nouns as NE. How would you go with such a problem?
machine-learning classification named-entity-recognition
machine-learning classification named-entity-recognition
edited Oct 17 '18 at 6:46
Jero Machuca
31
31
asked Aug 8 '18 at 11:05
CharleneCharlene
261
261
bumped to the homepage by Community♦ 27 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 27 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
In a situation like this, where you have very little labeled data, it might be worth labeling some manually, either yourself or by hiting someone to do it.
$endgroup$
– Josh Friedlander
Aug 9 '18 at 19:42
add a comment |
$begingroup$
In a situation like this, where you have very little labeled data, it might be worth labeling some manually, either yourself or by hiting someone to do it.
$endgroup$
– Josh Friedlander
Aug 9 '18 at 19:42
$begingroup$
In a situation like this, where you have very little labeled data, it might be worth labeling some manually, either yourself or by hiting someone to do it.
$endgroup$
– Josh Friedlander
Aug 9 '18 at 19:42
$begingroup$
In a situation like this, where you have very little labeled data, it might be worth labeling some manually, either yourself or by hiting someone to do it.
$endgroup$
– Josh Friedlander
Aug 9 '18 at 19:42
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
One approach you can take is Multi-Task Learning. This approach is a little more complicated but tackles your problem in hand.
The idea is that you train a neural network to perform different NLP tasks. For example:
- Translation
- Part-of-speech tagging
- Named entity recognition
- Chunking
or any other task.
However, you might have data for translation (e.g. Spanish to Latin, or English to Latin), you have data for NER in Spanish and English but not for NER in Latin.
The theory and studies behind Multi-Task learning suggest that if you learn a task with that language, translation for example, you can get better at other tasks as well, NER for example.
So you can learn NER in Latin by learning NER in other languages and learning translation, chunking and POS tagging.
This should take care of your lack of NER data in Latin.
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f36632%2fimprove-ner-label-results-on-non-english-text%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
One approach you can take is Multi-Task Learning. This approach is a little more complicated but tackles your problem in hand.
The idea is that you train a neural network to perform different NLP tasks. For example:
- Translation
- Part-of-speech tagging
- Named entity recognition
- Chunking
or any other task.
However, you might have data for translation (e.g. Spanish to Latin, or English to Latin), you have data for NER in Spanish and English but not for NER in Latin.
The theory and studies behind Multi-Task learning suggest that if you learn a task with that language, translation for example, you can get better at other tasks as well, NER for example.
So you can learn NER in Latin by learning NER in other languages and learning translation, chunking and POS tagging.
This should take care of your lack of NER data in Latin.
$endgroup$
add a comment |
$begingroup$
One approach you can take is Multi-Task Learning. This approach is a little more complicated but tackles your problem in hand.
The idea is that you train a neural network to perform different NLP tasks. For example:
- Translation
- Part-of-speech tagging
- Named entity recognition
- Chunking
or any other task.
However, you might have data for translation (e.g. Spanish to Latin, or English to Latin), you have data for NER in Spanish and English but not for NER in Latin.
The theory and studies behind Multi-Task learning suggest that if you learn a task with that language, translation for example, you can get better at other tasks as well, NER for example.
So you can learn NER in Latin by learning NER in other languages and learning translation, chunking and POS tagging.
This should take care of your lack of NER data in Latin.
$endgroup$
add a comment |
$begingroup$
One approach you can take is Multi-Task Learning. This approach is a little more complicated but tackles your problem in hand.
The idea is that you train a neural network to perform different NLP tasks. For example:
- Translation
- Part-of-speech tagging
- Named entity recognition
- Chunking
or any other task.
However, you might have data for translation (e.g. Spanish to Latin, or English to Latin), you have data for NER in Spanish and English but not for NER in Latin.
The theory and studies behind Multi-Task learning suggest that if you learn a task with that language, translation for example, you can get better at other tasks as well, NER for example.
So you can learn NER in Latin by learning NER in other languages and learning translation, chunking and POS tagging.
This should take care of your lack of NER data in Latin.
$endgroup$
One approach you can take is Multi-Task Learning. This approach is a little more complicated but tackles your problem in hand.
The idea is that you train a neural network to perform different NLP tasks. For example:
- Translation
- Part-of-speech tagging
- Named entity recognition
- Chunking
or any other task.
However, you might have data for translation (e.g. Spanish to Latin, or English to Latin), you have data for NER in Spanish and English but not for NER in Latin.
The theory and studies behind Multi-Task learning suggest that if you learn a task with that language, translation for example, you can get better at other tasks as well, NER for example.
So you can learn NER in Latin by learning NER in other languages and learning translation, chunking and POS tagging.
This should take care of your lack of NER data in Latin.
answered Aug 10 '18 at 16:13
BrunoGLBrunoGL
1,046222
1,046222
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f36632%2fimprove-ner-label-results-on-non-english-text%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
In a situation like this, where you have very little labeled data, it might be worth labeling some manually, either yourself or by hiting someone to do it.
$endgroup$
– Josh Friedlander
Aug 9 '18 at 19:42