Having trouble figuring out how loss was calculated for SQuAD task in BERT paper Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) 2019 Moderator Election Q&A - Questionnaire 2019 Community Moderator Election ResultsWhere does the sum of squared errors function in neural networks come from?Maximum Entropy modelling - likelihood equationDifference between mathematical and Tensorflow implementation of Softmax Crossentropy with logitHow to use a cross entropy loss function for each letter/digit in a captcha?Loss function for an RNN used for binary classificationHow the combination of cross entropy loss and gradient descent penalizes and rewardsword2vec - log in the objective softmax functionIncrementally Train BERT with minimum QnA records - to get improved resultsHow does GlobalMaxPooling work on the output of Conv1D?Policy gradient/REINFORCE algorithm with RNN: why does this converge with SGM but not Adam?
If Windows 7 doesn't support WSL, then what is "Subsystem for UNIX-based Applications"?
Google .dev domain strangely redirects to https
What is an "asse" in Elizabethan English?
preposition before coffee
What does this say in Elvish?
Would it be easier to apply for a UK visa if there is a host family to sponsor for you in going there?
Dyck paths with extra diagonals from valleys (Laser construction)
The Nth Gryphon Number
Converted a Scalar function to a TVF function for parallel execution-Still running in Serial mode
What is the chair depicted in Cesare Maccari's 1889 painting "Cicerone denuncia Catilina"?
How long can equipment go unused before powering up runs the risk of damage?
Is it possible for SQL statements to execute concurrently within a single session in SQL Server?
A letter with no particular backstory
How did Fremen produce and carry enough thumpers to use Sandworms as de facto Ubers?
Karn the great creator - 'card from outside the game' in sealed
Random body shuffle every night—can we still function?
An adverb for when you're not exaggerating
Is there hard evidence that the grant peer review system performs significantly better than random?
Amount of permutations on an NxNxN Rubik's Cube
Why do early math courses focus on the cross sections of a cone and not on other 3D objects?
Do I really need to have a message in a novel to appeal to readers?
How can I set the aperture on my DSLR when it's attached to a telescope instead of a lens?
What does it mean that physics no longer uses mechanical models to describe phenomena?
How would a mousetrap for use in space work?
Having trouble figuring out how loss was calculated for SQuAD task in BERT paper
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern)
2019 Moderator Election Q&A - Questionnaire
2019 Community Moderator Election ResultsWhere does the sum of squared errors function in neural networks come from?Maximum Entropy modelling - likelihood equationDifference between mathematical and Tensorflow implementation of Softmax Crossentropy with logitHow to use a cross entropy loss function for each letter/digit in a captcha?Loss function for an RNN used for binary classificationHow the combination of cross entropy loss and gradient descent penalizes and rewardsword2vec - log in the objective softmax functionIncrementally Train BERT with minimum QnA records - to get improved resultsHow does GlobalMaxPooling work on the output of Conv1D?Policy gradient/REINFORCE algorithm with RNN: why does this converge with SGM but not Adam?
$begingroup$
The BERT Paper
https://arxiv.org/pdf/1810.04805.pdf
Section 4.2 covers the SQuAD training.
So from my understanding, there are two extra parameters trained, they are two vectors with the same dimension as the hidden size, so the same dimensions as the contextualized embeddings in BERT. They are S (for start) and E (for End).
For each, a softmax is taken with S and each of the final contextualized embeddings to get a score for the correct Start position. And the same thing is done for E and the correct end position.
I get up to this part. But I am having trouble figuring out how the did the labeling and final loss calculations, which is described in this paragraph
"and the maximum scoring span is used as the prediction. The training objective is the loglikelihood of the correct start and end positions."
What do they mean by "maximum scoring span is used as the prediction"?
Furthermore, how does that play into "The training objective is the loglikelihood of the correct start and end positions"?
From this Source:
https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/
It says the log-likelihood is only applied to the correct classes. So the we are only calculating the softmax for the correct positions only, Not any of the in correct positions.
If this interpretation is correct, then the loss will be
Loss = -Log( Softmax(S*T(predictedStart) / Sum(S*Ti) ) -Log( Softmax(E*T(predictedEnd) / Sum(S*Ti) )
machine-learning nlp loss-function
asked 34 mins ago
SantoshGupta7SantoshGupta7
1134
$endgroup$
add a comment |
$begingroup$
The BERT Paper
https://arxiv.org/pdf/1810.04805.pdf
Section 4.2 covers the SQuAD training.
So from my understanding, there are two extra parameters trained, they are two vectors with the same dimension as the hidden size, so the same dimensions as the contextualized embeddings in BERT. They are S (for start) and E (for End).
For each, a softmax is taken with S and each of the final contextualized embeddings to get a score for the correct Start position. And the same thing is done for E and the correct end position.
I get up to this part. But I am having trouble figuring out how the did the labeling and final loss calculations, which is described in this paragraph
"and the maximum scoring span is used as the prediction. The training objective is the loglikelihood of the correct start and end positions."
What do they mean by "maximum scoring span is used as the prediction"?
Furthermore, how does that play into "The training objective is the loglikelihood of the correct start and end positions"?
From this Source:
https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/
It says the log-likelihood is only applied to the correct classes. So the we are only calculating the softmax for the correct positions only, Not any of the in correct positions.
If this interpretation is correct, then the loss will be
Loss = -Log( Softmax(S*T(predictedStart) / Sum(S*Ti) ) -Log( Softmax(E*T(predictedEnd) / Sum(S*Ti) )
machine-learning nlp loss-function
asked 34 mins ago
SantoshGupta7SantoshGupta7
1134
$endgroup$
add a comment |
$begingroup$
The BERT Paper
https://arxiv.org/pdf/1810.04805.pdf
Section 4.2 covers the SQuAD training.
So from my understanding, there are two extra parameters trained, they are two vectors with the same dimension as the hidden size, so the same dimensions as the contextualized embeddings in BERT. They are S (for start) and E (for End).
For each, a softmax is taken with S and each of the final contextualized embeddings to get a score for the correct Start position. And the same thing is done for E and the correct end position.
I get up to this part. But I am having trouble figuring out how the did the labeling and final loss calculations, which is described in this paragraph
"and the maximum scoring span is used as the prediction. The training objective is the loglikelihood of the correct start and end positions."
What do they mean by "maximum scoring span is used as the prediction"?
Furthermore, how does that play into "The training objective is the loglikelihood of the correct start and end positions"?
From this Source:
https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/
It says the log-likelihood is only applied to the correct classes. So the we are only calculating the softmax for the correct positions only, Not any of the in correct positions.
If this interpretation is correct, then the loss will be
Loss = -Log( Softmax(S*T(predictedStart) / Sum(S*Ti) ) -Log( Softmax(E*T(predictedEnd) / Sum(S*Ti) )
machine-learning nlp loss-function
asked 34 mins ago
SantoshGupta7SantoshGupta7
1134
$endgroup$
The BERT Paper
https://arxiv.org/pdf/1810.04805.pdf
Section 4.2 covers the SQuAD training.
So from my understanding, there are two extra parameters trained, they are two vectors with the same dimension as the hidden size, so the same dimensions as the contextualized embeddings in BERT. They are S (for start) and E (for End).
For each, a softmax is taken with S and each of the final contextualized embeddings to get a score for the correct Start position. And the same thing is done for E and the correct end position.
I get up to this part. But I am having trouble figuring out how the did the labeling and final loss calculations, which is described in this paragraph
"and the maximum scoring span is used as the prediction. The training objective is the loglikelihood of the correct start and end positions."
What do they mean by "maximum scoring span is used as the prediction"?
Furthermore, how does that play into "The training objective is the loglikelihood of the correct start and end positions"?
From this Source:
https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/
It says the log-likelihood is only applied to the correct classes. So the we are only calculating the softmax for the correct positions only, Not any of the in correct positions.
If this interpretation is correct, then the loss will be
Loss = -Log( Softmax(S*T(predictedStart) / Sum(S*Ti) ) -Log( Softmax(E*T(predictedEnd) / Sum(S*Ti) )
machine-learning nlp loss-function
machine-learning nlp loss-function
asked 34 mins ago
SantoshGupta7SantoshGupta7
1134
asked 34 mins ago
SantoshGupta7SantoshGupta7
1134
asked 34 mins ago
SantoshGupta7SantoshGupta7
1134
asked 34 mins ago
SantoshGupta7SantoshGupta7
1134
asked 34 mins ago
SantoshGupta7SantoshGupta7
1134
1134
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f49621%2fhaving-trouble-figuring-out-how-loss-was-calculated-for-squad-task-in-bert-paper%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f49621%2fhaving-trouble-figuring-out-how-loss-was-calculated-for-squad-task-in-bert-paper%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
