Unable to understand Kernel Ridge regression Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) 2019 Moderator Election Q&A - Questionnaire 2019 Community Moderator Election ResultsWhat does the Ip mean in the Bayesian Ridge Regression formula?Regression in KerasWhat exactly is a Gini IndexLasso Regression DoubtWhat is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms?Neural Network unable to track training dataTimestamps in Ridge Regression Scikit LearnCost function in linear regressionHow can someone avoid over fitting or data leak in ridge and lasso regression when the training score is high and test score is low?Basis expansion for regression using neural network?
What does 丫 mean? 丫是什么意思？
Any stored/leased 737s that could substitute for grounded MAXs?
Adapting the Chinese Remainder Theorem (CRT) for integers to polynomials
Why not use the yoke to control yaw, as well as pitch and roll?
Baking rewards as operations
New Order #6: Easter Egg
How to achieve cat-like agility?
Did John Wesley plagiarize Matthew Henry...?
Should man-made satellites feature an intelligent inverted "cow catcher"?
"Destructive power" carried by a B-52?
Understanding piped commands in GNU/Linux
Was the pager message from Nick Fury to Captain Marvel unnecessary?
Keep at all times, the minus sign above aligned with minus sign below
Does a random sequence of vectors span a Hilbert space?
Noise in Eigenvalues plot
How can I list files in reverse time order by a command and pass them as arguments to another command?
Where and when has Thucydides been studied?
Did any compiler fully use 80-bit floating point?
Pointing to problems without suggesting solutions
Why are current probes so expensive?
Why is there so little support for joining EFTA in the British parliament?
The Nth Gryphon Number
Is this Kuo-toa homebrew race balanced?
In musical terms, what properties are varied by the human voice to produce different words / syllables?
Unable to understand Kernel Ridge regression
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern)
2019 Moderator Election Q&A - Questionnaire
2019 Community Moderator Election ResultsWhat does the Ip mean in the Bayesian Ridge Regression formula?Regression in KerasWhat exactly is a Gini IndexLasso Regression DoubtWhat is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms?Neural Network unable to track training dataTimestamps in Ridge Regression Scikit LearnCost function in linear regressionHow can someone avoid over fitting or data leak in ridge and lasso regression when the training score is high and test score is low?Basis expansion for regression using neural network?
$begingroup$
I am trying to read kernel ridge regression from this link
But , I am unable to get the intution behind the derivation.
Can anyone please help me ?
machine-learning regression kernel
asked May 21 '18 at 8:38
DukeLoverDukeLover
767
$endgroup$
bumped to the homepage by Community♦ 2 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I am trying to read kernel ridge regression from this link
But , I am unable to get the intution behind the derivation.
Can anyone please help me ?
machine-learning regression kernel
asked May 21 '18 at 8:38
DukeLoverDukeLover
767
$endgroup$
bumped to the homepage by Community♦ 2 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Which part, specifically, is unclear?
$endgroup$
– marco_gorelli
May 21 '18 at 8:45
$begingroup$
hi @Lupacante , thanks for your reply. I am specifically unclear about the equation (4) and henceforth.
$endgroup$
– DukeLover
May 21 '18 at 11:42
add a comment |
$begingroup$
I am trying to read kernel ridge regression from this link
But , I am unable to get the intution behind the derivation.
Can anyone please help me ?
machine-learning regression kernel
asked May 21 '18 at 8:38
DukeLoverDukeLover
767
$endgroup$
I am trying to read kernel ridge regression from this link
But , I am unable to get the intution behind the derivation.
Can anyone please help me ?
machine-learning regression kernel
machine-learning regression kernel
asked May 21 '18 at 8:38
DukeLoverDukeLover
767
asked May 21 '18 at 8:38
DukeLoverDukeLover
767
asked May 21 '18 at 8:38
DukeLoverDukeLover
767
asked May 21 '18 at 8:38
DukeLoverDukeLover
767
asked May 21 '18 at 8:38
DukeLoverDukeLover
767
767
bumped to the homepage by Community♦ 2 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 2 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Which part, specifically, is unclear?
$endgroup$
– marco_gorelli
May 21 '18 at 8:45
$begingroup$
hi @Lupacante , thanks for your reply. I am specifically unclear about the equation (4) and henceforth.
$endgroup$
– DukeLover
May 21 '18 at 11:42
add a comment |
$begingroup$
Which part, specifically, is unclear?
$endgroup$
– marco_gorelli
May 21 '18 at 8:45
$begingroup$
hi @Lupacante , thanks for your reply. I am specifically unclear about the equation (4) and henceforth.
$endgroup$
– DukeLover
May 21 '18 at 11:42
$begingroup$
Which part, specifically, is unclear?
$endgroup$
– marco_gorelli
May 21 '18 at 8:45
$begingroup$
Which part, specifically, is unclear?
$endgroup$
– marco_gorelli
May 21 '18 at 8:45
$begingroup$
hi @Lupacante , thanks for your reply. I am specifically unclear about the equation (4) and henceforth.
$endgroup$
– DukeLover
May 21 '18 at 11:42
$begingroup$
hi @Lupacante , thanks for your reply. I am specifically unclear about the equation (4) and henceforth.
$endgroup$
– DukeLover
May 21 '18 at 11:42
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
Equation (4) simply gives us an identity (without proof):
$$(P^-1 + B^TR^-1B)^-1B^TR^-1 = PB^T (BPB^T + R)^-1.$$
Let's check that it does indeed hold. The left hand side equals
beginalign*
(P^-1 + B^TR^-1B)^-1B^TR^-1 &= left(left((P^-1 + B^TR^-1B)^-1B^TR^-1right)^-1right)^-1\
&= left(RB^-T(P^-1 + B^TR^-1B)right)^-1\
&= left(RB^-TP^-1 + Bright)^-1.
endalign*
The right hand side equals
beginalign*
PB^T (BPB^T + R)^-1 &= left(left(PB^T (BPB^T + R)^-1right)^-1right)^-1\
&= left((BPB^T + R)B^-TP^-1 right)^-1\
&= left(B + RB^-TP^-1 right)^-1.
endalign*
So indeed, the equality holds.
If you take the right hand side of equation (3)
$$left(lambdamathbfI + sum_i mathbfx_i mathbfx_i^Tright)^-1left(sum_j y_j mathbfx_jright),$$
replace $mathbfx_i$ with $phi_i$ and rewrite it as
$$left(lambdamathbfI + PhiPhi^Tright)^-1left(Phimathbfyright),$$
then it matches the left hand side of the identity in equation (4) (right-multiplied by $mathbfy$), where
$$P^-1 = lambda^-1mathbfI$$
$$B^T = Phi$$
$$R = mathbfI.$$
Therefore, we can rewrite the right hand side of equation (4) as
$$lambda^-1mathbfIPhi(Philambda^-1Phi + mathbfI)^-1mathbfy.$$ As $lambda$ is a scalar, we can rewrite this as
$$Phi(PhiPhi + lambdamathbfI)^-1mathbfy.$$
answered May 21 '18 at 12:32
marco_gorellimarco_gorelli
4819
$endgroup$
$begingroup$
Hi , I am not able to understand from where this identity is coming ?
$endgroup$
– DukeLover
May 21 '18 at 13:28
$begingroup$
@DukeLover I've edited my answer accordingly - is it clear now?
$endgroup$
– marco_gorelli
May 22 '18 at 8:30
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f31908%2funable-to-understand-kernel-ridge-regression%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Equation (4) simply gives us an identity (without proof):
$$(P^-1 + B^TR^-1B)^-1B^TR^-1 = PB^T (BPB^T + R)^-1.$$
Let's check that it does indeed hold. The left hand side equals
beginalign*
(P^-1 + B^TR^-1B)^-1B^TR^-1 &= left(left((P^-1 + B^TR^-1B)^-1B^TR^-1right)^-1right)^-1\
&= left(RB^-T(P^-1 + B^TR^-1B)right)^-1\
&= left(RB^-TP^-1 + Bright)^-1.
endalign*
The right hand side equals
beginalign*
PB^T (BPB^T + R)^-1 &= left(left(PB^T (BPB^T + R)^-1right)^-1right)^-1\
&= left((BPB^T + R)B^-TP^-1 right)^-1\
&= left(B + RB^-TP^-1 right)^-1.
endalign*
So indeed, the equality holds.
If you take the right hand side of equation (3)
$$left(lambdamathbfI + sum_i mathbfx_i mathbfx_i^Tright)^-1left(sum_j y_j mathbfx_jright),$$
replace $mathbfx_i$ with $phi_i$ and rewrite it as
$$left(lambdamathbfI + PhiPhi^Tright)^-1left(Phimathbfyright),$$
then it matches the left hand side of the identity in equation (4) (right-multiplied by $mathbfy$), where
$$P^-1 = lambda^-1mathbfI$$
$$B^T = Phi$$
$$R = mathbfI.$$
Therefore, we can rewrite the right hand side of equation (4) as
$$lambda^-1mathbfIPhi(Philambda^-1Phi + mathbfI)^-1mathbfy.$$ As $lambda$ is a scalar, we can rewrite this as
$$Phi(PhiPhi + lambdamathbfI)^-1mathbfy.$$
answered May 21 '18 at 12:32
marco_gorellimarco_gorelli
4819
$endgroup$
$begingroup$
Hi , I am not able to understand from where this identity is coming ?
$endgroup$
– DukeLover
May 21 '18 at 13:28
$begingroup$
@DukeLover I've edited my answer accordingly - is it clear now?
$endgroup$
– marco_gorelli
May 22 '18 at 8:30
add a comment |
$begingroup$
Equation (4) simply gives us an identity (without proof):
$$(P^-1 + B^TR^-1B)^-1B^TR^-1 = PB^T (BPB^T + R)^-1.$$
Let's check that it does indeed hold. The left hand side equals
beginalign*
(P^-1 + B^TR^-1B)^-1B^TR^-1 &= left(left((P^-1 + B^TR^-1B)^-1B^TR^-1right)^-1right)^-1\
&= left(RB^-T(P^-1 + B^TR^-1B)right)^-1\
&= left(RB^-TP^-1 + Bright)^-1.
endalign*
The right hand side equals
beginalign*
PB^T (BPB^T + R)^-1 &= left(left(PB^T (BPB^T + R)^-1right)^-1right)^-1\
&= left((BPB^T + R)B^-TP^-1 right)^-1\
&= left(B + RB^-TP^-1 right)^-1.
endalign*
So indeed, the equality holds.
If you take the right hand side of equation (3)
$$left(lambdamathbfI + sum_i mathbfx_i mathbfx_i^Tright)^-1left(sum_j y_j mathbfx_jright),$$
replace $mathbfx_i$ with $phi_i$ and rewrite it as
$$left(lambdamathbfI + PhiPhi^Tright)^-1left(Phimathbfyright),$$
then it matches the left hand side of the identity in equation (4) (right-multiplied by $mathbfy$), where
$$P^-1 = lambda^-1mathbfI$$
$$B^T = Phi$$
$$R = mathbfI.$$
Therefore, we can rewrite the right hand side of equation (4) as
$$lambda^-1mathbfIPhi(Philambda^-1Phi + mathbfI)^-1mathbfy.$$ As $lambda$ is a scalar, we can rewrite this as
$$Phi(PhiPhi + lambdamathbfI)^-1mathbfy.$$
answered May 21 '18 at 12:32
marco_gorellimarco_gorelli
4819
$endgroup$
$begingroup$
Hi , I am not able to understand from where this identity is coming ?
$endgroup$
– DukeLover
May 21 '18 at 13:28
$begingroup$
@DukeLover I've edited my answer accordingly - is it clear now?
$endgroup$
– marco_gorelli
May 22 '18 at 8:30
add a comment |
$begingroup$
Equation (4) simply gives us an identity (without proof):
$$(P^-1 + B^TR^-1B)^-1B^TR^-1 = PB^T (BPB^T + R)^-1.$$
Let's check that it does indeed hold. The left hand side equals
beginalign*
(P^-1 + B^TR^-1B)^-1B^TR^-1 &= left(left((P^-1 + B^TR^-1B)^-1B^TR^-1right)^-1right)^-1\
&= left(RB^-T(P^-1 + B^TR^-1B)right)^-1\
&= left(RB^-TP^-1 + Bright)^-1.
endalign*
The right hand side equals
beginalign*
PB^T (BPB^T + R)^-1 &= left(left(PB^T (BPB^T + R)^-1right)^-1right)^-1\
&= left((BPB^T + R)B^-TP^-1 right)^-1\
&= left(B + RB^-TP^-1 right)^-1.
endalign*
So indeed, the equality holds.
If you take the right hand side of equation (3)
$$left(lambdamathbfI + sum_i mathbfx_i mathbfx_i^Tright)^-1left(sum_j y_j mathbfx_jright),$$
replace $mathbfx_i$ with $phi_i$ and rewrite it as
$$left(lambdamathbfI + PhiPhi^Tright)^-1left(Phimathbfyright),$$
then it matches the left hand side of the identity in equation (4) (right-multiplied by $mathbfy$), where
$$P^-1 = lambda^-1mathbfI$$
$$B^T = Phi$$
$$R = mathbfI.$$
Therefore, we can rewrite the right hand side of equation (4) as
$$lambda^-1mathbfIPhi(Philambda^-1Phi + mathbfI)^-1mathbfy.$$ As $lambda$ is a scalar, we can rewrite this as
$$Phi(PhiPhi + lambdamathbfI)^-1mathbfy.$$
answered May 21 '18 at 12:32
marco_gorellimarco_gorelli
4819
$endgroup$
Equation (4) simply gives us an identity (without proof):
$$(P^-1 + B^TR^-1B)^-1B^TR^-1 = PB^T (BPB^T + R)^-1.$$
Let's check that it does indeed hold. The left hand side equals
beginalign*
(P^-1 + B^TR^-1B)^-1B^TR^-1 &= left(left((P^-1 + B^TR^-1B)^-1B^TR^-1right)^-1right)^-1\
&= left(RB^-T(P^-1 + B^TR^-1B)right)^-1\
&= left(RB^-TP^-1 + Bright)^-1.
endalign*
The right hand side equals
beginalign*
PB^T (BPB^T + R)^-1 &= left(left(PB^T (BPB^T + R)^-1right)^-1right)^-1\
&= left((BPB^T + R)B^-TP^-1 right)^-1\
&= left(B + RB^-TP^-1 right)^-1.
endalign*
So indeed, the equality holds.
If you take the right hand side of equation (3)
$$left(lambdamathbfI + sum_i mathbfx_i mathbfx_i^Tright)^-1left(sum_j y_j mathbfx_jright),$$
replace $mathbfx_i$ with $phi_i$ and rewrite it as
$$left(lambdamathbfI + PhiPhi^Tright)^-1left(Phimathbfyright),$$
then it matches the left hand side of the identity in equation (4) (right-multiplied by $mathbfy$), where
$$P^-1 = lambda^-1mathbfI$$
$$B^T = Phi$$
$$R = mathbfI.$$
Therefore, we can rewrite the right hand side of equation (4) as
$$lambda^-1mathbfIPhi(Philambda^-1Phi + mathbfI)^-1mathbfy.$$ As $lambda$ is a scalar, we can rewrite this as
$$Phi(PhiPhi + lambdamathbfI)^-1mathbfy.$$
answered May 21 '18 at 12:32
marco_gorellimarco_gorelli
4819
edited May 22 '18 at 8:29
answered May 21 '18 at 12:32
marco_gorellimarco_gorelli
4819
answered May 21 '18 at 12:32
marco_gorellimarco_gorelli
4819
answered May 21 '18 at 12:32
marco_gorellimarco_gorelli
4819
4819
$begingroup$
Hi , I am not able to understand from where this identity is coming ?
$endgroup$
– DukeLover
May 21 '18 at 13:28
$begingroup$
@DukeLover I've edited my answer accordingly - is it clear now?
$endgroup$
– marco_gorelli
May 22 '18 at 8:30
add a comment |
$begingroup$
Hi , I am not able to understand from where this identity is coming ?
$endgroup$
– DukeLover
May 21 '18 at 13:28
$begingroup$
@DukeLover I've edited my answer accordingly - is it clear now?
$endgroup$
– marco_gorelli
May 22 '18 at 8:30
$begingroup$
Hi , I am not able to understand from where this identity is coming ?
$endgroup$
– DukeLover
May 21 '18 at 13:28
$begingroup$
Hi , I am not able to understand from where this identity is coming ?
$endgroup$
– DukeLover
May 21 '18 at 13:28
$begingroup$
@DukeLover I've edited my answer accordingly - is it clear now?
$endgroup$
– marco_gorelli
May 22 '18 at 8:30
$begingroup$
@DukeLover I've edited my answer accordingly - is it clear now?
$endgroup$
– marco_gorelli
May 22 '18 at 8:30
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f31908%2funable-to-understand-kernel-ridge-regression%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Which part, specifically, is unclear?
$endgroup$
– marco_gorelli
May 21 '18 at 8:45
$begingroup$
hi @Lupacante , thanks for your reply. I am specifically unclear about the equation (4) and henceforth.
$endgroup$
– DukeLover
May 21 '18 at 11:42