Time horizon T in policy gradients (actor-critic) Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) 2019 Moderator Election Q&A - Questionnaire 2019 Community Moderator Election ResultsWhat is “experience replay” and what are its benefits?What is the Q function and what is the V function in reinforcement learning?Why do we normalize the discounted rewards when doing policy gradient reinforcement learning?Reinforcement Learning - What's the formula for the value functionWhat is the difference between “expected return” and “expected reward” in the context of RL?How is Importance-Sampling Used in Off-Policy Monte Carlo Prediction?Deep RL: Proximal policy optimization gradient calculationPolicy Gradients - gradient Log probabilities favor less likely actions?RL's policy gradient (REINFORCE) pipeline clarificationNatural actor-critic with Q function approximation
What is the chair depicted in Cesare Maccari's 1889 painting "Cicerone denuncia Catilina"?
Universal covering space of the real projective line?
Why not send Voyager 3 and 4 following up the paths taken by Voyager 1 and 2 to re-transmit signals of later as they fly away from Earth?
What does it mean that physics no longer uses mechanical models to describe phenomena?
Is there public access to the Meteor Crater in Arizona?
Is there hard evidence that the grant peer review system performs significantly better than random?
After Sam didn't return home in the end, were he and Al still friends?
What are the main differences between Stargate SG-1 cuts?
Google .dev domain strangely redirects to https
How can a team of shapeshifters communicate?
Trying to understand entropy as a novice in thermodynamics
Does the Black Tentacles spell do damage twice at the start of turn to an already restrained creature?
Mounting TV on a weird wall that has some material between the drywall and stud
Why is std::move not [[nodiscard]] in C++20?
Simple Http Server
NERDTreeMenu Remapping
Why complex landing gears are used instead of simple,reliability and light weight muscle wire or shape memory alloys?
Tips to organize LaTeX presentations for a semester
Should a wizard buy fine inks every time he want to copy spells into his spellbook?
AppleTVs create a chatty alternate WiFi network
I can't produce songs
How does light 'choose' between wave and particle behaviour?
What does 丫 mean? 丫是什么意思？
Does silver oxide react with hydrogen sulfide?
Time horizon T in policy gradients (actor-critic)
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern)
2019 Moderator Election Q&A - Questionnaire
2019 Community Moderator Election ResultsWhat is “experience replay” and what are its benefits?What is the Q function and what is the V function in reinforcement learning?Why do we normalize the discounted rewards when doing policy gradient reinforcement learning?Reinforcement Learning - What's the formula for the value functionWhat is the difference between “expected return” and “expected reward” in the context of RL?How is Importance-Sampling Used in Off-Policy Monte Carlo Prediction?Deep RL: Proximal policy optimization gradient calculationPolicy Gradients - gradient Log probabilities favor less likely actions?RL's policy gradient (REINFORCE) pipeline clarificationNatural actor-critic with Q function approximation
$begingroup$
I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture.
At the bottom of that slide, the gradient of the expected sum of rewards function is given by
$$
nabla J(theta) = frac1N sum_i=1^N sum_t=1^T nabla_theta logpi_theta(a_i,t vert s_i,t) (Q(s_i,t,a_i,t) - V(s_i,t))
$$
The q-value function is defined as
$$Q(s_t,a_t) = sum_t'=t^T mathbbE_pi_theta[r(s_t',a_t')vert s_t,a_t]$$
At first glance, this makes sense, because I compare the value of taking the chosen action $a_i,t$ to the average value in time step $t$ and can evaluate how good my action was.
My question is: a specific state $s_spec$ can occur in any timestep, for example, $s_1 = s_spec = s_10$. But isn't there a difference in value depending on whether I hit $s_spec$ at timestep 1 or 10 when $T$ is fixed? Does this mean that for every state there is a different q value for each possible $t in 0,ldots,T$? I somehow doubt that this is the case, but I don't quite understand how the time horizon $T$ fits in.
Or is $T$ not fixed (perhaps it's defined as the time step in which the trajectory ends in a terminal state - but that'd mean that during trajectory sampling, each simulation would take a different number of timesteps)?
machine-learning deep-learning reinforcement-learning policy-gradients actor-critic
asked Aug 28 '18 at 14:56
Dummie VariableDummie Variable
362
$endgroup$
bumped to the homepage by Community♦ 27 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture.
At the bottom of that slide, the gradient of the expected sum of rewards function is given by
$$
nabla J(theta) = frac1N sum_i=1^N sum_t=1^T nabla_theta logpi_theta(a_i,t vert s_i,t) (Q(s_i,t,a_i,t) - V(s_i,t))
$$
The q-value function is defined as
$$Q(s_t,a_t) = sum_t'=t^T mathbbE_pi_theta[r(s_t',a_t')vert s_t,a_t]$$
At first glance, this makes sense, because I compare the value of taking the chosen action $a_i,t$ to the average value in time step $t$ and can evaluate how good my action was.
My question is: a specific state $s_spec$ can occur in any timestep, for example, $s_1 = s_spec = s_10$. But isn't there a difference in value depending on whether I hit $s_spec$ at timestep 1 or 10 when $T$ is fixed? Does this mean that for every state there is a different q value for each possible $t in 0,ldots,T$? I somehow doubt that this is the case, but I don't quite understand how the time horizon $T$ fits in.
Or is $T$ not fixed (perhaps it's defined as the time step in which the trajectory ends in a terminal state - but that'd mean that during trajectory sampling, each simulation would take a different number of timesteps)?
machine-learning deep-learning reinforcement-learning policy-gradients actor-critic
asked Aug 28 '18 at 14:56
Dummie VariableDummie Variable
362
$endgroup$
bumped to the homepage by Community♦ 27 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
I think I have found something helpful - John Schulman notes in his thesis: !screenshot So it sounds like either I sample trajectories with variable length (arrival in a terminal state) or I encode the time step into the state and the state value function will also take the time step into consideration?
$endgroup$
– Dummie Variable
Sep 15 '18 at 13:49
add a comment |
$begingroup$
I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture.
At the bottom of that slide, the gradient of the expected sum of rewards function is given by
$$
nabla J(theta) = frac1N sum_i=1^N sum_t=1^T nabla_theta logpi_theta(a_i,t vert s_i,t) (Q(s_i,t,a_i,t) - V(s_i,t))
$$
The q-value function is defined as
$$Q(s_t,a_t) = sum_t'=t^T mathbbE_pi_theta[r(s_t',a_t')vert s_t,a_t]$$
At first glance, this makes sense, because I compare the value of taking the chosen action $a_i,t$ to the average value in time step $t$ and can evaluate how good my action was.
My question is: a specific state $s_spec$ can occur in any timestep, for example, $s_1 = s_spec = s_10$. But isn't there a difference in value depending on whether I hit $s_spec$ at timestep 1 or 10 when $T$ is fixed? Does this mean that for every state there is a different q value for each possible $t in 0,ldots,T$? I somehow doubt that this is the case, but I don't quite understand how the time horizon $T$ fits in.
Or is $T$ not fixed (perhaps it's defined as the time step in which the trajectory ends in a terminal state - but that'd mean that during trajectory sampling, each simulation would take a different number of timesteps)?
machine-learning deep-learning reinforcement-learning policy-gradients actor-critic
asked Aug 28 '18 at 14:56
Dummie VariableDummie Variable
362
$endgroup$
I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture.
At the bottom of that slide, the gradient of the expected sum of rewards function is given by
$$
nabla J(theta) = frac1N sum_i=1^N sum_t=1^T nabla_theta logpi_theta(a_i,t vert s_i,t) (Q(s_i,t,a_i,t) - V(s_i,t))
$$
The q-value function is defined as
$$Q(s_t,a_t) = sum_t'=t^T mathbbE_pi_theta[r(s_t',a_t')vert s_t,a_t]$$
At first glance, this makes sense, because I compare the value of taking the chosen action $a_i,t$ to the average value in time step $t$ and can evaluate how good my action was.
My question is: a specific state $s_spec$ can occur in any timestep, for example, $s_1 = s_spec = s_10$. But isn't there a difference in value depending on whether I hit $s_spec$ at timestep 1 or 10 when $T$ is fixed? Does this mean that for every state there is a different q value for each possible $t in 0,ldots,T$? I somehow doubt that this is the case, but I don't quite understand how the time horizon $T$ fits in.
Or is $T$ not fixed (perhaps it's defined as the time step in which the trajectory ends in a terminal state - but that'd mean that during trajectory sampling, each simulation would take a different number of timesteps)?
machine-learning deep-learning reinforcement-learning policy-gradients actor-critic
machine-learning deep-learning reinforcement-learning policy-gradients actor-critic
asked Aug 28 '18 at 14:56
Dummie VariableDummie Variable
362
asked Aug 28 '18 at 14:56
Dummie VariableDummie Variable
362
edited Aug 29 '18 at 8:16
Dummie Variable
asked Aug 28 '18 at 14:56
Dummie VariableDummie Variable
362
asked Aug 28 '18 at 14:56
Dummie VariableDummie Variable
362
asked Aug 28 '18 at 14:56
Dummie VariableDummie Variable
362
362
bumped to the homepage by Community♦ 27 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 27 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
I think I have found something helpful - John Schulman notes in his thesis: !screenshot So it sounds like either I sample trajectories with variable length (arrival in a terminal state) or I encode the time step into the state and the state value function will also take the time step into consideration?
$endgroup$
– Dummie Variable
Sep 15 '18 at 13:49
add a comment |
$begingroup$
I think I have found something helpful - John Schulman notes in his thesis: !screenshot So it sounds like either I sample trajectories with variable length (arrival in a terminal state) or I encode the time step into the state and the state value function will also take the time step into consideration?
$endgroup$
– Dummie Variable
Sep 15 '18 at 13:49
$begingroup$
I think I have found something helpful - John Schulman notes in his thesis: !screenshot So it sounds like either I sample trajectories with variable length (arrival in a terminal state) or I encode the time step into the state and the state value function will also take the time step into consideration?
$endgroup$
– Dummie Variable
Sep 15 '18 at 13:49
$begingroup$
I think I have found something helpful - John Schulman notes in his thesis: !screenshot So it sounds like either I sample trajectories with variable length (arrival in a terminal state) or I encode the time step into the state and the state value function will also take the time step into consideration?
$endgroup$
– Dummie Variable
Sep 15 '18 at 13:49
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
In this case, I think it doesn't matter when you reach $s_spec$, but how the q-value gets updated because of taking an action at that state.
Therefore there shouldn't be different q-values for each possible $tin 0, ..., T$, only q-values for each possible actions.
I'm sure it does make a difference for being at a state at a specific timestep, but it's the agents job to learn this by using the RL algorithms (like policy gradient method in the lecture).
In regards to $T$ being fixed or not, horizon $T$ can be infinite or fixed to a finite number.
For example, if $T$ is fixed to $10$, the agent should learn a policy that maximizes the total discounted rewards in the finite amount of time, but it may not be the most optimal policy. When $T$ is infinite, there is more time to explore and figure out the most optimal policy.
The closest method I know that takes notice to when the state-action pair was encountered is Experience Replay that is used in DQN.
I'm also learning Reinforcement Learning right now! I recommend Deep RL Bootcamp since they give you labs in Python which are really intuitive.
answered Sep 18 '18 at 15:00
haruishiharuishi
85
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f37518%2ftime-horizon-t-in-policy-gradients-actor-critic%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
In this case, I think it doesn't matter when you reach $s_spec$, but how the q-value gets updated because of taking an action at that state.
Therefore there shouldn't be different q-values for each possible $tin 0, ..., T$, only q-values for each possible actions.
I'm sure it does make a difference for being at a state at a specific timestep, but it's the agents job to learn this by using the RL algorithms (like policy gradient method in the lecture).
In regards to $T$ being fixed or not, horizon $T$ can be infinite or fixed to a finite number.
For example, if $T$ is fixed to $10$, the agent should learn a policy that maximizes the total discounted rewards in the finite amount of time, but it may not be the most optimal policy. When $T$ is infinite, there is more time to explore and figure out the most optimal policy.
The closest method I know that takes notice to when the state-action pair was encountered is Experience Replay that is used in DQN.
I'm also learning Reinforcement Learning right now! I recommend Deep RL Bootcamp since they give you labs in Python which are really intuitive.
answered Sep 18 '18 at 15:00
haruishiharuishi
85
$endgroup$
add a comment |
$begingroup$
In this case, I think it doesn't matter when you reach $s_spec$, but how the q-value gets updated because of taking an action at that state.
Therefore there shouldn't be different q-values for each possible $tin 0, ..., T$, only q-values for each possible actions.
I'm sure it does make a difference for being at a state at a specific timestep, but it's the agents job to learn this by using the RL algorithms (like policy gradient method in the lecture).
In regards to $T$ being fixed or not, horizon $T$ can be infinite or fixed to a finite number.
For example, if $T$ is fixed to $10$, the agent should learn a policy that maximizes the total discounted rewards in the finite amount of time, but it may not be the most optimal policy. When $T$ is infinite, there is more time to explore and figure out the most optimal policy.
The closest method I know that takes notice to when the state-action pair was encountered is Experience Replay that is used in DQN.
I'm also learning Reinforcement Learning right now! I recommend Deep RL Bootcamp since they give you labs in Python which are really intuitive.
answered Sep 18 '18 at 15:00
haruishiharuishi
85
$endgroup$
add a comment |
$begingroup$
In this case, I think it doesn't matter when you reach $s_spec$, but how the q-value gets updated because of taking an action at that state.
Therefore there shouldn't be different q-values for each possible $tin 0, ..., T$, only q-values for each possible actions.
I'm sure it does make a difference for being at a state at a specific timestep, but it's the agents job to learn this by using the RL algorithms (like policy gradient method in the lecture).
In regards to $T$ being fixed or not, horizon $T$ can be infinite or fixed to a finite number.
For example, if $T$ is fixed to $10$, the agent should learn a policy that maximizes the total discounted rewards in the finite amount of time, but it may not be the most optimal policy. When $T$ is infinite, there is more time to explore and figure out the most optimal policy.
The closest method I know that takes notice to when the state-action pair was encountered is Experience Replay that is used in DQN.
I'm also learning Reinforcement Learning right now! I recommend Deep RL Bootcamp since they give you labs in Python which are really intuitive.
answered Sep 18 '18 at 15:00
haruishiharuishi
85
$endgroup$
In this case, I think it doesn't matter when you reach $s_spec$, but how the q-value gets updated because of taking an action at that state.
Therefore there shouldn't be different q-values for each possible $tin 0, ..., T$, only q-values for each possible actions.
I'm sure it does make a difference for being at a state at a specific timestep, but it's the agents job to learn this by using the RL algorithms (like policy gradient method in the lecture).
In regards to $T$ being fixed or not, horizon $T$ can be infinite or fixed to a finite number.
For example, if $T$ is fixed to $10$, the agent should learn a policy that maximizes the total discounted rewards in the finite amount of time, but it may not be the most optimal policy. When $T$ is infinite, there is more time to explore and figure out the most optimal policy.
The closest method I know that takes notice to when the state-action pair was encountered is Experience Replay that is used in DQN.
I'm also learning Reinforcement Learning right now! I recommend Deep RL Bootcamp since they give you labs in Python which are really intuitive.
answered Sep 18 '18 at 15:00
haruishiharuishi
85
edited Sep 18 '18 at 15:10
answered Sep 18 '18 at 15:00
haruishiharuishi
85
answered Sep 18 '18 at 15:00
haruishiharuishi
85
answered Sep 18 '18 at 15:00
haruishiharuishi
85
85
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f37518%2ftime-horizon-t-in-policy-gradients-actor-critic%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
I think I have found something helpful - John Schulman notes in his thesis: !screenshot So it sounds like either I sample trajectories with variable length (arrival in a terminal state) or I encode the time step into the state and the state value function will also take the time step into consideration?
$endgroup$
– Dummie Variable
Sep 15 '18 at 13:49