Hfrxdjt

Night of Shab e Meraj

Notepad++ delete until colon for every line with replace all

GFCI outlets - can they be repaired? Are they really needed at the end of a circuit?

How to install cross-compiler on Ubuntu 18.04?

Forgetting the musical notes while performing in concert

Why do I get negative height?

Can compressed videos be decoded back to their uncompresed original format?

What is an equivalently powerful replacement spell for the Yuan-Ti's Suggestion spell?

Is there a hemisphere-neutral way of specifying a season?

How can saying a song's name be a copyright violation?

What historical events would have to change in order to make 19th century "steampunk" technology possible?

Finitely generated matrix groups whose eigenvalues are all algebraic

What do you call someone who asks many questions?

How to stretch the corners of this image so that it looks like a perfect rectangle?

files created then deleted at every second in tmp directory

Is this answer explanation correct?

How to show a landlord what we have in savings?

What does the same-ish mean?

Where would I need my direct neural interface to be implanted?

Fair gambler's ruin problem intuition

How seriously should I take size and weight limits of hand luggage?

OP Amp not amplifying audio signal

How do conventional missiles fly?

How to compactly explain secondary and tertiary characters without resorting to stereotypes?

Vanishing gradient problem for recent stochastic recurrent neural networks

0

$begingroup$

Recently, I've found some papers about generative recurrent models. All have attached sub-networks like prior/encoder/decoder/etc. to well-known LSTM cell for composing an aggregation of new-type RNN cell.

I am just curious about whether the gradient vanishing/exploding happens or not to those new RNN cell. Isn't there any problem about that kind of combination?

References:

It seems like they all have similar pattern as mentioned above.

A Recurrent Latent Variable Model for Sequential Data

Learning Stochastic Recurrent Networks

Z-Forcing: Training Stochastic Recurrent Networks

Pseudocode

The pseudocode for recurrent architecture is below:

def new_rnncell_call(x, htm1):
 #prior_net/posterior_net/decoder_net is single layer or mlp each
 q_prior = prior_net(htm1) # prior step
 q = posterior_net([htm1, x]) # inference step
 z = sample_from(q) # reparameterization trick
 target_dist = decoder_net(z) # generation step
 ht = innerLSTM([z, x], htm1) # recurrent step
 return [q_prior, q, target_dist], ht

What concerns me are those naked weights outside of well-known LSTM (or GRU etc.) cell during processing bptt without any gating logic for activations as the weights inside LSTM. For me, this looks not similar to stacked-rnn layers or additional dense layers just to outputs.

Doesn't that have any gradient vanishing/exploding problem?

python deep-learning gradient-descent recurrent-neural-net

share|improve this question

asked 26 mins ago

Sehee ParkSehee Park

1

New contributor

Sehee Park is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

0

$begingroup$

Recently, I've found some papers about generative recurrent models. All have attached sub-networks like prior/encoder/decoder/etc. to well-known LSTM cell for composing an aggregation of new-type RNN cell.

I am just curious about whether the gradient vanishing/exploding happens or not to those new RNN cell. Isn't there any problem about that kind of combination?

References:

It seems like they all have similar pattern as mentioned above.

A Recurrent Latent Variable Model for Sequential Data

Learning Stochastic Recurrent Networks

Z-Forcing: Training Stochastic Recurrent Networks

Pseudocode

The pseudocode for recurrent architecture is below:

def new_rnncell_call(x, htm1):
 #prior_net/posterior_net/decoder_net is single layer or mlp each
 q_prior = prior_net(htm1) # prior step
 q = posterior_net([htm1, x]) # inference step
 z = sample_from(q) # reparameterization trick
 target_dist = decoder_net(z) # generation step
 ht = innerLSTM([z, x], htm1) # recurrent step
 return [q_prior, q, target_dist], ht

What concerns me are those naked weights outside of well-known LSTM (or GRU etc.) cell during processing bptt without any gating logic for activations as the weights inside LSTM. For me, this looks not similar to stacked-rnn layers or additional dense layers just to outputs.

Doesn't that have any gradient vanishing/exploding problem?

python deep-learning gradient-descent recurrent-neural-net

share|improve this question

asked 26 mins ago

Sehee ParkSehee Park

1

New contributor

Sehee Park is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

$begingroup$

Recently, I've found some papers about generative recurrent models. All have attached sub-networks like prior/encoder/decoder/etc. to well-known LSTM cell for composing an aggregation of new-type RNN cell.

I am just curious about whether the gradient vanishing/exploding happens or not to those new RNN cell. Isn't there any problem about that kind of combination?

References:

It seems like they all have similar pattern as mentioned above.

A Recurrent Latent Variable Model for Sequential Data

Learning Stochastic Recurrent Networks

Z-Forcing: Training Stochastic Recurrent Networks

Pseudocode

The pseudocode for recurrent architecture is below:

def new_rnncell_call(x, htm1):
 #prior_net/posterior_net/decoder_net is single layer or mlp each
 q_prior = prior_net(htm1) # prior step
 q = posterior_net([htm1, x]) # inference step
 z = sample_from(q) # reparameterization trick
 target_dist = decoder_net(z) # generation step
 ht = innerLSTM([z, x], htm1) # recurrent step
 return [q_prior, q, target_dist], ht

What concerns me are those naked weights outside of well-known LSTM (or GRU etc.) cell during processing bptt without any gating logic for activations as the weights inside LSTM. For me, this looks not similar to stacked-rnn layers or additional dense layers just to outputs.

Doesn't that have any gradient vanishing/exploding problem?

python deep-learning gradient-descent recurrent-neural-net

share|improve this question

asked 26 mins ago

Sehee ParkSehee Park

1

New contributor

Sehee Park is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

def new_rnncell_call(x, htm1):
 #prior_net/posterior_net/decoder_net is single layer or mlp each
 q_prior = prior_net(htm1) # prior step
 q = posterior_net([htm1, x]) # inference step
 z = sample_from(q) # reparameterization trick
 target_dist = decoder_net(z) # generation step
 ht = innerLSTM([z, x], htm1) # recurrent step
 return [q_prior, q, target_dist], ht

share|improve this question

asked 26 mins ago

Sehee ParkSehee Park

1

New contributor

Sehee Park is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|improve this question

asked 26 mins ago

Sehee ParkSehee Park

1

New contributor

Sehee Park is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 26 mins ago

Sehee ParkSehee Park

1

New contributor

Sehee Park is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 26 mins ago

Sehee ParkSehee Park

1