Hfrxdjt

I'm looking at the code behind a Keras LSTM, and I noticed something I find odd.

Suppose we're feeding in input of size (batch_size, time_steps, input_dim)
where:

• 
  batch_size is the number of examples in the minibatch,


• 
  time_steps is the number of time steps to look back (i.e the window size)


• 
  input_dim is the number of input variables.

Suppose we want to input minibatches of size batch_size=5, time_steps=2, and input_dim=1.

That is, we have a univariate time series with five examples per minibatch and we use the previous two values of the time series to predict the next value.

Say we want to accomplish this using an LSTM with size 3 (that is, this is the number of hidden units and the number of output units). Call this size units.

In lines 1871-1995 of the code here, the LSTM is being built. The equation I'm confused about is on lines 1989-1990, which corresponds to the calculation of the input gate:

i = self.recurrent_activation(x_i + K.dot(h_tm1_i, self.recurrent_kernel_i))

The variable x_i is calculated as:

x_i = K.dot(inputs_i, self.kernel_i)

where

• 
  inputs_i = the input at time i, which has shape (batch_size, time_steps, input_dim), in our case would have shape (5, 2, 1)


• 
  self.kernel_i = the weight matrix that is multiplied by the current input at time t, and has shape (input_dim, units), in our case would have shape (1, 3)

I understand this dot product is accomplished via broadcasting, and the final shape is (batch_size, time_steps, units), which in our case is (5, 2, 3).

Now let's examine the dot product:

K.dot(h_tm1_i, self.recurrent_kernel_i))

where

• h_tm1_i = the recurrent hidden state at time i, and has shape equal to (batch_size, units) according to line 295. In our case, that's (5, 3)




• self.recurrent_kernel = the weight matrix that is multiplied by the previous hidden state, and has shape (units, units), which in our case would be (3, 3).

Somehow via the magic of broadcasting, the dot product gives something with shape (5, 3)

We're left with needing to add x_i and K.dot(h_tm1_i, self.recurrent_kernel_i)), which have shapes (5, 2, 3) and (5, 3) respectively. When I try to do that myself in tensorflow, I get an error:

ValueError: Dimensions must be equal, but are 2 and 5 for 'add_1' (op: 'Add') with input shapes: [5,2,3], [5,3].

Clearly I've done something wrong somewhere, but I can't see my logic error. Can anyone help?

EDIT: To reproduce the error:

>>> import tensorflow as tf
>>> import keras
>>> from keras import backend as K
>>> inputs_i = tf.ones([5, 2, 1])
>>> kernel_i = tf.ones([1,3])
>>> h_tm1_i = tf.ones([5,3])
>>> rec_i = tf.ones([3,3])
>>> x_i = K.dot(inputs_i, kernel_i)
>>> x_i
<tf.Tensor 'Reshape_9:0' shape=(5, 2, 3) dtype=float32>
>>> K.dot(h_tm1_i, rec_i)
<tf.Tensor 'MatMul_4:0' shape=(5, 3) dtype=float32>
>>> x_i + K.dot(h_tm1, rec_i) #Raises ValueError