What problem does LSTM solve that basic RNNs struggle with?

Question

BlackTom AI · Accepted Answer

Vanishing gradients in long sequences.

Relevant knowledge points and concepts:
- Standard (vanilla) RNNs compute hidden states h_t = f(Wh_{t-1} + Ux_t + b). Training uses backpropagation through time (BPTT), which requires propagating gradients across many time steps.
- The vanishing (or exploding) gradient problem arises because gradients are products of many Jacobian matrices; values less than 1 shrink exponentially, values greater than 1 blow up.
- LSTM (Long Short-Term Memory) networks introduce a cell state and gating mechanisms (input, forget, output) that enable better gradient flow across long time spans, often called the constant error carousel.

Step-by-step reasoning and intermediate calculations:
1. Consider an RNN recurrence h_t = f(Wh_{t-1} + ...). The partial derivative of the loss L at time T with respect to a hidden state at time t < T involves a product over intervening time steps:
   ∂L/∂h_t = (∏_{k=t+1..T} ∂h_k/∂h_{k-1}) · ∂L/∂h_T.
2. Each term ∂h_k/∂h_{k-1} contains factors like f'(net_k)·W. If the spectral radius (or typical magnitude) of these factors is α < 1, then after m steps the product scales like α^m.
3. Numerical example: if average factor α ≈ 0.1 and m = 50 time steps, the gradient magnitude scales roughly as 0.1^50 ≈ 1e-50 — effectively zero (vanished).
4. LSTM changes the recurrence for the cell state c_t to an additive form: c_t = f_t * c_{t-1} + i_t * g_t, where f_t (forget gate) can be close to 1. The derivative ∂c_t/∂c_{t-1} = f_t allows gradients to pass through many steps without exponential decay. This preserves learning signals over long sequences.

Why the correct answer is correct:
- LSTM was specifically designed to mitigate vanishing gradients, enabling learning of long-range dependencies by using gated, mostly multiplicative control but additive persistence in the cell state (the constant error carousel). This architectural change prevents the repeated multiplicative shrinking that crippled vanilla RNNs.

Why the other options are incorrect:
- "Lack of parallel computation.": Both vanilla RNNs and LSTMs are sequential by nature; LSTM does not remove the inherent sequential dependency that limits parallelism across time steps.
- "Overfitting due to very few parameters.": LSTMs typically have more parameters than basic RNNs; they do not exist to solve a “too few parameters” overfitting problem.
- "Inability to process non-sequential data.": Both RNNs and LSTMs are designed for sequential data; the problem LSTM solves is about long-term dependencies within sequences, not processing non-sequential inputs.

Conclusion:
The LSTM addresses vanishing gradients in long sequences, which matches the provided correct answer: "Vanishing gradients in long sequences."

What problem does LSTM solve that basic RNNs struggle with? 单项选择题

类似问题

What distinguishes RNNs from feedforward neural networks in terms of processing sequences?

Which of the following best describes the primary purpose of Long Short-Term Memory (LSTM) networks in deep learning?

更多留学生实用工具

智能学习助手

风格化写作助手

论文查重助手

文献引用助手

课堂转译助手

课堂笔记助手

Quiz搜索助手

学校历年真题

智能刷题助手

智能匹配练习

希望你的学习变得更简单