Aside from the transformer architecture itself, what is the major technological breakthrough in natural language techniques that significantly advanced natural language AI capabilities, particularly with respect to understanding the relationships of words and their context, even when there are interdependencies and long-range dependencies (e.g., within sentences, paragraphs, etc).

As a bonus, this breakthrough also solved many of the problems that recurrent neural network (RNN) techniques such as LSTMs had with long-range dependencies and contextual understanding, and is one of the major reasons that transformers are replacing the use of RNNs.

Question

Aside from the transformer architecture itself, what is the major technological breakthrough in natural language techniques that significantly advanced natural language AI capabilities, particularly with respect to understanding the relationships of words and their context, even when there are interdependencies and long-range dependencies (e.g., within sentences, paragraphs, etc).

As a bonus, this breakthrough also solved many of the problems that recurrent neural network (RNN) techniques such as LSTMs had with long-range dependencies and contextual understanding, and is one of the major reasons that transformers are replacing the use of RNNs.

BlackTom AI · Accepted Answer

Self-attention (the attention mechanism) computes explicit, dynamic weights between every pair of tokens so the model can directly model relationships and context across short and long distances.

1) Relevant knowledge points and concepts
- Attention: mechanism that produces context-aware weighted sums of token representations. The common formula is Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V, where Q (queries), K (keys), V (values) are linear projections of token embeddings and d_k is the key dimension.
- Self-attention: each token attends to all tokens in the sequence (including itself), enabling direct interactions between distant tokens.
- Transformer advantages: parallel computation across positions, multi-head attention to capture multiple types of relationships, and O(n^2) pairwise interactions rather than strictly sequential recurrence.
- RNN/LSTM limitations: sequential processing, difficulty with very long-range dependencies, and slower training due to lack of full parallelism; LSTMs alleviate but do not fully solve long-range contextual modeling.

2) Step-by-step reasoning
- Project embeddings into Q, K, V for all tokens.
- Compute similarity scores via QK^T, scale by sqrt(d_k), and apply softmax to get attention weights between every pair of tokens.
- Multiply weights by V to produce context-aware representations that integrate information from relevant distant tokens in one layer.
- Stack multiple layers and heads so different relational patterns and contexts are captured.

3) Intermediate calculation example
- For two tokens with query q and keys k_i, attention weight for token i is: a_i = softmax(q · k_i / sqrt(d_k)). These weights sum to 1 and dynamically choose which tokens influence the output.

4) Why the correct answer is correct
- Attention directly models interdependencies and long-range context by allowing any token to attend to any other token in a single layer. This overcomes RNNs’ sequential bottleneck and LSTMs’ partial fixes, enabling superior contextual understanding and parallel training—key advances in modern NLP.

5) Why other options are incorrect
- Bi-directionality: a useful property (e.g., BERT) but relies on attention to function—it’s not the fundamental mechanism that models relationships.
- Embeddings: capture static semantic similarity (word meaning) but don’t by themselves model dynamic context or long-range dependencies.
- Auto-regressiveness: a generation strategy (predict next token) not the core technique that models token-to-token contextual relationships.

6) Conclusion
The correct answer is Attention, which matches the provided answer.

类似问题

What is the main role of the attention mechanism in an LLM?

Which innovation is at the core of the transformer architecture and enables modeling long-range dependencies effectively?

Which of the following attention models uses a subset of the input to derive the output, and can not be trained directly with gradient methods?

Why is the attention mechanism particularly suitable for modeling financial time series?

Which of the following statements is correct about query, key, and value in transformer models?

Consider a single-headed attention layer. What happens to the dimensions of the value weight matrix Wv, when we double the maximum input sequence length? Select all that apply

更多留学生实用工具

智能学习助手

风格化写作助手

论文查重助手

文献引用助手

课堂转译助手

课堂笔记助手

Quiz搜索助手

学校历年真题

智能刷题助手

智能匹配练习

希望你的学习变得更简单