Which of the following best describes a key property of Transformer architecture used in LLMs?

Question

BlackTom AI · Accepted Answer

Transformer architecture uses self-attention to capture relationships between tokens, allowing each token to directly attend to every other token in the sequence.

Relevant knowledge points:
- Self-attention computes pairwise interactions between tokens using queries (Q), keys (K), and values (V). The core formula is Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V.  
- Positional encodings provide token order information because self-attention itself is order-agnostic.  
- Transformers process tokens in parallel during training (not strictly sequentially), which yields efficiency and better modeling of long-range dependencies compared with recurrent models.

Step-by-step reasoning:
1. Represent each token as an embedding; produce Q, K, V vectors by linear projections of embeddings.  
2. Compute raw scores by multiplying Q and K^T for all token pairs; this yields a matrix of similarity scores where each entry indicates how much one token should attend to another.  
3. Scale by 1/sqrt(d_k) to stabilize gradients, then apply softmax across each row to get attention weights that sum to 1.  
4. Multiply the attention weights by V to produce context-aware token representations combining information from related tokens.  
5. Multi-head attention repeats this in parallel subspaces to capture diverse relations.

Why the correct answer is correct:
- The described mechanism is exactly “self-attention” and is the defining, distinguishing property of Transformers. It enables modeling of long-range and context-dependent relationships without recurrence.

Why other options are incorrect:
- “Convolution layers for sequence modeling”: Convolutions (CNNs) capture local patterns and were used in some sequence models, but they do not form the core of Transformer design.  
- “Relies heavily on recurrence”: Recurrence is the hallmark of RNNs/LSTMs, not Transformers.  
- “Processes input tokens sequentially”: Transformers allow parallel processing of tokens during training; only autoregressive decoding is sequential.

Conclusion: The correct choice is “It uses self-attention to capture relationships between tokens,” which matches the provided answer.

Which of the following best describes a key property of Transformer architecture used in LLMs?Single choice

Similar Questions

The transformer architecture only consists of a single encoder component and single decoder component.

What is the main architectural innovation in ChatGPT that allows it to handle complex language tasks?

A team of data scientists is experiencing poor performance with their Transformer model. After inspecting their implementation, you find several suspicious design choices. Which one would you identify as the MOST problematic based on the architectural principles discussed in the lecture?

What operations are part of a standard Transformer block? (Select all that apply.)

Is the following statement true or false? In the products of methylation-hydrolysis, every -OH group corresponds to the position of a glycosidic bond in the starting polysaccharide.

Which of the follwoing structures represents amylopectin?

Which of the following is true about an aldopentose?I. It is a monosaccharide.II. It contains a CHO groupIII. It is a disaccharide.IV. It is an oligosaccharide.

More Practical Tools for Students Powered by AI Study Helper

Homework AI Solver

Stylized AI Paper Writer

Plagiarism Checker Assistant

Citation AI Academic Writing Tool

In-Class Translation Assistant

AI Note Generator

AI Quiz Answers

Past Exam Questions from University Test Bank

Smart Practice Assistant

Adaptive Practice

Making Your Study Simpler