Which of the following best describes positional encoding in Transformers?

Question

BlackTom AI · Accepted Answer

Positional encoding supplies tokens with information about their order in the sequence by augmenting token embeddings so the self-attention mechanism can distinguish different positions.

1) Relevant concepts
- Self-attention in Transformers computes pairwise interactions between token embeddings and is permutation-invariant: if you permute the input tokens, attention scores depend only on content, not order, unless order information is injected.
- Positional encoding provides either fixed (sinusoidal) or learned vectors that encode absolute or relative positions and are added to (or concatenated with) token embeddings so the model can use sequence order.
- Common fixed form: PE(pos,2i) = sin(pos / 10000^(2i/d_model)), PE(pos,2i+1) = cos(pos / 10000^(2i/d_model)). Alternately, models can learn an embedding vector for each position.

2) Step-by-step reasoning
- Without positional encodings, two sequences that are permutations of each other produce identical sets of embeddings and thus identical self-attention outputs, so the model cannot learn order-sensitive tasks.
- Adding positional encodings makes each token embedding unique to its position: embedding(token) + PE(position).
- The attention mechanism then uses both content and position; downstream layers can learn patterns that depend on relative or absolute ordering (e.g., subject before verb).

3) Example calculation (intermediate steps)
For d_model = 4 and pos = 1:
- i = 0: angle = pos / 10000^(0) = 1 → PE(1,0)=sin(1) ≈ 0.8415, PE(1,1)=cos(1) ≈ 0.5403
- i = 1: angle = pos / 10000^(2/4)= pos / 100 = 0.01 → PE(1,2)=sin(0.01) ≈ 0.0099998, PE(1,3)=cos(0.01) ≈ 0.99995
These four numbers form the positional vector added to the token’s embedding.

4) Why the correct answer is correct
The correct option states positional encoding helps the model understand the position of words in a sequence. That directly matches the purpose described above: to inject order information so attention can be order-aware.

5) Why other options are incorrect
- “It encodes syntactic rules of grammar.” Positional encodings do not explicitly encode grammar rules; they only provide positional signals that the model may use to learn syntactic patterns.
- “It stores semantic information in words.” Semantic meaning is captured by token embeddings trained from data; positional encodings carry position, not word meaning.
- “It allows the model to ignore token order.” This is the opposite: positional encodings are introduced so the model does not ignore token order.

Final confirmation: The correct answer is “It helps the model understand the position of words in a sequence.”

Which of the following best describes positional encoding in Transformers? 单项选择题

类似问题

What is combined with the inputs (embeddings) to the transformer architecture that encodes contextual information that can be used by attention mechanisms to create embeddings with more context?

In a self-attention transformer network, which of the following is true for sinusoid-based positional encoding vectors

You own a bond that pays $64 in interest annually. The face value is $1,000 and the current market price is $1,021.61. The bond matures in 11 years. What is the yield to maturity?

Which of the following statements is CORRECT?

更多留学生实用工具

智能学习助手

风格化写作助手

论文查重助手

文献引用助手

课堂转译助手

课堂笔记助手

Quiz搜索助手

学校历年真题

智能刷题助手

智能匹配练习

希望你的学习变得更简单