What is the key difference between pre-training and fine-tuning stages in transformer model development?

Hint: Lec 19, Slide 50.

Question

What is the key difference between pre-training and fine-tuning stages in transformer model development?

Hint: Lec 19, Slide 50.

BlackTom AI · Accepted Answer

The question asks about the key difference between pre-training and fine-tuning stages in transformer model development, referencing Lec 19, Slide 50.
Option: 'Pre-training involves learning general things with a large dataset and serves as parameter initialization while fine-tuning adapts the model to a specific task with a smaller dataset.'
This statement captures the standard division of labor in transformer training pipelines: pre-training uses a large, general corpus to learn broad language patterns and to establish initial parameter values; fine-tuning then takes that pre-trained model and adjusts its weights on a smaller, task-specific dataset to specialize the model for a particular task.
Understanding the pre-training phase helps establish robust representations, while the fine-tuning phase specializes those representations to the target task, often with a smaller dataset and sometimes with adjustments like a lower learning rate or selective freezing of layers.
If we consider why other possibilities would be incorrect, we would typically see claims that pre-training is task-specific and small-scale, or that fine-tuning does not adapt parameters, or that there is no initialization effect. Those ideas conflict with the widely accepted view that pre-training provides general-purpose representations and initialization, while fine-tuning specializes the model to the task at hand using more focused data.
In short, the described distinction aligns with the conventional two-stage training paradigm for transformers: broad, general-purpose learning followed by targeted adaptation through fine-tuning.

What is the key difference between pre-training and fine-tuning stages in transformer model development? Hint: Lec 19, Slide 50.单项选择题

类似问题

A medical team wants an LLM that understands radiology terminology but cannot afford to retrain a model from scratch. Which approach is most appropriate?

Question at position 10 What is transfer learning in neural networks? Using a pre-trained neural network to extract features from a new datasetUsing a pre-trained neural network as a starting point for training a new modelUsing a pre-trained neural network to generate new dataNone of the above

Transfer learning is invariably effective. eg. Irrespective of the amount of data, we can always rely on transfer learning.

When attempting to transfer learn for an image captioning task, we must use a source dataset for image captioning or visual question answering, since image classification by itself is not similar enough

Which of the following statements about transfer learning in CNNs are correct? (mark all that apply)

Which of the following statements is correct about pre-training and fine-tuning?

更多留学生实用工具

智能学习助手

风格化写作助手

论文查重助手

文献引用助手

课堂转译助手

课堂笔记助手

Quiz搜索助手

学校历年真题

智能刷题助手

智能匹配练习

希望你的学习变得更简单