In gradient descent for linear regression, what is the main role of the step size (learning rate)?

Question

BlackTom AI · Accepted Answer

The gradient descent update rule w := w - η ∇J(w) shows the central role of the step size (learning rate) η: it multiplies the gradient and thus directly scales the magnitude of each weight update.

1) Relevant knowledge points and concepts
- Cost function J(w) (e.g., mean squared error for linear regression) and its gradient ∇J(w).
- Gradient descent update rule w(t+1) = w(t) - η ∇J(w(t)).
- Effects of η: convergence speed, overshooting/divergence, stability; relationship to convexity (MSE is convex for linear regression).

2) Step-by-step reasoning
- For linear regression with MSE, J(w) = (1/2n) Σ (y_i - x_i^T w)^2; gradient ∇J(w) is a vector of partial derivatives.
- Each iteration computes ∇J(w) and multiplies it by η to form the change Δw = -η ∇J(w).
- Thus η directly sets how large Δw is: larger η → larger steps; smaller η → smaller steps.

3) Example calculation (intermediate steps)
- Suppose for a particular iteration a component of the gradient is 2.0.
- If η = 0.1, the update for that component is Δw = -0.1 * 2.0 = -0.2.
- If η = 1.0, Δw = -1.0 * 2.0 = -2.0.
- If η is too large the updates may overshoot the minimum and diverge; if too small convergence will be very slow.

4) Why the correct answer is correct
- Because η explicitly multiplies the gradient in the update rule, its main role is controlling the magnitude of weight updates each iteration, impacting convergence speed and stability.

5) Why the other options are incorrect
- It does not determine the number of features used (feature selection is unrelated to η).
- It does not guarantee finding the global minimum in a fixed number of steps; convergence depends on η, problem conditioning, and number of iterations (though convexity ensures eventual convergence under proper η).
- It does not prevent multicollinearity or shrink coefficients to zero (that is regularization like L1/L2).

6) Conclusion
The correct answer is: It controls the magnitude of the the weight updates on each iteration. This matches the provided answer.

In gradient descent for linear regression, what is the main role of the step size (learning rate)?单项选择题

类似问题

Question19 Consider the function [math]. Run gradient descent on this function with a starting point of [math] and learning rate [math]. Which of the following is true after 2 iterations? [math] [math] [math] [math] [math] ResetMaximum marks: 1 Flag question undefined

Continuing from the question above, one may introduce a step length parameter, 𝛼 , into the formula as follows: 𝑊 𝑘 = 𝑊 𝑘 − 1 + 𝛼 𝑑 𝑘 − 1 Please select all the correct answers below.

Which of the following is NOT true about the steepest descent method?

When trying to find the minimum of a function f using the steepest descent method, which of the following is a plausible termination criteria?

In the steepest descent method, the direction at every iteration is

Which of the following is NOT true about the steepest descent method?

When trying to find the minimum of a function f using the steepest descent method, which of the following is a plausible termination criteria?

What was the world's first widely adopted biodiversity policy?

更多留学生实用工具

智能学习助手

风格化写作助手

论文查重助手

文献引用助手

课堂转译助手

课堂笔记助手

Quiz搜索助手

学校历年真题

智能刷题助手

智能匹配练习

希望你的学习变得更简单