Which statements are correct regarding "Activation" Function?

Question

BlackTom AI · Accepted Answer

Activation functions introduce nonlinearity into neural networks, which is the key property that enables them to model complex functions beyond simple linear mappings.

1) Relevant knowledge points and concepts
- Linear vs. nonlinear mapping: A composition of linear transformations is itself linear (e.g., W2(W1 x + b1) = (W2 W1) x + (W2 b1 + b2)). Without nonlinearities, a deep network collapses to a single linear transform.
- Universal approximation: With nonlinear activation functions, feedforward networks can approximate a wide class of functions; without them they cannot.
- Output-layer activations: The last layer may use a specific activation (softmax for classification, sigmoid for binary, linear for regression) but that does not replace the need for nonlinearity in hidden layers.
- Modern architectures: Transformers, CNNs, RNNs all use nonlinear activations (e.g., ReLU, GELU, tanh).

2) Step-by-step reasoning
- Suppose all layers are linear. For input x and two layers: y = W2(W1 x + b1) + b2 = (W2 W1) x + (W2 b1 + b2). That's a single linear function, so depth provides no representational advantage.
- Introducing an activation a(·) after each linear step makes the mapping nonlinear: y = W2 a(W1 x + b1) + b2. Nonlinear a(·) allows compositions to produce complex decision boundaries.
- Last-layer activation choices serve specific tasks (probabilities, bounded outputs) but don’t remove the necessity of hidden-layer nonlinearities.

3) Intermediate steps / demonstration (simple)
- Linear-only network: two linear layers reduce to one linear: compute product of weight matrices and sum biases as shown above.
- With ReLU: a(W1 x) changes the subspace; subsequent linear transforms act on a nonlinear transformation of x — not reducible to one linear map.

4) Why the correct answer is correct
- “Activation functions are essential for neural nets to create non linear functions.” This is true because without activations a network cannot represent nonlinear functions and thus cannot learn complex mappings.

5) Why other options are incorrect
- “Optional part” — false; without them depth is meaningless for nonlinearity.
- “Only needed in the last layer” — false; hidden layers need activations to create nonlinearity.
- “Not needed anymore due to more processing power” — false; compute does not change the mathematical need for nonlinearity.
- “Not needed in new architectures” — false; modern architectures explicitly include nonlinearities (e.g., GELU in transformers).

6) Conclusion
The correct choice is: Activation functions are essential for neural nets to create non linear functions. This matches the provided answer.

Which statements are correct regarding "Activation" Function?单项选择题

类似问题

Question at position 17 Which activation function is commonly used in the target layer when modeling a binary target?exponentiallogisticidentitytanh

Question at position 16 Which of following activation function can’t be used at output layer of a neural network to classify an image?TanhSigmoidIf(x>5,1,0)ReLUNone of the above

Which of the followings is not a widely used activation function?

Question at position 13 What activation function should we use for the output layer for the MNIST dataset?SoftmaxReLU

Question at position 13 What activation function should we use for the output layer for the MNIST dataset?SoftmaxReLU

Question at position 13 What activation function should we use for the output layer for the MNIST dataset?SoftmaxReLU

Question at position 11 Which of the following computation adds nonlinearity to an ANN?Performing activation functionTaking weighed sum of inputs

Question at position 11 Which of the following computation adds nonlinearity to an ANN?Performing activation functionTaking weighed sum of inputs

更多留学生实用工具

智能学习助手

风格化写作助手

论文查重助手

文献引用助手

课堂转译助手

课堂笔记助手

Quiz搜索助手

学校历年真题

智能刷题助手

智能匹配练习

希望你的学习变得更简单