Which preprocessing step reduces feature dimensions?

Question

BlackTom AI · Accepted Answer

Lemmatisation maps inflected word forms to their canonical base form (lemma), collapsing multiple surface tokens into a single feature and thus reducing the dimensionality of a text feature space.

1) Relevant concepts
- Feature dimensionality: the number of distinct features (e.g., vocabulary size in bag-of-words) used to represent data. High dimensionality increases sparsity and computational cost.
- Text preprocessing operations: tokenization, lowercasing, stop-word removal, stemming, lemmatisation, POS tagging, n-gram generation, etc.
- Lemmatisation specifically reduces morphological variation (e.g., runs, ran, running → run), so multiple features become one.

2) Step-by-step reasoning with an example
Consider a small corpus with tokens:
runs, ran, running, run, eats, eaten, eating, eat, the, a
Step A — count distinct tokens before lemmatisation:
Distinct tokens = 10.
Step B — apply lemmatisation (map each inflected form to lemma):
runs → run
ran → run
running → run
run → run
eats → eat
eaten → eat
eating → eat
eat → eat
the → the
a → a
Step C — distinct tokens after lemmatisation:
Unique lemmas = {run, eat, the, a} = 4.

3) Calculations (intermediate steps shown)
- Initial distinct features = 10
- Final distinct features = 4
- Absolute reduction = 10 − 4 = 6 features
- Relative reduction = (6 / 10) × 100% = 60% reduction in feature count

4) Why lemmatisation is correct
Lemmatisation explicitly joins multiple morphological variants into a single canonical form, which directly reduces the number of features produced by text vectorizers (one-hot, TF-IDF, etc.). This lowers sparsity, reduces model size, and can improve generalization by aggregating evidence across word forms.

5) Why other preprocessing steps are not the intended answer (brief)
- Tokenization splits text into tokens and typically increases raw feature candidates rather than reducing them.
- Lowercasing normalizes case (can slightly reduce unique tokens) but does not address morphological variants as effectively.
- Stop-word removal does reduce features but removes frequent functional words rather than consolidating morphological variants; if the multiple-choice options included both, lemmatisation is the method that reduces dimensions by collapsing variants.
- POS tagging and n-gram generation typically add information and increase feature dimensionality.
- Stemming also reduces morphological variation but is cruder than lemmatisation (may over-truncate); if the option set favored precise reduction to lemmas, lemmatisation is correct.

6) Conclusion
Because lemmatisation collapses inflected forms into single lemmas and the worked example shows a 60% reduction, lemmatisation reduces feature dimensions. This matches the provided correct answer: Lemmatisation.

Which preprocessing step reduces feature dimensions?单项选择题

类似问题

Bag of Words in text preprocessing is a-

Google Search's ability to recognize the meaning and intent behind user queries rather than merely matching keywords is based on which of the following?

In AI, what does NLP stand for?

What does the acronym “RAG” stand for?

Natural Language Processing is concerned with the interactions between computers and human (natural) languages. The main stages are

An AI system can recognize speech, convert it to text, and respond appropriately in real time. This capability primarily relies on:

What problem of Natural Language Processing was ChatGPT designed to fix?

更多留学生实用工具

智能学习助手

风格化写作助手

论文查重助手

文献引用助手

课堂转译助手

课堂笔记助手

Quiz搜索助手

学校历年真题

智能刷题助手

智能匹配练习

希望你的学习变得更简单