Sample questions
Fe Text Highcardinality FeaturesDifficulty 3
In scikit-learn 1.6:
from sklearn.feature_extraction.text import CountVectorizer
v = CountVectorizer(tokenizer=str.split)
print(v.fit(['Buy A now', 'buy a NOW']).get_feature_names_out())
The same two documents under a default CountVectorizer report ['buy' 'now']. What is printed here?
- a['buy' 'now'] — a supplied tokenizer still hands its pieces to the token pattern, which discards the one-letter one.
- b['a' 'buy' 'now'] — the token pattern is bypassed, since lowercasing survives the swap while the length filter does not.✓
- c['A' 'Buy' 'NOW' 'a' 'buy' 'now'] — a supplied tokenizer takes over the whole text-handling chain, casing included.
- d['Buy A now' 'buy a NOW'] — str.split is handed the corpus, so each whole document ends up as one token.
Explanation:A custom tokenizer replaces only the regex step, and scikit-learn even warns that token_pattern will go unused; every other stage stays where it was. Preprocessing still lowercases the document before the callable sees it, and a stop list would still be applied to whatever the callable returns, so the single visible change is that a one-character piece is no longer filtered away.
Fe Text Highcardinality FeaturesDifficulty 2
A team indexes the two sentences 'dog bites man' and 'man bites dog' with a default CountVectorizer and is surprised that no classifier can separate them. What does the count matrix actually contain for these two rows?
- aTwo identical rows of [1, 1, 1]: unigram counting keeps no positional information.✓
- bTwo rows differing in one column, since the vectorizer also stores each token's first position.
- cTwo identical rows only when binary=True is set; with raw counts the rows would differ.
- dTwo rows over different column sets, because the vocabulary is rebuilt for each document.
Explanation:The bag-of-words representation records how many times each vocabulary term occurs and nothing else, so any permutation of the same tokens maps to the same vector. Recovering adjacency requires features that span more than one token, such as an n-gram range wider than a single word.
Fe Text Highcardinality FeaturesDifficulty 2
In scikit-learn 1.6:
from sklearn.feature_extraction.text import CountVectorizer
v = CountVectorizer(ngram_range=(1, 2))
X = v.fit_transform(['red blue red'])
print(X.shape, X.toarray())
What is printed?
- a(1, 2) [[2 1]] — only unigrams are produced; bigrams need a separate analyzer setting.
- b(1, 3) [[1 2 1]] — the repeated token collapses one of the adjacent pairs.
- c(1, 4) [[1 1 2 1]] — two unigrams plus the two distinct adjacent pairs.✓
- d(1, 5) [[1 1 2 1 1]] — 'red red' is also emitted as a bigram of the two occurrences.
Explanation:An ngram_range of (1, 2) emits unigrams and bigrams together. Here the unigrams are 'blue' and 'red', and the adjacent pairs are 'red blue' and 'blue red'; sorted alphabetically the four columns are 'blue', 'blue red', 'red', 'red blue', and only 'red' has a count of two.
Fe Text Highcardinality FeaturesDifficulty 2
The document 'buy now buy now buy' is vectorized twice: once with CountVectorizer() and once with CountVectorizer(binary=True). The default run yields the row [3, 2]. What does the second run yield, and why?
- a[1, 1] — each present term is recorded as 1 regardless of how often it occurs.✓
- b[0.6, 0.4] — counts above one are divided by the document's token length.
- c[3, 2] again — binary only switches the stored dtype from integer to boolean, values unchanged.
- d[3, 0] — only the term with the highest count is kept and the rest are zeroed.
Explanation:The binary flag clamps every non-zero count to one after tokenization, so the row carries presence rather than frequency. The vocabulary and the column layout are untouched; only the stored values change, which is why the matrix still has two columns.
Fe Text Highcardinality FeaturesDifficulty 3
An engineer feeds a column of short product codes such as 'A-1', 'B-2' and 'C-7' into a default CountVectorizer and gets ValueError: empty vocabulary. Which property of the default tokenizer explains this?
- aThe default analyzer rejects documents shorter than three tokens before the vocabulary is built.
- bHyphens are treated as document separators, so each code is read as two empty documents.
- cDigits are stripped during preprocessing and the leftover single letters match the built-in stop list.
- dThe default token pattern demands at least two word characters, so every one-character piece is discarded.✓
Explanation:CountVectorizer's default token_pattern is (?u)\b\w\w+\b, which only matches runs of two or more word characters. The hyphen splits each code into a single letter and a single digit, both too short to match, so nothing survives tokenization and the vocabulary comes out empty.
Fe Text Highcardinality FeaturesDifficulty 1
A product catalogue writes the same brand as 'Apple', 'apple' and 'APPLE'. A default CountVectorizer is fitted on it. How many vocabulary entries do these three spellings produce?
- aThree — casing is part of the token and only accent stripping could merge them.
- bOne, because lowercasing is on by default and folds all three into the same token.✓
- cTwo, because all-caps is folded to title case while lower case stays a separate entry.
- dOne, but only when binary=True is also passed; otherwise each casing gets its own column.
Explanation:The default preprocessing step lowercases the document before the token pattern is applied, so the three variants become the same string and share one column. Setting lowercase=False would keep them apart and produce three separate vocabulary entries.