Penalized regression, SVM, KNN, CART and random forests

Penalized regression, SVM, KNN, CART and random forests

Published September 28, 2026

Penalized regression, SVM, KNN, CART and random forests

On this page

Learning Outcome Statement

Describe supervised machine learning algorithms-including penalized regression, support vector machine, k-nearest neighbor, classification and regression tree, ensemble learning, and random forest-and determine the problems for which they are best suited.

On the exam: You get a supervised target, predictor structure and scale clues, and algorithm traits. You choose the best-suited method or interpret a ridge calculation in an item set or MCQ.

You start with observations where each known target is paired with predictors. Training uses those paired observations to learn a mapping that predicts the target for new predictor values. Each algorithm imposes a different way of finding that mapping. On the exam, read the clues about predictor structure, scale, and the need for interpretation, then choose the best fit or read a penalized coefficient.

MethodMechanismBest fit
Penalized regressionLinear coefficients with shrinkageLinear or high-dimensional structure
Support vector machineMaximum-margin boundary, optional kernelsMargin-based nonlinear boundaries
K-nearest neighborLocal distance prediction, scaled predictorsLocal patterns
Classification and regression tree (CART)Binary recursive splits, depth and pruningInterpretable rules and interactions
Ensemble learningBagging in parallel, boosting sequentiallyCombined base learners
Random forestBootstrap samples, random predictor subsets, many treesStable nonlinear prediction

Do not choose the method with the fanciest name just because it sounds advanced. Match the mechanism to the case. An individual tree keeps its rules readable, while a forest gives up some transparency for steadier nonlinear prediction.

Ridge is the calculation the exam can ask you to perform. When the target is $Y$ and the predictor is $X$, the ridge coefficient, $\hat{\beta}_{\mathrm{ridge}}$, measures target-units per signal-unit. The sample covariance, $\operatorname{Cov}(X,Y)$, captures their joint movement, and the sample variance, $\operatorname{Var}(X)$, sets the predictor scale. The penalty, $\lambda$, uses the variance's units. Ridge shrinks regression coefficients toward zero, while lasso can set some coefficients exactly to zero.

$$ \hat{\beta}_{\mathrm{ridge}}(\lambda) = \frac{\operatorname{Cov}(X,Y)} {\operatorname{Var}(X)+\lambda} $$

Read the numerator first. It carries the sign and strength of the sample relationship. The variance sets the scale of an unpenalized slope, while $\lambda$ adds restraint to the denominator. As $\lambda$ increases, the absolute coefficient moves toward zero.

Loading diagram…

Use a support vector machine when the case calls for a separating boundary. It chooses a maximum-margin boundary, and a kernel lets that boundary represent nonlinear relationships.

For KNN, $k$ is the number of neighbors used for each prediction. For a new observation, the algorithm looks at nearby observations and uses their targets. Scale predictors before measuring distance, because unlike units can let one feature dominate. Choose $k$ to balance noise against local detail.

CART makes binary splits for classification or regression. The splits create readable rules and can capture interactions. Depth and pruning control overfitting.

Ensemble learning combines multiple base learners. Bagging trains learners in parallel on resampled data, whereas boosting trains sequentially so later learners correct earlier errors. Random forest combines bootstrap samples, random predictor subsets, and many decision trees to reduce correlated-tree variance.

Worked example

You are handed monthly sample estimates for a portfolio signal. The target is next-period excess return, and you need the ridge coefficient at the supplied penalty plus the direction across penalties. What does the model say about the signal weight?

InputValue
TargetNext-period excess return
PredictorSignal
DataMonthly sample
Covariance24 target-units times signal-units
Variance16 signal-units squared
Penalties0, 1, 4, 9 signal-units squared
Loading diagram…
Penalty, $\lambda$Ridge coefficient
01.5000
11.4118
41.2000
90.9600
  1. Select the requested penalty, $\lambda=4$, so the denominator matches the case. This produces $\lambda=4$.
  2. Substitute the sample covariance and variance with that penalty. The calculation $\hat{\beta}_{\mathrm{ridge}}(4)=\frac{24}{16+4}=1.2000$ produces a coefficient of $1.2000$ target-units per signal-unit.
  3. Repeat the substitution at the other supplied penalties to test the direction rather than guess it. This produces $\hat{\beta}{\mathrm{ridge}}(0)=1.5000$, $\hat{\beta}{\mathrm{ridge}}(1)=1.4118$, $\hat{\beta}{\mathrm{ridge}}(4)=1.2000$, and $\hat{\beta}{\mathrm{ridge}}(9)=0.9600$.

The portfolio manager reads this as a positive signal weight. The falling sequence shows that a larger penalty restrains the signal weight.

Changed condition

The same analyst now sees weaker sample comovement between signal and excess return. The variance and penalty values stay fixed, so only covariance changes.

QuantityOriginalWeaker covariance
$\operatorname{Cov}(X,Y)$2420
$\hat{\beta}_{\mathrm{ridge}}(0)$1.50001.2500
$\hat{\beta}_{\mathrm{ridge}}(9)$0.96000.8000

Predict a lower coefficient at every penalty before looking at the right column. The denominator is unchanged, while the smaller covariance lowers the numerator, so each coefficient should remain positive but move closer to zero. Before reading on, calculate $\hat{\beta}_{\mathrm{ridge}}(4)$ for the weaker covariance.

Common error

When you choose KNN from raw feature distances, you may trust the nearest observation because distance feels objective. That feels right because the algorithm appears to let the data decide without a model assumption. A vignette whose predictors use unlike units gives away the mistake: the large-unit feature dominates the distance and can decide which observations KNN calls nearest.

Calculator

KeysDisplay
24 ENTER24.00
16 ENTER16.00
4 ENTER4.00
24 / ( 16 + 4 ) CPT1.2000

Check yourself

Practice

A portfolio analyst estimates next-period excess return from one signal using monthly sample data. The covariance between the signal and target is 24 target-units times signal-units, the signal variance is 16 signal-units squared, and the ridge penalty is 4 signal-units squared. A separate team studies local classification patterns with predictors in unlike units and compares bagging with boosting.

Question 1

Calculate the ridge coefficient at the stated penalty.

Question 2

As the ridge penalty moves from 0 to 9, which interpretation is correct?

Question 3

Which choice best fits the local classification problem with unlike predictor units?

Question 4

Which statement correctly distinguishes bagging from boosting?

A classification case has a nonlinear boundary, and the analyst wants a maximum-margin method that can use a kernel. Which method is best suited?

Revision snapshot

Ridge shrinks correlated regression coefficients toward zero. Lasso can set some coefficients exactly to zero.

$$ \hat{\beta}_{\mathrm{ridge}}(\lambda) = \frac{\operatorname{Cov}(X,Y)} {\operatorname{Var}(X)+\lambda} $$

The covariance sets direction, variance sets scale, and higher $\lambda$ increases shrinkage. SVM chooses a maximum-margin boundary and kernels handle nonlinear relationships. KNN makes local distance predictions. Scale predictors and choose $k$ to balance noise against local detail. CART makes binary splits for classification or regression. Depth and pruning control overfitting. Ensemble learning combines base learners. Bagging is parallel on resampled data. Boosting is sequential and corrects errors. Random forest combines bootstrap samples, random predictor subsets, and many trees to reduce correlated-tree variance. Select penalized regression for linear or high-dimensional structure, SVM for margin-based nonlinear boundaries, KNN for local patterns, CART for interpretable rules and interactions, and random forest for stable nonlinear prediction. Raw KNN distance with unlike units is the scaling trap. On the exam, match mechanism to vignette.

Unlock the full lesson and continue reading

Signup to continue reading this lesson and unlimited access questions, flashcards, AI notes, and more

Scan to download app

Scan to download
UNLOCK FREE ACCESS

Practice Questions: Penalized regression, SVM, KNN, CART and random forests

  1. 1. An analyst compares one-period simple returns with continuously compounded returns for an asset whose price remains positive. Which conclusion is most accurate?

    • A. Both returns have a lower bound of −100%
    • B. Neither return has a finite lower bound
    • C. Only the continuously compounded return is unbounded below
    Show answer

    Answer: C. Only the continuously compounded return is unbounded below

    Only the continuously compounded return is unbounded below - The simple return, \(P1/P0-1\), has a lower bound of −100% because both prices are positive. - The continuously compounded return, \(\ln(P1/P0)\), approaches negative infinity as the ending price approaches zero. Both returns have a lower bound of −100% - −100% is the lower bound for the simple return, not the continuously compounded return. - The logarithm of a positive price relative has no finite lower bound. Neither return has a finite lower bound - The continuously compounded return is unbounded below, but the simple return is not. - A positive ending price makes the simple return greater than −100%, however close the price gets to zero.

  2. 2. For 36 independent monthly active returns, the mean is 0.80% and sample standard deviation is 2.40%. At 5%, test H₀: μ ≤ 0 against H₁: μ > 0; the one-tailed critical t-value is 1.69.

    • A. Fail to reject the null because 0.80% is economically small
    • B. Fail to reject the null because the test statistic is below 1.96
    • C. Reject the null in favor of a positive mean return
    Show answer

    Answer: C. Reject the null in favor of a positive mean return

    Reject the null in favor of a positive mean return - The standard error is 2.40%/√36 = 0.40%, giving a t-statistic of 0.80%/0.40% = 2.00. - Because 2.00 exceeds the supplied one-tailed critical value of 1.69, reject H₀ at 5% in favor of a positive mean return. Fail to reject the null because 0.80% is economically small - Economic size does not determine whether the sample mean is statistically significant in this hypothesis test. - The mean of 0.80% produces a t-statistic of 2.00, above the rejection threshold of 1.69. Fail to reject the null because the test statistic is below 1.96 - The test statistic is 2.00, so it is not below 1.96. - Moreover, 1.96 is not the supplied one-tailed t critical value; the applicable threshold is 1.69.

  3. 3. An analyst obtains a p-value of 0.030 for a two-sided test conducted at the 5% significance level. The analyst should most likely:

    • A. Reject the null hypothesis
    • B. Fail to reject the null hypothesis
    • C. Accept the null hypothesis as true
    Show answer

    Answer: A. Reject the null hypothesis

    Reject the null hypothesis - The p-value of 0.030 is below the 0.050 significance level. - The test result is therefore statistically significant, so the analyst rejects the null hypothesis. Fail to reject the null hypothesis - That decision would apply if the p-value were greater than 0.050, not 0.030. - Choosing it reverses the decision rule for this test. Accept the null hypothesis as true - A hypothesis test cannot establish that the null hypothesis is true. - Even when a p-value exceeds the significance level, the decision is to fail to reject, not to accept the null hypothesis.

  4. 4. A deposit earns a nominal annual rate of 7.20%, compounded monthly. The calculated value of its effective annual return is closest to:

    • A. 7.20%
    • B. 7.44%
    • C. 7.47%
    Show answer

    Answer: B. 7.44%

    7.44% - The monthly rate is 0.60% (7.20% ÷ 12). - Compounding for 12 months gives (1 + 0.072/12)^12 − 1 ≈ 7.44%. 7.20% - This is the stated nominal annual rate, not the effective annual return. - It omits the interest earned on interest during the 12 monthly compounding periods. 7.47% - This results from using continuous compounding: e^0.072 − 1 ≈ 7.47%. - The deposit compounds monthly, so its effective annual return is 7.44% instead.

  5. 5. An investor deposits USD 1,000 at the beginning of each quarter for two years. At 8% nominal compounded quarterly, the value immediately after the eighth quarter's interest is credited is closest to:

    • A. USD 8,000.00
    • B. USD 8,582.97
    • C. USD 8,754.63
    Show answer

    Answer: C. USD 8,754.63

    USD 8,754.63 - The eight beginning-of-quarter deposits form an annuity due earning 8% ÷ 4 = 2% per quarter. - Immediately after the eighth quarter’s interest is credited, their value is USD 1,000(1.02⁸ − 1) ÷ 0.02 = USD 8,754.63. USD 8,000.00 - This is eight deposits × USD 1,000 = USD 8,000, with no interest included. - It misses the interest earned because every deposit is made before a quarter’s interest is credited. USD 8,582.97 - USD 1,000[(1.02⁸ − 1) ÷ 0.02] = USD 8,582.97 treats the deposits as an ordinary annuity made at quarter-end. - Beginning-of-quarter deposits each earn one additional quarter of interest, requiring multiplication by 1.02.

  6. 6. An investor deposits USD 10,000 for 18 months at a stated annual rate of 6%, compounded monthly. The calculated future value is closest to:

    • A. USD 10,900
    • B. USD 10,913
    • C. USD 10,939
    Show answer

    Answer: C. USD 10,939

    USD 10,939 - The monthly rate is 0.5% (6% ÷ 12), and 18 months gives 18 compounding periods. - The future value is USD 10,000 × (1.005)^18 ≈ USD 10,939. USD 10,900 - This comes from simple interest: USD 10,000 × [1 + (0.06 × 1.5)] = USD 10,900. - It is too low because it omits the interest earned on interest during the 18 monthly periods. USD 10,913 - This comes from using annual rather than monthly compounding: USD 10,000 × (1.06)^1.5 ≈ USD 10,913. - The stated rate is compounded monthly, so the calculation must use 0.005 per month for 18 periods.

  7. 7. For 36 independent monthly returns, the sample mean is 0.8% and sample standard deviation is 2.4%. Testing H0: μ ≤ 0 against Ha: μ > 0 at 5%, with critical t = 1.690, the analyst should:

    • A. Reject the null hypothesis
    • B. Fail to reject because the test is one-tailed
    • C. Fail to reject because the sample mean is below 1.0%
    Show answer

    Answer: A. Reject the null hypothesis

    Reject the null hypothesis - The standard error is 2.4%/√36 = 0.4%, so the t-statistic is (0.8% − 0)/0.4% = 2.00. - Because 2.00 exceeds the stated 5% critical t-value of 1.690, reject H0 in favor of μ > 0. Fail to reject because the test is one-tailed - The alternative μ > 0 places the rejection region in the upper tail; being one-tailed does not itself justify failing to reject. - The calculated t-statistic of 2.00 is above the stated upper-tail cutoff of 1.690. Fail to reject because the sample mean is below 1.0% - The null hypothesis compares μ with 0, not with a 1.0% threshold. - Although the sample mean is 0.8%, its t-statistic of 2.00 exceeds the critical value of 1.690.

  8. 8. A portfolio's mean annual return is 8.00% from 36 independent observations, with sample standard deviation 12.00%. Using a 1.96 critical value, the calculated 95% confidence interval is closest to:

    • A. 7.35% to 8.65%
    • B. 4.08% to 11.92%
    • C. 3.30% to 12.70%
    Show answer

    Answer: B. 4.08% to 11.92%

    4.08% to 11.92% - The standard error is 12.00% ÷ √36 = 2.00% because the sample contains 36 independent observations. - Using the specified critical value gives 8.00% ± 1.96(2.00%), or 4.08% to 11.92%. 7.35% to 8.65% - This interval results from dividing the standard deviation by 36 instead of √36. - That slip gives 8.00% ± 1.96(12.00% ÷ 36), which rounds to 7.35% to 8.65%. 3.30% to 12.70% - This interval results from using 25 instead of 36 observations when calculating the standard error. - With 25 observations, 8.00% ± 1.96(12.00% ÷ √25) rounds to 3.30% to 12.70%.

Flashcards: Penalized regression, SVM, KNN, CART and random forests

  1. 1. What are reinforced learning algorithms?

    Show answer

    programs that learn from their own prediction errors

  2. 2. What is supervised learning?

    Show answer

    uses labeled training data to guide the ML program toward superior forecasting accuracy

  3. 3. What is LASSO?

    Show answer

    penalized regression in which the penalty term comprises summing the absolute values of the regression coefficients

  4. 4. What are typical tasks for supervised learning?

    Show answer

    classification and regression

  5. 5. What is accuracy?

    Show answer

    the % of correctly predicted classes out of total predictions

  6. 6. When given the prob. of correct call, IC =

    Show answer

    \= 2(% correct) - 1

  7. 7. What are the investment applications of KNN?

    Show answer

    - predicting bankruptcy - assigning a bond to a ratings class - predicting stock prices - creating customized indices

  8. 8. What are the investment applications of random forest?

    Show answer

    - factor-based asset allocation - prediction models for success of an IPO

  9. 9. When is LASSO useful?

    Show answer

    in building parsimonious models through feature reduction

  10. 10. What kind of mean is a F1 score?

    Show answer

    the harmonic mean of recall and precision

Everything you need for CFA prep

Get full Oncourse access with lessons, practice questions, flashcards and AI study tools.

GET STARTED FOR FREE
75% OFFLimited time offer
GET 75% OFF