Learning Outcome Statement
Describe supervised machine learning algorithms-including penalized regression, support vector machine, k-nearest neighbor, classification and regression tree, ensemble learning, and random forest-and determine the problems for which they are best suited.
On the exam: You get a supervised target, predictor structure and scale clues, and algorithm traits. You choose the best-suited method or interpret a ridge calculation in an item set or MCQ.
You start with observations where each known target is paired with predictors. Training uses those paired observations to learn a mapping that predicts the target for new predictor values. Each algorithm imposes a different way of finding that mapping. On the exam, read the clues about predictor structure, scale, and the need for interpretation, then choose the best fit or read a penalized coefficient.
| Method | Mechanism | Best fit |
|---|---|---|
| Penalized regression | Linear coefficients with shrinkage | Linear or high-dimensional structure |
| Support vector machine | Maximum-margin boundary, optional kernels | Margin-based nonlinear boundaries |
| K-nearest neighbor | Local distance prediction, scaled predictors | Local patterns |
| Classification and regression tree (CART) | Binary recursive splits, depth and pruning | Interpretable rules and interactions |
| Ensemble learning | Bagging in parallel, boosting sequentially | Combined base learners |
| Random forest | Bootstrap samples, random predictor subsets, many trees | Stable nonlinear prediction |
Do not choose the method with the fanciest name just because it sounds advanced. Match the mechanism to the case. An individual tree keeps its rules readable, while a forest gives up some transparency for steadier nonlinear prediction.
Ridge is the calculation the exam can ask you to perform. When the target is $Y$ and the predictor is $X$, the ridge coefficient, $\hat{\beta}_{\mathrm{ridge}}$, measures target-units per signal-unit. The sample covariance, $\operatorname{Cov}(X,Y)$, captures their joint movement, and the sample variance, $\operatorname{Var}(X)$, sets the predictor scale. The penalty, $\lambda$, uses the variance's units. Ridge shrinks regression coefficients toward zero, while lasso can set some coefficients exactly to zero.
$$ \hat{\beta}_{\mathrm{ridge}}(\lambda) = \frac{\operatorname{Cov}(X,Y)} {\operatorname{Var}(X)+\lambda} $$
Read the numerator first. It carries the sign and strength of the sample relationship. The variance sets the scale of an unpenalized slope, while $\lambda$ adds restraint to the denominator. As $\lambda$ increases, the absolute coefficient moves toward zero.
Loading diagram…
Use a support vector machine when the case calls for a separating boundary. It chooses a maximum-margin boundary, and a kernel lets that boundary represent nonlinear relationships.
For KNN, $k$ is the number of neighbors used for each prediction. For a new observation, the algorithm looks at nearby observations and uses their targets. Scale predictors before measuring distance, because unlike units can let one feature dominate. Choose $k$ to balance noise against local detail.
CART makes binary splits for classification or regression. The splits create readable rules and can capture interactions. Depth and pruning control overfitting.
Ensemble learning combines multiple base learners. Bagging trains learners in parallel on resampled data, whereas boosting trains sequentially so later learners correct earlier errors. Random forest combines bootstrap samples, random predictor subsets, and many decision trees to reduce correlated-tree variance.
Worked example
You are handed monthly sample estimates for a portfolio signal. The target is next-period excess return, and you need the ridge coefficient at the supplied penalty plus the direction across penalties. What does the model say about the signal weight?
| Input | Value |
|---|---|
| Target | Next-period excess return |
| Predictor | Signal |
| Data | Monthly sample |
| Covariance | 24 target-units times signal-units |
| Variance | 16 signal-units squared |
| Penalties | 0, 1, 4, 9 signal-units squared |
Loading diagram…
| Penalty, $\lambda$ | Ridge coefficient |
|---|---|
| 0 | 1.5000 |
| 1 | 1.4118 |
| 4 | 1.2000 |
| 9 | 0.9600 |
- Select the requested penalty, $\lambda=4$, so the denominator matches the case. This produces $\lambda=4$.
- Substitute the sample covariance and variance with that penalty. The calculation $\hat{\beta}_{\mathrm{ridge}}(4)=\frac{24}{16+4}=1.2000$ produces a coefficient of $1.2000$ target-units per signal-unit.
- Repeat the substitution at the other supplied penalties to test the direction rather than guess it. This produces $\hat{\beta}{\mathrm{ridge}}(0)=1.5000$, $\hat{\beta}{\mathrm{ridge}}(1)=1.4118$, $\hat{\beta}{\mathrm{ridge}}(4)=1.2000$, and $\hat{\beta}{\mathrm{ridge}}(9)=0.9600$.
The portfolio manager reads this as a positive signal weight. The falling sequence shows that a larger penalty restrains the signal weight.
Changed condition
The same analyst now sees weaker sample comovement between signal and excess return. The variance and penalty values stay fixed, so only covariance changes.
| Quantity | Original | Weaker covariance |
|---|---|---|
| $\operatorname{Cov}(X,Y)$ | 24 | 20 |
| $\hat{\beta}_{\mathrm{ridge}}(0)$ | 1.5000 | 1.2500 |
| $\hat{\beta}_{\mathrm{ridge}}(9)$ | 0.9600 | 0.8000 |
Predict a lower coefficient at every penalty before looking at the right column. The denominator is unchanged, while the smaller covariance lowers the numerator, so each coefficient should remain positive but move closer to zero. Before reading on, calculate $\hat{\beta}_{\mathrm{ridge}}(4)$ for the weaker covariance.
Common error
When you choose KNN from raw feature distances, you may trust the nearest observation because distance feels objective. That feels right because the algorithm appears to let the data decide without a model assumption. A vignette whose predictors use unlike units gives away the mistake: the large-unit feature dominates the distance and can decide which observations KNN calls nearest.
Calculator
| Keys | Display |
|---|---|
| 24 ENTER | 24.00 |
| 16 ENTER | 16.00 |
| 4 ENTER | 4.00 |
| 24 / ( 16 + 4 ) CPT | 1.2000 |
Check yourself
Practice
A portfolio analyst estimates next-period excess return from one signal using monthly sample data. The covariance between the signal and target is 24 target-units times signal-units, the signal variance is 16 signal-units squared, and the ridge penalty is 4 signal-units squared. A separate team studies local classification patterns with predictors in unlike units and compares bagging with boosting.
Question 1
Calculate the ridge coefficient at the stated penalty.
Question 2
As the ridge penalty moves from 0 to 9, which interpretation is correct?
Question 3
Which choice best fits the local classification problem with unlike predictor units?
Question 4
Which statement correctly distinguishes bagging from boosting?
A classification case has a nonlinear boundary, and the analyst wants a maximum-margin method that can use a kernel. Which method is best suited?
Revision snapshot
Ridge shrinks correlated regression coefficients toward zero. Lasso can set some coefficients exactly to zero.
$$ \hat{\beta}_{\mathrm{ridge}}(\lambda) = \frac{\operatorname{Cov}(X,Y)} {\operatorname{Var}(X)+\lambda} $$
The covariance sets direction, variance sets scale, and higher $\lambda$ increases shrinkage. SVM chooses a maximum-margin boundary and kernels handle nonlinear relationships. KNN makes local distance predictions. Scale predictors and choose $k$ to balance noise against local detail. CART makes binary splits for classification or regression. Depth and pruning control overfitting. Ensemble learning combines base learners. Bagging is parallel on resampled data. Boosting is sequential and corrects errors. Random forest combines bootstrap samples, random predictor subsets, and many trees to reduce correlated-tree variance. Select penalized regression for linear or high-dimensional structure, SVM for margin-based nonlinear boundaries, KNN for local patterns, CART for interpretable rules and interactions, and random forest for stable nonlinear prediction. Raw KNN distance with unlike units is the scaling trap. On the exam, match mechanism to vignette.
Unlock the full lesson and continue reading
Signup to continue reading this lesson and unlimited access questions, flashcards, AI notes, and more