Learning Outcome Statement
Contrast in-sample and out-of-sample forecasts and compare the forecasting accuracy of different time-series models based on the root mean squared error criterion.
On the exam. The question gives a time-series forecasting vignette with actual outcomes, model forecasts, estimation and holdout periods. It asks you to classify forecasts, calculate RMSE, and identify the more accurate model in a calculation or multiple-choice format.
The analyst estimates the model with an estimation sample, then asks it to forecast observations that were not used during estimation. An in-sample forecast describes an observation used to fit the model, while an out-of-sample forecast tests the model on a held-out observation. You care about the distinction because a model can fit its learning sample closely and still miss what comes next.
| Feature | In-sample | Out-of-sample |
|---|---|---|
| Observations | Used to estimate model | Held out from estimation |
| Forecasts | For estimation observations | For held-out observations |
| Accuracy reading | Fit can look optimistic | Tests evaluation performance |
The estimation sample tells you how well the model fits what it learned. The holdout sample tells you how well it performs on later observations, so use the holdout result when selecting between competing models. Keep the comparison tied to the question's purpose: a strong fit describes the data that shaped the model, while a useful forecast must also perform on data kept aside. That separation helps you avoid rewarding a model simply for matching its own estimation history.
| Comparison | Estimation sample | Holdout sample |
|---|---|---|
| Data used | Observations used to fit the model | Later observations kept out of estimation |
| Forecast reading | Fit can look optimistic | Use the result when selecting between models |
Use $t$ to identify an observation in the evaluation sample. The actual outcome at that observation is $y_t$. The corresponding forecast is $\hat{y}_t$. Their difference, the forecast error, is $e_t=y_t-\hat{y}_t$. The count of evaluation observations is $n$.
Root mean squared error, RMSE, turns those forecast errors into one measure in the original forecast units.
$$ \mathrm{RMSE}=\sqrt{\frac{\sum_{t=1}^{n}e_t^2}{n}} $$
Squaring makes positive and negative misses contribute equally, and it makes a large miss count more. The summation combines the observations. Dividing by $n$ uses the population-style denominator for forecast-error RMSE, not $n-1$. The square root returns the result to the original units. A positive $e_t$ means the model underpredicted, while a negative $e_t$ means it overpredicted. Compare RMSEs only when the models use the same frequency, horizon, and observations.
The calculation path is short.
Loading diagram…
Read the process diagram from the split through the final comparison. The estimation observations support fitting and in-sample forecasts; later observations remain unavailable until evaluation. For each period, compare the forecast with the actual outcome, then square, average, and take the root. The final ranking is meaningful only when the models share the same frequency, horizon, and evaluation data. This sequence keeps fit separate from evidence about forecasting accuracy.
.
The figure makes the time boundary concrete: the estimation segment comes first, and the held-out segment supplies the later evaluation observations. Forecasts in that second segment are genuinely out of sample because their outcomes did not help estimate the model. When reading a vignette, locate that boundary before doing arithmetic. Then use only the designated evaluation outcomes and matching forecasts to calculate each model's errors and compare accuracy.
Worked example
You are handed a portfolio manager's monthly net-subscription forecasts for an exchange-traded fund. The analyst used 4 monthly observations to estimate each model and held out 3 later observations. The exam asks for each model's out-of-sample RMSE and the better forecast on the common evaluation sample. All outcomes and forecasts are measured in USD millions per month.
| Given | Value |
|---|---|
| Estimation sample | 4 monthly observations |
| Holdout sample | 3 monthly observations |
| Actual holdout outcomes | 120, 125, 130 USD millions/month |
| Model A forecasts | 117, 121, 126 USD millions/month |
| Model B forecasts | 119, 124, 129 USD millions/month |
The bars show out-of-sample RMSE in USD millions per month. Model A is on the left, and Model B is on the right. Read their heights as a visual summary of the calculation, not as a new measure: each bar represents the square root of the average squared holdout error. Model A's larger bar reflects its three misses of 3, 4, and 4, while Model B's equal errors of 1 produce the shorter bar. The table that follows gives the precise values for the ranking.
Loading diagram…
The chart emphasizes the size of the gap between the models, while the exact RMSE values let you state that gap precisely. Since both models use the same three monthly observations and the same units, the comparison is fair. A lower bar means smaller typical error under the RMSE calculation on this held-out sample; it does not guarantee the same ranking for every future observation. Check the reported figures before choosing the better model.
| Model | Out-of-sample RMSE |
|---|---|
| Model A | 3.6968 USD millions/month |
| Model B | 1.0000 USD millions/month |
- Select the holdout observations because the question asks for out-of-sample accuracy. Subtract each forecast from its actual outcome. Model A's errors are 3.00, 4.00, and 4.00. Model B's errors are 1.00, 1.00, and 1.00 USD millions per month.
- Square and add the errors so opposite signs cannot cancel. The squared-error totals are 41.00 for Model A and 3.00 for Model B.
- Divide by the common count, $n=3$, and take the square root. The substitutions are $\sqrt{\frac{41}{3}}=3.6968$ for Model A and $\sqrt{\frac{3}{3}}=1.0000$ for Model B.
Model B is the better choice on this evaluation sample because its RMSE of 1.0000 is below Model A's 3.6968.
Changed condition
Only the last actual holdout outcome changes in the shock case. The forecasts and the monthly frequency stay unchanged.
| Holdout input | Before | Shock case |
|---|---|---|
| Actual outcome | 130 | 118 |
| Model A forecast | 126 | 126 |
| Model B forecast | 129 | 129 |
Predict the movement before calculating. The actual outcome falls while both forecasts stay fixed. Because actual minus forecast becomes negative, each model changes from underprediction to overprediction. Model B's absolute miss becomes larger than Model A's, so its RMSE should rise more and Model A should rank better. Now compute the new squared errors and rank the models before reading on.
Common error
A candidate selects the model with the lowest in-sample RMSE because it appears to fit the historical data best. That feels right because closer fitted values look like stronger forecasts. The question identifies a held-out evaluation period, so the relevant comparison is out-of-sample RMSE calculated on the same observations.
Calculator
| Keys | Display |
|---|---|
| 3 ENTER | 3.00 |
| 3 | 3.00 |
| ÷ | 1.00 |
| 2ND x^2 | 1.0000 |
Check yourself
Practice
A portfolio manager estimates competing monthly net-subscription models. The models use 4 observations for estimation and evaluate later forecasts on 3 held-out observations. Actual holdout outcomes are 120, 125, and 130 USD millions per month. Model A forecasts 117, 121, and 126. Model B forecasts 119, 124, and 129.
Question 1
Which classification is correct?
Question 2
Using Model B's 3 held-out errors, which out-of-sample RMSE is correct?
Question 3
Before comparing the models' RMSEs, which condition must hold?
Question 4
What conclusion follows from the holdout RMSEs?
In the shock case, the actual holdout outcome is 118 while the forecasts remain 126 for Model A and 129 for Model B. Which statement is correct?
Revision snapshot
In-sample forecasts use observations used to estimate the model. Out-of-sample forecasts use observations held out from estimation. Forecast error is actual minus forecast.
$$ \mathrm{RMSE}=\sqrt{\frac{\sum_{t=1}^{n}e_t^2}{n}} $$
Use the population-style denominator $n$ for forecast-error RMSE, not $n-1$. Compare RMSEs only when models are evaluated over the same frequency, horizon, and observations. Lower out-of-sample RMSE indicates better forecast accuracy on the evaluation sample. Lower in-sample RMSE alone can reflect overfitting. On the exam, classify the sample split first, then calculate RMSE on the common evaluation sample.
Unlock the full lesson and continue reading
Signup to continue reading this lesson and unlimited access questions, flashcards, AI notes, and more