In-sample vs out-of-sample forecasts and RMSE

In-sample vs out-of-sample forecasts and RMSE

Published September 28, 2026

In-sample vs out-of-sample forecasts and RMSE

On this page

Learning Outcome Statement

Contrast in-sample and out-of-sample forecasts and compare the forecasting accuracy of different time-series models based on the root mean squared error criterion.

On the exam. The question gives a time-series forecasting vignette with actual outcomes, model forecasts, estimation and holdout periods. It asks you to classify forecasts, calculate RMSE, and identify the more accurate model in a calculation or multiple-choice format.

The analyst estimates the model with an estimation sample, then asks it to forecast observations that were not used during estimation. An in-sample forecast describes an observation used to fit the model, while an out-of-sample forecast tests the model on a held-out observation. You care about the distinction because a model can fit its learning sample closely and still miss what comes next.

FeatureIn-sampleOut-of-sample
ObservationsUsed to estimate modelHeld out from estimation
ForecastsFor estimation observationsFor held-out observations
Accuracy readingFit can look optimisticTests evaluation performance

The estimation sample tells you how well the model fits what it learned. The holdout sample tells you how well it performs on later observations, so use the holdout result when selecting between competing models. Keep the comparison tied to the question's purpose: a strong fit describes the data that shaped the model, while a useful forecast must also perform on data kept aside. That separation helps you avoid rewarding a model simply for matching its own estimation history.

ComparisonEstimation sampleHoldout sample
Data usedObservations used to fit the modelLater observations kept out of estimation
Forecast readingFit can look optimisticUse the result when selecting between models

Use $t$ to identify an observation in the evaluation sample. The actual outcome at that observation is $y_t$. The corresponding forecast is $\hat{y}_t$. Their difference, the forecast error, is $e_t=y_t-\hat{y}_t$. The count of evaluation observations is $n$.

Root mean squared error, RMSE, turns those forecast errors into one measure in the original forecast units.

$$ \mathrm{RMSE}=\sqrt{\frac{\sum_{t=1}^{n}e_t^2}{n}} $$

Squaring makes positive and negative misses contribute equally, and it makes a large miss count more. The summation combines the observations. Dividing by $n$ uses the population-style denominator for forecast-error RMSE, not $n-1$. The square root returns the result to the original units. A positive $e_t$ means the model underpredicted, while a negative $e_t$ means it overpredicted. Compare RMSEs only when the models use the same frequency, horizon, and observations.

The calculation path is short.

Loading diagram…

Read the process diagram from the split through the final comparison. The estimation observations support fitting and in-sample forecasts; later observations remain unavailable until evaluation. For each period, compare the forecast with the actual outcome, then square, average, and take the root. The final ranking is meaningful only when the models share the same frequency, horizon, and evaluation data. This sequence keeps fit separate from evidence about forecasting accuracy.

.

The figure makes the time boundary concrete: the estimation segment comes first, and the held-out segment supplies the later evaluation observations. Forecasts in that second segment are genuinely out of sample because their outcomes did not help estimate the model. When reading a vignette, locate that boundary before doing arithmetic. Then use only the designated evaluation outcomes and matching forecasts to calculate each model's errors and compare accuracy.

Worked example

You are handed a portfolio manager's monthly net-subscription forecasts for an exchange-traded fund. The analyst used 4 monthly observations to estimate each model and held out 3 later observations. The exam asks for each model's out-of-sample RMSE and the better forecast on the common evaluation sample. All outcomes and forecasts are measured in USD millions per month.

GivenValue
Estimation sample4 monthly observations
Holdout sample3 monthly observations
Actual holdout outcomes120, 125, 130 USD millions/month
Model A forecasts117, 121, 126 USD millions/month
Model B forecasts119, 124, 129 USD millions/month

The bars show out-of-sample RMSE in USD millions per month. Model A is on the left, and Model B is on the right. Read their heights as a visual summary of the calculation, not as a new measure: each bar represents the square root of the average squared holdout error. Model A's larger bar reflects its three misses of 3, 4, and 4, while Model B's equal errors of 1 produce the shorter bar. The table that follows gives the precise values for the ranking.

Loading diagram…

The chart emphasizes the size of the gap between the models, while the exact RMSE values let you state that gap precisely. Since both models use the same three monthly observations and the same units, the comparison is fair. A lower bar means smaller typical error under the RMSE calculation on this held-out sample; it does not guarantee the same ranking for every future observation. Check the reported figures before choosing the better model.

ModelOut-of-sample RMSE
Model A3.6968 USD millions/month
Model B1.0000 USD millions/month
  1. Select the holdout observations because the question asks for out-of-sample accuracy. Subtract each forecast from its actual outcome. Model A's errors are 3.00, 4.00, and 4.00. Model B's errors are 1.00, 1.00, and 1.00 USD millions per month.
  2. Square and add the errors so opposite signs cannot cancel. The squared-error totals are 41.00 for Model A and 3.00 for Model B.
  3. Divide by the common count, $n=3$, and take the square root. The substitutions are $\sqrt{\frac{41}{3}}=3.6968$ for Model A and $\sqrt{\frac{3}{3}}=1.0000$ for Model B.

Model B is the better choice on this evaluation sample because its RMSE of 1.0000 is below Model A's 3.6968.

Changed condition

Only the last actual holdout outcome changes in the shock case. The forecasts and the monthly frequency stay unchanged.

Holdout inputBeforeShock case
Actual outcome130118
Model A forecast126126
Model B forecast129129

Predict the movement before calculating. The actual outcome falls while both forecasts stay fixed. Because actual minus forecast becomes negative, each model changes from underprediction to overprediction. Model B's absolute miss becomes larger than Model A's, so its RMSE should rise more and Model A should rank better. Now compute the new squared errors and rank the models before reading on.

Common error

A candidate selects the model with the lowest in-sample RMSE because it appears to fit the historical data best. That feels right because closer fitted values look like stronger forecasts. The question identifies a held-out evaluation period, so the relevant comparison is out-of-sample RMSE calculated on the same observations.

Calculator

KeysDisplay
3 ENTER3.00
33.00
÷1.00
2ND x^21.0000

Check yourself

Practice

A portfolio manager estimates competing monthly net-subscription models. The models use 4 observations for estimation and evaluate later forecasts on 3 held-out observations. Actual holdout outcomes are 120, 125, and 130 USD millions per month. Model A forecasts 117, 121, and 126. Model B forecasts 119, 124, and 129.

Question 1

Which classification is correct?

Question 2

Using Model B's 3 held-out errors, which out-of-sample RMSE is correct?

Question 3

Before comparing the models' RMSEs, which condition must hold?

Question 4

What conclusion follows from the holdout RMSEs?

In the shock case, the actual holdout outcome is 118 while the forecasts remain 126 for Model A and 129 for Model B. Which statement is correct?

Revision snapshot

In-sample forecasts use observations used to estimate the model. Out-of-sample forecasts use observations held out from estimation. Forecast error is actual minus forecast.

$$ \mathrm{RMSE}=\sqrt{\frac{\sum_{t=1}^{n}e_t^2}{n}} $$

Use the population-style denominator $n$ for forecast-error RMSE, not $n-1$. Compare RMSEs only when models are evaluated over the same frequency, horizon, and observations. Lower out-of-sample RMSE indicates better forecast accuracy on the evaluation sample. Lower in-sample RMSE alone can reflect overfitting. On the exam, classify the sample split first, then calculate RMSE on the common evaluation sample.

Unlock the full lesson and continue reading

Signup to continue reading this lesson and unlimited access questions, flashcards, AI notes, and more

Scan to download app

Scan to download
UNLOCK FREE ACCESS

Practice Questions: In-sample vs out-of-sample forecasts and RMSE

  1. 1. An investor deposits USD 100,000 for three years at a stated annual rate of 4.25%, compounded monthly. The calculated future value is closest to:

    • A. USD 112,750
    • B. USD 113,638
    • C. USD 113,300
    Show answer

    Answer: B. USD 113,638

    USD 113,638 - Monthly compounding uses a periodic rate of 0.0425/12 over 36 months, giving USD 100,000(1 + 0.0425/12)^36 ≈ USD 113,573. - USD 113,638 is the closest listed value, although it is not the calculated future value. USD 112,750 - This comes from using simple interest: USD 100,000 × (1 + 0.0425 × 3) = USD 112,750. - It omits the interest earned on earlier months’ interest. USD 113,300 - This comes from annual rather than monthly compounding: USD 100,000(1.0425)^3 ≈ USD 113,300. - The stated rate is compounded monthly, so the calculation requires 36 compounding periods, not three.

  2. 2. A fund needs USD 2,000,000 in four years. It invests USD 400,000 today and USD 300,000 at each year-end for three years. At 5% annually, the additional amount required today is closest to:

    • A. USD 428,400
    • B. USD 449,900
    • C. USD 520,700
    Show answer

    Answer: A. USD 428,400

    USD 428,400 - The scheduled deposits grow to USD 1,479,240 in year four: USD 400,000(1.05)^4 + USD 300,000[(1.05)^3 + (1.05)^2 + 1.05]. - The year-four shortfall is USD 2,000,000 − USD 1,479,240 = USD 520,760, whose value today is USD 520,760/(1.05)^4 ≈ USD 428,431. USD 449,900 - Discounting the USD 520,760 shortfall for three years instead of four gives approximately USD 449,852, closest to USD 449,900. - The additional investment is made today, so it earns interest for all four years before the fund needs the money. USD 520,700 - This is close to the year-four shortfall of USD 520,760, rather than the amount required today. - A deposit made today grows at 5% for four years, so the future shortfall must be discounted by (1.05)^4.

  3. 3. A bank quotes a nominal annual rate of 6.00%, compounded monthly. The calculated effective annual rate is closest to:

    • A. 6.00%
    • B. 6.17%
    • C. 6.36%
    Show answer

    Answer: B. 6.17%

    6.17% - The monthly rate is 6.00% ÷ 12 = 0.50%. - Compounding that rate for 12 months gives \((1 + 0.06/12)^{12} - 1 \approx \mathbf{6.17\%}\). 6.00% - 6.00% is the quoted nominal annual rate, not the effective annual rate. - It omits the interest earned on earlier months’ interest. 6.36% - 6.36% results from adding \(6.00\% + (6.00\%)^2\). - That calculation uses the full annual rate as an extra compounding gain instead of applying the 0.50% monthly rate for 12 months.

  4. 4. An analyst simulates a stock price using a lognormal model with a normally distributed continuously compounded return. Which conclusion is most appropriate?

    • A. It cannot produce a negative simulated price.
    • B. It produces a normally distributed simulated price.
    • C. It produces an arithmetic return with zero skewness.
    Show answer

    Answer: A. It cannot produce a negative simulated price.

    It cannot produce a negative simulated price. - The simulated price is \(PT=P0e^{r{cc}}\), where \(P0\) is positive and the exponential is always positive. - Thus, even a very negative normally distributed continuously compounded return produces a price above zero. It produces a normally distributed simulated price. - The continuously compounded return is normally distributed in this model, not the simulated price. - Exponentiating that return produces a lognormal price distribution. It produces an arithmetic return with zero skewness. - The arithmetic return is \(e^{r{cc}}-1\), so it cannot fall below −100%. - This shifted lognormal return is generally positively skewed, not symmetric with zero skewness.

  5. 5. An analyst models continuously compounded returns as normally distributed. The corresponding asset-price distribution is best described as:

    • A. Prices are normally distributed and may be negative
    • B. Prices are lognormally distributed and remain nonnegative
    • C. Prices are uniformly distributed around their expected value
    Show answer

    Answer: B. Prices are lognormally distributed and remain nonnegative

    Prices are lognormally distributed and remain nonnegative - A continuously compounded return is \(r=\ln(PT/P0)\), so the future price is \(PT=P0e^r\). - If \(r\) is normal, exponentiating it makes \(PT\) lognormal and strictly positive when \(P0>0\). Prices are normally distributed and may be negative - The normal distribution applies to continuously compounded returns, not directly to price levels. - Although a normal return can be negative, \(P0e^r\) cannot be negative. Prices are uniformly distributed around their expected value - Exponentiating a normally distributed return produces a lognormal, not uniform, price distribution. - The resulting prices are bounded below by zero, rather than uniformly spread around their expected value.

  6. 6. A portfolio report presents performance after transaction costs but before investment-management fees. The reported measure is best classified as:

    • A. Net return
    • B. Gross-of-fees return
    • C. Real return
    Show answer

    Answer: B. Gross-of-fees return

    Gross-of-fees return - The report deducts transaction costs but not investment-management fees, so it presents a gross-of-fees return. - “Gross-of-fees” refers to management fees; it does not mean that trading costs are excluded. Net return - A net return deducts investment-management fees, but the report presents performance before those fees. - Deducting transaction costs alone does not make the reported measure net of management fees. Real return - A real return adjusts for inflation, not for investment-management fees. - The report describes its treatment of transaction costs and fees but gives no inflation adjustment.

  7. 7. An investment committee wants to assess a manager's investment skill independently of the management fee negotiated by each client. The committee should most likely compare:

    • A. Net-of-fees return
    • B. Gross-of-fees return
    • C. Real net-of-fees return
    Show answer

    Answer: B. Gross-of-fees return

    Gross-of-fees return - Gross-of-fees return measures the manager’s investment performance before each client’s negotiated management fee is deducted. - Comparing it across clients keeps differences in fee agreements from obscuring the committee’s assessment of investment skill. Net-of-fees return - Net-of-fees return subtracts the management fee, so clients with different negotiated fees can have different returns from the same investment performance. - It is useful for assessing what a client retains, but not for isolating the manager’s skill from the fee agreement. Real net-of-fees return - Real net-of-fees return reflects both management fees and an adjustment for inflation. - Neither adjustment helps the committee isolate investment skill from the client-specific fees it wants to set aside.

Flashcards: In-sample vs out-of-sample forecasts and RMSE

  1. 1. Three types of momentum indicators

    Show answer

    - earnings surprise = reported EPS - expected EPS - standardized unexpected earnings (SUE) = (earnings surprise)/(stdev of earnings surprise) - relative strength: patterns of persistence or reversal may exist in stock returns

  2. 2. What does the root mean squared error (RMSE) criterion used for?

    Show answer

    to compare the accuracy of AR models in forecasting out-of-sample values

  3. 3. What is the tstat for seasonality?

    Show answer

    tstat = residual autocorrelation / (1/sqrt(T))

  4. 4. What is the procedure for determining the structure of an AR model?

    Show answer

    1\. estimate an AR model\ 2\. calculate the autocorrelatins for the model's residuals\ 3\. test whether the autocorrelations are different from zero\ 4\. revise the model if there are significant correlations

  5. 5. Describe the distribution characteristics of asset returns

    Show answer

    skewness and excess kurtosis (fat tails)

  6. 6. For invalid results between two data series, either..

    Show answer

    - if one data series has a unit root - both have a unit root and they are not cointegrated

  7. 7. For regression results between two data series to be valid, either

    Show answer

    - neither data series has a unit root - both have a unit root but they are cointegrated

  8. 8. What is the Breusch-Godfrey test for?

    Show answer

    test for multiple lag serial correlation

  9. 9. What is serial correlation?

    Show answer

    \- situation in which regression residual terms are correlated with one another (not independent)\ \- poses problem with regressions using time series data

Everything you need for CFA prep

Get full Oncourse access with lessons, practice questions, flashcards and AI study tools.

GET STARTED FOR FREE
75% OFFLimited time offer
GET 75% OFF