Introduction
Throughout this series we have progressively built an NFL Elo forecasting model, starting with a simple rating system before introducing home advantage, margin of victory, seasonal carryover and probability forecasting. Each extension improved the model while preserving the simplicity and interpretability that have made Elo one of the most widely used rating systems in sport.
That brings us to the final question in the series:
"How do we know whether our forecasting model is actually any good?"
Building a forecasting model is only the first half of the process. Once a model begins producing probabilities, we need objective ways to measure how well those probabilities describe real-world outcomes. Without evaluation, it is impossible to distinguish between a model that genuinely forecasts well and one that merely appears convincing.
In this article we evaluate the four Elo variants developed throughout the series using established forecasting metrics and calibration analysis. The probability equation remains unchanged; only the Elo ratings supplied to it differ between models. This allows us to measure the contribution of each structural improvement under identical conditions.
By the end of this article, we will have completed the full Elo modelling cycle: building ratings, converting those ratings into probabilities and objectively evaluating the quality of the resulting forecasts.
Why Forecast Evaluation Matters
It is tempting to judge a forecasting model simply by how often it predicts the winning team correctly. However, probability forecasts contain far more information than a binary prediction of win or loss.
Imagine two models that both predict the same team to win. One assigns a probability of 51%, while the other assigns 95%. If that team loses, the second model has made the much larger error because it expressed far greater confidence in an outcome that did not occur.
This is why forecasting models are evaluated using probability-based metrics rather than accuracy alone. Good forecasting models reward appropriate confidence and penalise overconfidence, producing probabilities that reflect the uncertainty inherent in sporting events.
"The goal of a forecasting model is not simply to predict winners, but to assign probabilities that faithfully represent uncertainty."
Choosing the Right Evaluation Metrics
Every forecasting model needs an objective way to measure its performance. Without quantitative evaluation, it is impossible to know whether one model genuinely produces better forecasts than another or whether any apparent improvement is simply due to chance.
The challenge is that probability forecasts cannot be judged using a single statistic. A useful evaluation should reward models that assign realistic probabilities, penalise overconfident predictions that prove incorrect, and reveal whether the predicted probabilities correspond closely to the outcomes observed in practice.
For this reason, forecasting models are typically evaluated using several complementary metrics, each measuring a different aspect of predictive performance.
"No single metric can fully describe the quality of a probability forecasting model."
Evaluation Metrics
Throughout this article we evaluate each Elo variant using the following measures:
- Log loss — evaluates the quality of the predicted probabilities while heavily penalising confident but incorrect predictions.
- Brier score — measures the average squared difference between predicted probabilities and the observed outcomes.
- Accuracy (50% threshold) — records how often the team assigned the higher probability wins.
- Average predicted probability — summarises the model's overall probability estimates.
- Average observed win rate — provides the corresponding historical outcome for comparison.
Log loss and Brier score are the primary measures because they evaluate the probability forecasts themselves. Accuracy is included as a familiar benchmark, but it cannot distinguish between cautious and overconfident predictions that produce the same number of correct winners.
"Good forecasting models predict both the correct outcome and an appropriate level of confidence."
Evaluating the Elo Models
To compare the four Elo variants fairly, every model must be evaluated under exactly the same conditions. The only difference between the models is how the pre-match Elo ratings were generated. Everything else remains unchanged.
Each model uses the same home advantage adjustment, the same Elo probability equation, the same historical NFL games and the same evaluation metrics. This ensures that any differences in performance reflect the structural improvements made to the Elo ratings rather than changes to the probability calculation or evaluation procedure.
"A fair comparison changes one part of the model at a time while keeping everything else constant."
The evaluation process is identical for every Elo variant:
- Load the pre-match Elo ratings.
- Apply the same home advantage adjustment.
- Convert Elo rating differences into win probabilities.
- Calculate the forecasting metrics.
- Summarise the results in a single comparison table.
The complete Python implementation is shown below.
# ============================================================
# ARTICLE 8 — CODE BLOCK 1
# Compare the forecasting performance of each Elo variant
# ============================================================
import polars as pl
# ------------------------------------------------------------------
# Elo models developed throughout this series
# ------------------------------------------------------------------
ELO_FILES = [
("Clean Elo", "nfl_elo_game_level.pq"),
("Home Advantage", "nfl_elo_game_level_home.pq"),
("Home + MOV", "nfl_elo_game_level_home_mov.pq"),
("Home + MOV + Season", "nfl_elo_game_level_home_mov_season.pq"),
]
# ------------------------------------------------------------------
# Constants used for every model evaluation
# ------------------------------------------------------------------
# Educational value used consistently throughout this series
HOME_ADVANTAGE = 65.0
# Standard Elo probability scale
ELO_SCALE = 400.0
# Prevent numerical issues when calculating log loss
EPS = 1e-15
# ------------------------------------------------------------------
# Convert Elo rating differences into probabilities
# ------------------------------------------------------------------
def elo_prob(expr):
return 1.0 / (1.0 + pl.lit(10.0).pow(-(expr / ELO_SCALE)))
# Store the evaluation summary for each model
rows = []
# ------------------------------------------------------------------
# Evaluate each Elo variant under identical conditions
# ------------------------------------------------------------------
for label, path in ELO_FILES:
# Load the completed Elo ratings
df = pl.read_parquet(path).filter(~pl.col("neutral"))
# Calculate the match outcome, adjusted Elo difference
# and predicted home win probability
df = df.with_columns(
(pl.col("home_score") > pl.col("away_score"))
.cast(pl.Int8)
.alias("home_win"),
(pl.col("home_elo_pre")
+ HOME_ADVANTAGE
- pl.col("away_elo_pre"))
.alias("elo_diff_adj"),
elo_prob(
pl.col("home_elo_pre")
+ HOME_ADVANTAGE
- pl.col("away_elo_pre")
).alias("p_home")
)
# Clip probabilities before calculating log loss
df = df.with_columns(
pl.col("p_home")
.clip(EPS, 1 - EPS)
.alias("p_clip")
)
# Calculate the forecasting metrics
log_loss = (
-pl.when(pl.col("home_win") == 1)
.then(pl.col("p_clip").log())
.otherwise((1 - pl.col("p_clip")).log())
).mean()
brier = (
(pl.col("p_home") - pl.col("home_win")) ** 2
).mean()
accuracy = (
((pl.col("p_home") >= 0.5) ==
(pl.col("home_win") == 1))
).mean()
# Store the summary statistics for this model
rows.append({
"model": label,
"games": df.height,
"log_loss": float(df.select(log_loss).item()),
"brier": float(df.select(brier).item()),
"accuracy_50pct": float(df.select(accuracy).item()),
"avg_pred": float(df.select(pl.mean("p_home")).item()),
"avg_actual": float(df.select(pl.mean("home_win")).item()),
})
# ------------------------------------------------------------------
# Create the final comparison table
# ------------------------------------------------------------------
summary = (
pl.DataFrame(rows)
.sort("log_loss")
)
print(summary)
Evaluation Results
The table below summarises the forecasting performance of each Elo variant when evaluated on the same set of historical NFL games using identical probability calculations and evaluation metrics.
model games log_loss brier accuracy_50pct avg_pred avg_actual
-------------------------------------------------------------------------------------
Home + MOV + Season 8642 0.636228 0.222764 0.642212 0.582228 0.570238
Clean Elo 8642 0.651110 0.229630 0.619995 0.583667 0.570238
Home Advantage 8642 0.651987 0.229995 0.619995 0.585452 0.570238
Home + MOV 8642 0.655034 0.229310 0.633650 0.576825 0.570238
Interpreting the Results
The evaluation shows that the structural improvements introduced throughout the series lead to measurable improvements in forecasting performance. The complete model, incorporating home advantage, margin of victory and seasonal carryover, achieves the strongest overall results across the primary probability forecasting metrics.
Rather than producing one dramatic improvement, each extension contributes a modest refinement to the underlying Elo ratings. As these refinements accumulate, the quality of the resulting probability forecasts also improves. This is exactly what we would expect from a well-designed modelling process.
"Robust forecasting models are usually built through many small improvements rather than one dramatic breakthrough."
This gradual progression has been a consistent theme throughout the series. Every structural enhancement was introduced because it reflected additional information about how NFL games are played, not because it happened to improve the historical results. Better forecasts are therefore a consequence of building a more realistic rating system, rather than optimising directly for the evaluation metrics.
The average predicted probabilities also remain close to the observed home win rate, indicating that the Elo probability mapping remains stable as the rating model becomes progressively more sophisticated.
Taken together, these results demonstrate that careful, interpretable improvements to the Elo framework can produce meaningful gains in forecasting quality while preserving the simplicity that makes the Elo system so effective.
Why Calibration Matters
A forecasting model should do more than identify the most likely winner. Its predicted probabilities should also accurately reflect the uncertainty surrounding each game. This property is known as calibration.
Calibration asks a simple but fundamental question:
"When a model predicts a 70% chance of winning, does the team actually win about 70% of the time?"
If the answer is yes across the full range of predicted probabilities, the model is well calibrated. If teams predicted to win 70% of the time actually win only 60%, the model is overconfident. If they win closer to 80%, the model is underconfident.
This distinction is important because probability forecasting is not simply about predicting winners. A model that consistently identifies the winning team may still produce poorly calibrated probabilities if it expresses either too much or too little confidence in its predictions.
"Good forecasting models predict both outcomes and the uncertainty surrounding those outcomes."
Calibration Curves
Calibration curves provide a simple visual way to assess this behaviour. Games are grouped according to their predicted probabilities, and the observed win rate within each group is compared with the model's average predicted probability.
A perfectly calibrated model would produce points that lie exactly on the diagonal line. Points above the diagonal indicate that the model has underestimated the true probability, while points below the diagonal indicate that it has been overconfident.
The calibration curves for the four Elo variants are shown below.
Generating the Calibration Curves
To produce the calibration curves, we evaluate each Elo variant in exactly the same way. The games are sorted by predicted home win probability and split into equal-sized groups. Within each group we calculate both the average predicted probability and the actual home win rate.
Using equal-sized groups keeps the calibration curve stable across the probability range. It avoids giving too much visual weight to extreme probability bands that may contain only a small number of games.
If a model is well calibrated, those two values should be close to one another across the full probability range. The following code generates the calibration chart used in this article.
# ============================================================
# ARTICLE 8 — CODE BLOCK 2
# Generate calibration curves for each Elo variant
# ============================================================
import polars as pl
import matplotlib.pyplot as plt
# ------------------------------------------------------------------
# Elo models developed throughout this series
# ------------------------------------------------------------------
ELO_FILES = [
("Clean Elo", "nfl_elo_game_level.pq"),
("Home Advantage", "nfl_elo_game_level_home.pq"),
("Home + MOV", "nfl_elo_game_level_home_mov.pq"),
("Home + MOV + Season", "nfl_elo_game_level_home_mov_season.pq"),
]
# ------------------------------------------------------------------
# Constants used for every model evaluation
# ------------------------------------------------------------------
HOME_ADVANTAGE = 65.0
ELO_SCALE = 400.0
# Number of equal-sized calibration groups
N_BINS = 20
# ------------------------------------------------------------------
# Convert Elo rating differences into probabilities
# ------------------------------------------------------------------
def elo_prob(expr):
return 1.0 / (1.0 + pl.lit(10.0).pow(-(expr / ELO_SCALE)))
# ------------------------------------------------------------------
# Build calibration data for every Elo variant
# ------------------------------------------------------------------
calibration_frames = []
for label, path in ELO_FILES:
df = (
pl.read_parquet(path)
.filter(~pl.col("neutral"))
.with_columns(
(pl.col("home_score") > pl.col("away_score"))
.cast(pl.Int8)
.alias("home_win"),
elo_prob(
pl.col("home_elo_pre")
+ HOME_ADVANTAGE
- pl.col("away_elo_pre")
).alias("p_home")
)
.sort("p_home")
.with_row_index("row_number")
.with_columns(
(
pl.col("row_number")
* N_BINS
/ pl.len()
)
.floor()
.clip(0, N_BINS - 1)
.cast(pl.Int8)
.alias("prob_bin")
)
)
calibration = (
df.group_by("prob_bin")
.agg(
pl.mean("p_home").alias("predicted"),
pl.mean("home_win").alias("actual"),
pl.len().alias("games")
)
.sort("prob_bin")
.with_columns(pl.lit(label).alias("model"))
)
calibration_frames.append(calibration)
calibration_df = pl.concat(calibration_frames)
# ------------------------------------------------------------------
# Plot the calibration curves
# ------------------------------------------------------------------
plt.figure(figsize=(10, 6))
for label, _ in ELO_FILES:
curve = (
calibration_df
.filter(pl.col("model") == label)
.sort("prob_bin")
)
plt.plot(
curve["predicted"].to_list(),
curve["actual"].to_list(),
linewidth=2,
label=label
)
plt.plot(
[0, 1],
[0, 1],
linestyle="--",
linewidth=1.5
)
plt.xlim(-0.05, 1.05)
plt.ylim(-0.05, 1.05)
plt.xlabel("Predicted home win probability")
plt.ylabel("Actual home win rate")
plt.title("Calibration Curves (Elo Variants, Home-Adjusted Probabilities)")
plt.legend()
plt.grid(alpha=0.3)
plt.tight_layout()
plt.savefig(
"article8_elo_calibration_curves.png",
dpi=200,
bbox_inches="tight"
)
plt.show()
Interpreting the Calibration Curves
The calibration curves show that all four Elo variants produce probabilities that are broadly consistent with the historical outcomes. Across the probability range where most NFL games occur, the predicted probabilities remain close to the ideal diagonal, indicating that the models provide well-calibrated forecasts.
Small departures from the diagonal are entirely expected. NFL games are inherently unpredictable, and each probability band contains only a finite number of historical observations. As a result, the observed win rates naturally fluctuate around the underlying probabilities rather than matching them exactly.
The complete model, incorporating home advantage, margin of victory and seasonal carryover, remains closest to the diagonal across most of the probability range. This complements its superior log loss and Brier score, demonstrating that the structural improvements introduced throughout the series have enhanced both the accuracy and the calibration of the resulting probability forecasts.
"Well-calibrated forecasting models do not eliminate uncertainty. They quantify it realistically."
Equally important is the absence of large or erratic calibration errors. The remaining deviations are smooth and systematic, suggesting that the Elo framework provides stable and interpretable probability estimates rather than overreacting to individual results or random variation.
Why This Matters
Throughout this series we have progressively improved the underlying Elo ratings before converting them into probability forecasts. The calibration analysis provides independent evidence that these structural improvements have produced forecasts that are not only more accurate, but also better aligned with the outcomes observed across thousands of historical NFL games.
This highlights one of the central ideas of predictive modelling: improving the representation of the underlying process leads naturally to better forecasts. The goal is not to optimise a single evaluation metric, but to build a model that captures the important characteristics of the system being modelled.
"Better models produce better forecasts because they better represent reality."
What Have We Learned?
This article brings the NFL Elo series to its conclusion. Over the course of eight articles we have progressed from building a simple rating system to developing a complete probability forecasting model and, finally, evaluating that model using objective forecasting metrics and calibration analysis.
"A forecasting model is only as valuable as the quality of the probabilities it produces."
Along the way we introduced home advantage, margin of victory, seasonal carryover, probability forecasting and calibration. Each addition represented a small, interpretable improvement to the underlying rating system, and together those improvements produced more accurate and better-calibrated probability forecasts.
More importantly, the series has illustrated a broader principle of predictive modelling. Strong forecasting systems are rarely created through a single breakthrough. They evolve through a sequence of carefully justified improvements, with every enhancement representing a better understanding of the process being modelled.
Key Takeaways
- Build models around the underlying process rather than around evaluation metrics.
- Estimate team strength before attempting to forecast outcomes.
- Convert model outputs into probabilities that explicitly represent uncertainty.
- Evaluate probability forecasts using multiple complementary metrics, not accuracy alone.
- Use calibration to verify that predicted probabilities correspond to observed outcomes.
- Improve models through small, interpretable refinements rather than unnecessary complexity.
Final Thoughts
Although this series has focused on NFL Elo ratings, the principles extend far beyond American football and far beyond Elo itself. The same ideas underpin forecasting models across many fields, including sport, finance, economics, medicine and machine learning.
The specific algorithms may change, but the modelling process remains remarkably consistent: build a model that captures the important characteristics of the underlying system, convert its outputs into meaningful predictions and evaluate those predictions objectively against real-world outcomes.
Elo is deliberately simple, and that simplicity is one of its greatest strengths. It provides a transparent framework for understanding how rating systems evolve, how probabilities are generated and how forecasting models should be evaluated. Those ideas form the foundation of many more sophisticated predictive systems.
"Better forecasts begin with better models. Better models begin with a better understanding of reality."
Whether you continue refining Elo or move on to more advanced forecasting techniques, we hope this series has provided both a practical framework for building rating systems and a deeper understanding of the principles that underpin probability forecasting.