arviz_stats.loo_score

Contents

arviz_stats.loo_score#

arviz_stats.loo_score(data, var_name=None, kind='crps', pointwise=False, round_to=None, log_weights=None, pareto_k=None)[source]#

Compute CRPS or SCRPS with PSIS-LOO-CV weights.

Computes the continuous ranked probability score (CRPS) or its scale-invariant variant (SCRPS) using Pareto-smoothed importance sampling leave-one-out (PSIS-LOO-CV) weights. Both are returned as maximization scores where larger is better. The CRPS is negated to match this convention, which the SCRPS already satisfies by definition. This assumes that the PSIS-LOO-CV approximation is working well.

Parameters:
dataxarray.DataTree or InferenceData

Input data. It should contain the posterior_predictive, observed_data and log_likelihood groups. A posterior group is also required when log_weights and pareto_k are not provided, as it is used to compute the relative efficiency of the importance sampling estimate.

var_namestr, optional

The name of the variable in the log_likelihood group to use. If None, the only variable in the log_likelihood group is used (an error is raised if there are several) and assumed to match the observed_data and posterior_predictive names.

kindstr, default “crps”

The kind of score to compute. Available options are:

  • ‘crps’: continuous ranked probability score. Default.

  • ‘scrps’: scale-invariant continuous ranked probability score.

pointwisebool, default False

If True, include per-observation score values in the return object.

round_toint or str, optional

If integer, number of decimal places to round the result. If string of the form ‘2g’ number of significant digits to round the result. Defaults to None, which returns raw numbers.

log_weightsxarray.DataArray, optional

Pre-computed smoothed log weights from PSIS. Must be provided together with pareto_k. If not provided, PSIS will be computed internally.

pareto_kxarray.DataArray, optional

Pre-computed Pareto k-hat diagnostic values. Must be provided together with log_weights.

Returns:
collections.namedtuple

If pointwise is False (default), a namedtuple named CRPS or SCRPS with fields mean and se. If pointwise is True, the namedtuple also includes pointwise and pareto_k fields.

References

[1]

Bolin, D., & Wallin, J. (2023). Local scale invariance and robustness of proper scoring rules. Statistical Science, 38(1), 140–159. https://doi.org/10.1214/22-STS864 arXiv preprint https://arxiv.org/abs/1912.05642

[2]

Gneiting, T., & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359–378. https://doi.org/10.1198/016214506000001437

[3]

Taillardat M, Mestre O, Zamo M, Naveau P (2016). Calibrated ensemble forecasts using quantile regression forests and ensemble model output statistics. Mon Weather Rev 144(6):2375–2393. https://doi.org/10.1175/MWR-D-15-0260.1

[4]

Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432. https://doi.org/10.1007/s11222-016-9696-4 arXiv preprint https://arxiv.org/abs/1507.04544

[5]

Vehtari, A., et al. (2024). Pareto Smoothed Importance Sampling. Journal of Machine Learning Research, 25(72). https://jmlr.org/papers/v25/19-556.html arXiv preprint https://arxiv.org/abs/1507.02646

Examples

Compute scores and return the mean and standard error:

In [1]: from arviz_stats import loo_score
   ...: from arviz_base import load_arviz_data
   ...: dt = load_arviz_data("centered_eight")
   ...: loo_score(dt, kind="crps")
   ...: 
Out[1]: CRPS(mean=-6.2729052652508805, se=1.5227237760416075)

In [2]: loo_score(dt, kind="scrps")
Out[2]: SCRPS(mean=-2.2699810234628255, se=0.09431408374213679)
Contents