Metrics¶

Scoring functions. The mathematical treatment — what each score is proper for and how the leaderboard uses it — is Methods: verification.

Deterministic¶

Point-forecast metrics. All functions take equal-length float64 arrays.

EmptyScoreError ¶

Bases: ValueError

A metric was requested over zero samples.

bias ¶

bias(pred: FloatArray, y: FloatArray) -> float

Mean error; positive means the forecast runs high.

mae_skill ¶

mae_skill(
    pred: FloatArray, y: FloatArray, reference: FloatArray
) -> float

1 - MAE/MAE_ref: positive beats the reference, 0 ties, negative loses.

pct_within ¶

pct_within(
    pred: FloatArray, y: FloatArray, tolerance: float
) -> float

Fraction of forecasts within tolerance of truth (consumer view).

pop_hit_rate ¶

pop_hit_rate(
    pop: FloatArray,
    occurred: FloatArray,
    threshold: float = 0.5,
) -> float

Fraction of correct rain/no-rain calls at a probability threshold.

Probabilistic¶

Probabilistic scores: CRPS, pinball, Brier, reliability, coverage, PIT.

scoringrules is wrapped entirely inside this module so it can be swapped out in one place if its API moves.

pinball_loss ¶

pinball_loss(
    y: FloatArray, quantile_pred: FloatArray, level: float
) -> float

Mean pinball (quantile) loss at one level in (0, 1).

crps_from_quantiles ¶

crps_from_quantiles(
    y: FloatArray,
    quantiles: FloatArray,
    levels: tuple[float, ...],
) -> float

Approximate CRPS by integrating pinball loss over probability levels.

crps_ensemble ¶

crps_ensemble(y: FloatArray, ensemble: FloatArray) -> float

Mean CRPS of an ensemble forecast (rows: cases, columns: members).

brier ¶

brier(pop: FloatArray, occurred: FloatArray) -> float

Brier score for probability-of-precipitation against binary outcomes.

reliability_bins ¶

reliability_bins(
    pop: FloatArray, occurred: FloatArray, n_bins: int = 10
) -> pl.DataFrame

Reliability table: forecast probability vs observed frequency per bin.

empirical_coverage ¶

empirical_coverage(
    y: FloatArray, lower: FloatArray, upper: FloatArray
) -> float

Fraction of truths inside [lower, upper].

pit_from_quantiles ¶

pit_from_quantiles(
    y: FloatArray,
    quantiles: FloatArray,
    levels: tuple[float, ...],
) -> FloatArray

Approximate PIT values by interpolating truth into the quantile grid.

Diebold-Mariano¶

Diebold-Mariano equal-predictive-accuracy test with the HLN correction.

Compares two aligned loss series (e.g. absolute errors of two methods on the same cases). Uses a Bartlett-kernel HAC variance so multi-step forecast horizons with serially correlated loss differentials are handled, and the Harvey-Leybourne-Newbold small-sample correction with a Student-t reference.

diebold_mariano ¶

diebold_mariano(
    loss_a: FloatArray,
    loss_b: FloatArray,
    horizon_steps: int = 1,
) -> DMResult

Test H0: equal expected loss. Negative statistic favors method A.