Conformal Prediction#

CV+ Mondrian conformal prediction sets and intervals.

class nestkit.conformal.MondrianClassifierConformal[source]#

Bases: object

Class-conditional (Mondrian) conformal prediction sets.

Uses nonconformity score s(x, y) = 1 - p_hat(y | x) and computes a separate quantile threshold per class for class-conditional coverage.

static fit(oof_probas, oof_y_true, classes, alpha=0.1)[source]#

Compute per-class q-hat from OOF nonconformity scores.

Parameters:

  • oof_probas (ndarray of shape (n_cal, n_classes)) – Out-of-fold predicted probabilities (calibrated or raw).

  • oof_y_true (ndarray of shape (n_cal,)) – True labels for calibration samples.

  • classes (ndarray of shape (n_classes,)) – Ordered class labels (matching columns of oof_probas).

  • alpha (float) – Significance level (default 0.1 for 90% target coverage).

Return type:

ClassifierConformalResult

static predict(probas, conformal_result, classes=None)[source]#

Generate prediction sets for test data.

A class c is included in the prediction set for sample i if 1 - p_hat(c | x_i) <= q_hat[c].

Parameters:

  • probas (ndarray of shape (n_test, n_classes)) – Predicted probabilities (calibrated or raw, matching fit).

  • conformal_result (ClassifierConformalResult) – Result from fit().

  • classes (ndarray of shape (n_classes,) or None, optional) – Ordered class labels matching the columns of probas. When provided, prediction sets contain actual class labels instead of column indices.

Returns:

prediction_sets: list of lists (class labels if classes is provided, otherwise column indices). set_sizes: ndarray of int. is_uncertain: bool ndarray (True where set_size > 1). is_empty: bool ndarray (True where set_size == 0).

Return type:

dict

class nestkit.conformal.MondrianRegressorConformal[source]#

Bases: object

Mondrian conformal prediction intervals conditioned on predicted value.

Bins OOF predictions into equal-frequency groups and computes per-bin residual quantiles, yielding tighter intervals in easy-to-predict regions and wider intervals elsewhere.

static fit(oof_predictions, oof_residuals, alpha=0.05, n_bins=5, min_bin_size=20)[source]#

Compute per-bin residual quantiles from OOF data.

Parameters:

  • oof_predictions (ndarray of shape (n_cal,)) – Out-of-fold point predictions.

  • oof_residuals (ndarray of shape (n_cal,)) – Signed residuals y_true - y_pred.

  • alpha (float) – Significance level (default 0.05 for 95% coverage).

  • n_bins (int) – Number of equal-frequency bins for Mondrian conditioning.

  • min_bin_size (int) – Minimum calibration samples per bin; smaller bins are merged.

Return type:

RegressorConformalResult

static predict(test_predictions, conformal_result)[source]#

Generate per-bin prediction intervals for test predictions.

Parameters:

  • test_predictions (ndarray of shape (n_test,)) – Point predictions for test data.

  • conformal_result (RegressorConformalResult) – Result from fit().

Returns:

lower: ndarray of lower bounds. upper: ndarray of upper bounds. bin_assignments: ndarray of int bin indices.

Return type:

dict

class nestkit.conformal.ClassifierConformalResult(alpha, qhat_per_class, n_calibration_per_class)[source]#

Bases: object

Per-class quantile thresholds from CV+ Mondrian conformal calibration.

Parameters:
alpha#

Significance level (miscoverage rate).

Type:

float

qhat_per_class#

Mondrian q-hat threshold for each class.

Type:

ndarray of shape (n_classes,)

n_calibration_per_class#

Number of calibration samples used per class.

Type:

ndarray of shape (n_classes,)

class nestkit.conformal.RegressorConformalResult(alpha, n_bins, bin_edges, bin_quantiles, bin_counts, fallback_quantiles)[source]#

Bases: object

Per-bin residual quantiles from Mondrian regression conformal calibration.

Parameters:
alpha#

Significance level (miscoverage rate).

Type:

float

n_bins#

Number of Mondrian bins (after any merging).

Type:

int

bin_edges#

Bin edges computed from quantiles of OOF predictions.

Type:

ndarray of shape (n_bins + 1,)

bin_quantiles#

(q_lo, q_hi) residual quantile pair per bin.

Type:

list of (float, float)

bin_counts#

Number of calibration samples in each bin.

Type:

ndarray of shape (n_bins,)

fallback_quantiles#

Global residual quantiles used for extrapolation.

Type:

tuple of (float, float)