Callbacks#

Callback hooks that plug into the nested cross-validation loop.

class nestkit.callbacks.FoldCallback(*args, **kwargs)[source]#

Bases: Protocol

Runtime-checkable protocol for nested CV fold callbacks.

Any object that implements the five hook methods below can be used as a callback. The nested CV engine calls these hooks at well- defined points during execution.

on_outer_fold_start(fold_idx, train_idx, test_idx)[source]#: Called before inner search begins for an outer fold.

on_inner_search_complete(fold_idx, search)[source]#: Called after the inner hyperparameter search finishes.

on_post_processing_complete(fold_idx, artifacts)[source]#: Called after any post-processing (e.g., threshold tuning).

on_outer_fold_complete(fold_idx, result)[source]#: Called after the outer fold evaluation is complete.

on_nested_cv_complete(results)[source]#: Called once after all outer folds have been processed.

Examples

>>> class MyCallback:
...     def on_outer_fold_start(self, fold_idx, train_idx, test_idx):
...         print(f"Starting fold {fold_idx}")
...     def on_inner_search_complete(self, fold_idx, search): ...
...     def on_post_processing_complete(self, fold_idx, artifacts): ...
...     def on_outer_fold_complete(self, fold_idx, result): ...
...     def on_nested_cv_complete(self, results): ...

See also

LoggingCallback: Text-based logging alternative.

on_outer_fold_start(fold_idx, train_idx, test_idx)[source]#

on_inner_search_complete(fold_idx, search)[source]#

on_post_processing_complete(fold_idx, artifacts)[source]#

on_outer_fold_complete(fold_idx, result)[source]#

on_nested_cv_complete(results)[source]#

class nestkit.callbacks.CheckpointCallback(path)[source]#

Bases: object

Pickle intermediate fold results to disk after each outer fold.

After every outer fold, the fold result is saved as fold_<idx>.pkl inside the given directory. When the full nested CV completes, the final results object is saved as final_results.pkl.

Parameters:: path (str or pathlib.Path) – Directory in which checkpoint files are written. Created automatically (including parents) if it does not exist.

path#

Resolved checkpoint directory.

Type:: pathlib.Path

Examples

>>> cb = CheckpointCallback("/tmp/ncv_checkpoints")

on_outer_fold_start(fold_idx, train_idx, test_idx)[source]#

on_inner_search_complete(fold_idx, search)[source]#

on_post_processing_complete(fold_idx, artifacts)[source]#

on_outer_fold_complete(fold_idx, result)[source]#

on_nested_cv_complete(results)[source]#

class nestkit.callbacks.LoggingCallback(level=20)[source]#

Bases: object

Emit structured log messages at each nested CV lifecycle event.

Logs fold start (with train/test sizes), inner search completion (with best parameters and score), post-processing completion, fold completion (with elapsed time), and overall completion.

Parameters:: level (int, default=logging.INFO) – Python logging level for all emitted messages.

level#

Logging level.

Type:: int

_fold_start_times#

Mapping from fold index to wall-clock start time, used to compute elapsed seconds.

Type:: dict[int, float]

Examples

>>> import logging
>>> cb = LoggingCallback(level=logging.DEBUG)