Skip to content

Metrics

Training and external evaluation both rely on src.evaluation.metrics.compute_classification_metrics, but the stored shapes differ slightly between training artifacts and evaluation artifacts.

Shared scalar metrics

The scorer computes:

  • accuracy
  • balanced_accuracy
  • f1
  • precision
  • recall
  • mcc
  • kappa
  • auc_decision
  • auc_probability
  • pr_auc
  • specificity
  • total_tn
  • total_fp
  • total_fn
  • total_tp

specificity is only available for binary confusion matrices.

Training outputs

In per-model training metrics YAML files:

  • folds[] stores per-fold scalar metrics plus tn, fp, fn, tp
  • nested_cv stores aggregated outer-fold means
  • nested_cv.std stores aggregated outer-fold standard deviations
  • nested_cv.outer_train stores aggregated train metrics
  • nested_cv.inner_cv stores aggregated inner-CV diagnostics
  • nested_cv.confusion_matrices stores per-fold confusion dictionaries
  • nested_cv.tp, nested_cv.fp, nested_cv.fn, and nested_cv.tn store aggregate confusion totals

In the dataset summary CSV written by write_dataset_summary, confusion totals appear as:

  • total_tn
  • total_fp
  • total_fn
  • total_tp

That CSV is a flattened reporting view built from the richer per-model YAML files.

External evaluation outputs

src.cli.evaluate writes one YAML file per evaluated model. That YAML contains:

  • model metadata such as model_name, model_path, and test_csv
  • scalar metrics using the total_* confusion keys directly
  • identifier_column
  • records, one per scored row, with target, prediction, and correctness

When you evaluate a directory of models, the CLI also writes a summary CSV.

How to read the numbers

Accuracy

Useful sanity check, but it can hide skewed behavior on imbalanced assays.

Balanced accuracy

Better default comparison metric when one class is underrepresented.

F1

Useful when both false positives and false negatives matter.

Precision and recall

Read these together:

  • precision answers "How noisy is the positive hit list?"
  • recall answers "How many true actives are we missing?"

MCC

A good tie-breaker when you want one imbalance-aware summary statistic.

Kappa

Chance-adjusted agreement measure.

AUC metrics

  • auc_decision uses decision_function when the model exposes it
  • auc_probability uses predict_proba when the model exposes it

If the model lacks the required scoring method, the corresponding AUC field is None.

PR-AUC

Average precision from probabilities when available, otherwise from decision scores. This is often more informative than ROC-AUC on imbalanced screening problems.

Overfitting diagnostics in training runs

The current training outputs give you multiple ways to inspect overfitting:

  • compare nested_cv test means with nested_cv.outer_train.mean
  • inspect nested_cv.std for unstable outer-fold performance
  • inspect folds[].inner_cv.mean_train_score versus folds[].inner_cv.mean_test_score
  • inspect hold-out metrics, when configured, after the final refit