Training Pipeline¶

The current training implementation lives in src.training.pipeline, src.training.model_runner, src.training.pipeline_steps, and src.training.reporting.

Canonical config files¶

configs/train.yaml: restored default config used by the CLI help
configs/train_example.yaml: starter example aligned with the same workflow
configs/train_example_dpk.yaml: starter example that also includes DeepPK outputs
configs/hyperparameters*.yaml: model metadata and parameter grids

The refreshed starter configs assume you first run:

python -m src.cli.process_features data/example/raw/psychlight_a.csv \
    --output-dir data/example/processed/psychlight_a

Config structure¶

Training configs contain three top-level sections: datasets, training, and models.

`datasets`¶

Each dataset entry describes one CSV to train on.

Key	Meaning
`name`	Friendly identifier used in logs and `--datasets` filtering
`type`	`descriptor` or `fingerprint`; legacy aliases still resolve through `resolve_dataset_type`
`path`	CSV path to load
`target_column`	Required label column
`feature_columns` / `drop_columns`	Explicit allowlist or denylist for feature selection
`feature_engineering`	Descriptor-only helpers such as `numeric_from_strings`
`variance_threshold`	Optional bool, numeric threshold, or dict form
`svd`	Optional bool, integer component count, or dict form
`training_overrides`	Deep-merged into the top-level `training` block for that dataset
`holdout_path` / `test_path` / `test_csv`	Optional held-out CSV scored after final refit

`training`¶

Key	Meaning
`random_state`	Shared seed for dataset prep, CV, samplers, and model builders
`outer_folds` / `inner_folds`	Nested CV fold counts
`sampling`	Sampler toggle plus strategy and params
`scaler`	Scaler toggle, import path, and params
`polynomial_features`	Optional polynomial feature expansion block
`pca`	Optional PCA block, adjusted to feature count when necessary
`grid_search`	Search kwargs plus `strategy: grid\|random` and optional `n_iter`
`outputs`	Artifact directories for models, metrics, and logs

`models`¶

Key	Meaning
`selection`	`all`, a single model key, or a list of model keys
`hyperparameters`	YAML file containing model metadata and parameter grids
`model_extension`	Output file extension for persisted pipelines

Actual pipeline order¶

The current ordered feature pipeline is built by compose_feature_pipeline_steps and then the estimator is appended in run_single_model.

For each model, the order is:

polynomial features, when enabled
non-variance, non-dimensionality pre-model steps
variance-threshold steps
scaler, only when the model metadata says requires_scaler: true
dimensionality-reduction pre-model steps such as SVD
PCA, when enabled
sampler, when enabled and the model metadata says use_smote: true
model

That ordering is important because the sampler is currently applied late in the pipeline, immediately before the estimator.

Search and metrics behavior¶

run_single_model performs:

outer StratifiedKFold
inner GridSearchCV or RandomizedSearchCV
full-dataset refit with the winning hyperparameters
optional hold-out scoring after the final refit

Training metrics are aggregated through aggregate_outer_results, which means:

nested_cv stores mean outer-fold test metrics
nested_cv.std stores outer-fold standard deviations
nested_cv.outer_train stores aggregated train metrics
nested_cv.inner_cv stores aggregated inner-CV diagnostics for the best params
nested_cv.confusion_matrices stores per-fold confusion dictionaries
nested_cv.tp, nested_cv.fp, nested_cv.fn, and nested_cv.tn are aggregate confusion totals

If return_train_score is omitted from training.grid_search, the search helper forces it on so the inner-CV diagnostics are always available.

Artifacts and paths¶

Per dataset run, the pipeline creates unique timestamped/hash directories under the configured outputs.

Current filename patterns:

model: <model>_model_<dataset_hash>_<timestamp>.<extension>
per-model metrics YAML: <model>_metrics_<dataset_hash>_<timestamp>.yaml
dataset summary YAML: <safe_dataset_name>_summary_<dataset_hash>_<timestamp>.yaml
dataset summary CSV: same base name, .csv
dataset log text: <safe_dataset_name>_<timestamp>_<dataset_hash>_logs.txt
pipeline overview CSV: pipeline_steps_<timestamp>.csv

ensure_unique_directory will suffix the run directory if a collision already exists, so repeated executions do not overwrite tracked artifacts.

Note

The shared training ledger is always models/tracking/training_runs.csv, even if training.outputs.log_dir points somewhere else.

Running training¶

Narrow first run:

python -m src.cli.train --config configs/train.yaml \
    --datasets psychlight_a_descriptors \
    --models logisticregression

Module entry point:

python -m src.training.train --config configs/train.yaml \
    --datasets psychlight_a_descriptors \
    --models logisticregression

Useful overrides:

--config: choose a different YAML file
--models: restrict to one or more models from the hyperparameter registry
--datasets: restrict to dataset names or dataset types
--skip-existing: consult models/tracking/training_runs.csv and skip completed dataset-hash/model pairs

The current src.cli.train output is a concise completion string such as:

Training complete for 1 dataset(s): psychlight_a_descriptors

It does not print the full summary payload as YAML.

Hold-out scoring¶

If a dataset entry provides holdout_path, test_path, or test_csv, the pipeline:

prepares the hold-out CSV with the same dataset rules
aligns columns to the training feature schema
evaluates the final refit model on the hold-out set
writes those hold-out metrics into the per-model metrics YAML, the dataset summary YAML, the dataset summary CSV, and the central ledger