Data Preparation

This project separates raw chemistry data from model-ready feature tables. The current source of truth is src.cli.process_features for file generation and src.training.datasets.prepare_dataset for training-time ingestion.

Raw input expectations

python -m src.cli.process_features expects:

• a CSV with a SMILES column
• either an ID column or a name column

Any other columns are preserved in the generated outputs. That includes metadata such as Known, so training configs should explicitly drop non-feature columns.

Canonical example

python -m src.cli.process_features data/example/raw/psychlight_a.csv \
    --output-dir data/example/processed/psychlight_a

That command generates current-format files next to the checked-in legacy exports.

Generated files

The current CLI writes:

File	Source in code	Notes
<name>_descriptors.csv	RDKit descriptor block in src.cli.process_features	Includes the original columns, kekule_smiles, and descriptor columns from src/features/descriptors.py.
<name>_mordred.csv	Mordred descriptor block	Includes the original columns, kekule_smiles, and Mordred descriptors.
<name>_morgan.csv	Morgan fingerprint block	Includes the original columns, kekule_smiles, and morgan_bit_<i> columns.
<name>_dpk.csv	Optional DeepPK merge step	Written only when --deeppk-post is enabled.
<name>_dpk.smi	Optional DeepPK submission input	Exact SMILES file submitted to DeepPK.
<name>_deeppk_extra.csv	Optional DeepPK mismatch output	Written only when DeepPK returns unmatched rows.

Note

Older checked-in files in this repository may still use the legacy *_rdkit.csv descriptor name. The current CLI writes *_descriptors.csv.

CLI flags

python -m src.cli.process_features --help

Key flags:

• --output-dir: output directory, default data/processed
• --fingerprint-radius: Morgan radius, default 3
• --fingerprint-bits: number of fingerprint bits, default 1024
• --deeppk-post: enable DeepPK submission
• --deeppk-pred-type: DeepPK prediction mode, default admet
• --deeppk-email: optional contact email forwarded to DeepPK
• --deeppk-poll-interval: polling interval in seconds, default 60
• --deeppk-timeout: timeout in seconds, default 3600

What prepare_dataset does at training time

src.training.datasets.prepare_dataset performs the training-side cleanup:

1. Load the CSV from the dataset config.
2. Resolve target_column and either feature_columns or drop_columns.
3. Sanitize feature names with sanitize_feature_columns.
4. Apply descriptor-specific or fingerprint-specific cleaning.
5. Build pre_model_steps such as VarianceThreshold or TruncatedSVD.
6. Remove unusable rows, then hash the cleaned dataset for reproducibility.

Descriptor datasets

Descriptor datasets:

• can use feature_engineering.numeric_from_strings
• may set drop_na: true
• otherwise drop all-NaN columns and median-impute remaining numeric NaNs
• do not auto-enable variance thresholding or SVD, but configs can turn them on

Fingerprint datasets

Fingerprint datasets:

• are coerced to numeric
• drop rows with NaNs after coercion
• are cast to float32
• auto-enable variance_threshold with a default threshold of 0.01 when not overridden
• auto-enable svd with a default n_components of 10 when not overridden

Training config expectations

The refreshed starter configs assume the canonical psychlight_a flow and explicitly drop metadata columns:

drop_columns:
  - ID
  - Known
  - SMILES
  - kekule_smiles

That Known drop matters because the feature CLI preserves it in the generated CSVs.

DeepPK behavior

DeepPK handling in the current code path is:

1. aromatic_to_kekule converts SMILES to kekulized form.
2. _prepare_deeppk_artifacts writes the initial _dpk.csv and _dpk.smi.
3. _submit_and_poll_deeppk submits the .smi file and polls until completion or timeout.
4. _merge_deeppk_results merges returned records back into the CSV.
5. If extra records remain unmatched, the CLI writes <name>_deeppk_extra.csv.

If no valid kekulized SMILES are available, the command fails early instead of submitting malformed input.