Skip to content

Data Exploration

src.cli.explore is the quickest way to inspect a generated feature table before you commit to a training run.

Use one of the generated tables from process_features, for example:

python -m src.cli.explore \
    --dimensionality-reduction data/example/processed/psychlight_a/psychlight_a_descriptors.csv \
    --target-column Class \
    --output-dir data/example/exploration

The command reads a CSV, keeps numeric columns, optionally removes the target from the feature matrix, drops all-NaN feature columns, drops rows that are entirely missing, and median-imputes any remaining missing values.

Current defaults

From the live CLI:

  • --output-dir: data/example/exploration
  • --pca-components: 20
  • --feature-importance-top-k: 15
  • --feature-distributions-top-k: 6
  • --tsne-perplexity: 50.0
  • --tsne-learning-rate: 200.0
  • --tsne-max-samples: 1500
  • --umap-n-neighbors: 30
  • --umap-min-dist: 0.15
  • --umap-max-samples: 2000
  • --variance-threshold-value: 0.01
  • --missingness-top-features: 40
  • --correlation-top-k: 30
  • --mutual-information-top-k: 20
  • --binary-activity-top-k: 25
  • --outlier-top-n: 50
  • --random-state: 42

What gets generated

Artifact names are prefixed with:

<dataset_stem>_YYYYMMDD_HHMMSS_<6-char-id>

Depending on the enabled actions, the command can write:

  • *_class_distribution.csv
  • *_class_distribution.png
  • *_missingness_summary.csv
  • *_missingness_heatmap.png
  • *_pca_explained_variance.png
  • *_feature_importance.csv
  • *_feature_importance.png
  • *_mutual_information.csv
  • *_mutual_information.png
  • *_correlation_heatmap.png
  • *_feature_distributions.png
  • *_tsne_snapshot.png
  • *_umap_projection.png
  • *_variance_threshold_filtered.csv
  • *_binary_activity.csv
  • *_binary_activity.png
  • *_outlier_report.csv
  • *_outlier_scatter.png
  • *_summary.md

The Markdown summary is written by MarkdownReporter and includes absolute paths to generated assets.

Action notes

  • Feature importance uses a random forest classifier or regressor depending on the target.
  • Mutual information uses mutual_info_classif or mutual_info_regression using the same target heuristic.
  • Binary activity only includes columns whose non-null values are strictly in {0, 1}.
  • Variance-threshold exports preserve ID and SMILES when those columns exist in the source CSV.
  • UMAP requires umap-learn; the command raises a friendly runtime error if it is unavailable.
  • Outlier detection needs at least 10 samples.

When to use it

Use this command when you want to:

  • confirm target balance before training
  • inspect missingness before choosing preprocessing
  • spot redundant high-correlation features
  • compare descriptor and fingerprint tables visually
  • save a filtered feature export for downstream analysis