SME DataPrep Python

sme-dataprep-python contains the Python implementation for the Software Matemático y Estadístico coursework. It consists of the published sme-dataprep package (imported as dataprepkit) and a companion Jupyter notebook with two end-to-end examples.

Repository layout

.
├── README.md
├── data/
│   ├── diabetes/
│   │   └── diabetes_prediction_dataset.csv
│   └── titanic/
│       ├── test.csv
│       └── train.csv
├── usage_examples.ipynb
└── package/
    ├── pyproject.toml
    ├── dataprepkit/
    │   ├── __init__.py
    │   ├── associations.py
    │   ├── dataset.py
    │   ├── discretization.py
    │   ├── management.py
    │   ├── metrics.py
    │   ├── preprocessing.py
    │   └── visualization.py

Install locally

Install the released package from PyPI:

python -m pip install sme-dataprep

To work with this clone instead, run the following from this folder:

python -m pip install -e ./package

The project requires Python 3.10 or later. To run the notebook, also install Jupyter:

python -m pip install jupyter
jupyter notebook usage_examples.ipynb

The notebook installs the PyPI release with %pip install -q sme-dataprep. When testing local edits in the notebook, install ./package in the active kernel instead and restart that kernel.

What the package provides

dataprepkit provides:

• a DataPrepDataset wrapper for loading CSV data, retaining an optional target, and running common operations;
• dataset summaries, practical variable-type detection, binary-target validation, and missing-value reports or imputation;
• min-max normalization, z-score standardization, variable filtering, and several discretization methods;
• variance, entropy, single-variable AUC, correlation, mutual information, and association matrices; and
• ROC/AUC, distribution, and association-matrix plots.

The package is deliberately lightweight: its runtime dependencies are NumPy, pandas, and Matplotlib.

Example datasets

The notebook uses two datasets:

1. data/titanic/train.csv is used for a general preprocessing walkthrough because it has numerical variables, categorical variables, missing values, and a binary target.
2. data/diabetes/diabetes_prediction_dataset.csv is used for a compact realistic analysis focused on target validation, missing-value checks, AUC/ROC, distribution plots, and short conclusions.

• Titanic binary target: Survived

• Numerical variables: Age, SibSp, Parch, Fare

• Categorical variables: Pclass, Sex, Embarked

• Diabetes binary target: diabetes

• Main numerical variables: age, bmi, HbA1c_level, blood_glucose_level

• Other relevant variables: hypertension, heart_disease, gender, smoking_history

The notebook intentionally treats the first dataset as a generic supervised table rather than focusing on its story. The final diabetes section is more analytical and interprets the most relevant outputs.

The AUC visualization section includes two different plots:

• plot_roc_curve: the standard ROC curve for one numerical attribute.
• plot_roc_curves: several ROC curves in one figure, using selected columns or all numerical columns when columns=None.
• compare_auc_values: a barplot that compares final AUC values across several numerical attributes.

Additional implemented extras:

• discretize_by_thresholds: manual cut points chosen by the analyst.
• discretize_by_standard_deviation: groups values by distance from the mean.
• plot_variable_distribution: KDE-style distribution plot, optionally split by target class.
• dataset_summary: compact overview of rows, columns, types, and missing values.
• missing_value_report: missing counts and missing percentages by variable.
• detect_variable_types: practical type detection for numerical, categorical, binary, identifier, and high-cardinality variables.
• validate_binary_target: checks that a target exists and has exactly two classes.
• impute_missing_values: simple missing-value imputation with median/mode or explicit strategies.

Package source

The installable package lives in package/. Its project metadata and dependencies are in package/pyproject.toml; the public Python API is exported from package/dataprepkit/__init__.py.