Python Framework for Supervised ML Benchmarking

Automate Your ML Pipeline.
Benchmark in One Line.

Train, evaluate, and visualize multiple machine learning classifiers and regressors automatically. Enjoy fault-tolerant execution, DataFrame preservation, and publication-ready diagnostic charts.

View on PyPI
Windows PowerShell
PS C:\> py -m pip install multimodel-analysis

Recommended for Windows PowerShell and Command Prompt using the Python Launcher (py).

15+ Built-in Models
1-Line Execution Summary
100% DataFrame Safe
Apache 2.0 Open Source License
PePy Stat Download Analytics →

Platform Installation Guide

Full setup instructions with virtual environment configuration for every major platform.

Windows
PowerShell · CMD · Python Launcher
Recommended

Use the official Python 3 Launcher (py) to avoid PATH conflicts across multiple Python versions on Windows.

PowerShell
PS C:\> py -m pip install multimodel-analysis
VirtualEnv Setup
PS C:\> py -m venv venv
PS C:\> .\venv\Scripts\activate
(venv) pip install multimodel-analysis
macOS
Terminal · Homebrew · Apple Silicon

Works natively on Intel and Apple Silicon (M1/M2/M3). For Homebrew-managed Python, always call python3 explicitly.

Terminal (Quick Install)
$ python3 -m pip install multimodel-analysis
Homebrew + VirtualEnv
$ brew install python@3.11
$ python3.11 -m venv ml_env
$ source ml_env/bin/activate
(ml_env) $ pip install multimodel-analysis
Linux
Ubuntu · Debian · RHEL · Arch

For modern Debian/Ubuntu with PEP 668 protection, always install inside a virtual environment — the system Python is managed by the OS.

Bash (Quick Install)
$ pip3 install multimodel-analysis
PEP 668 Compliant Setup
$ sudo apt install python3-venv
$ python3 -m venv ~/.venv/ml
$ source ~/.venv/ml/bin/activate
(.venv) $ pip install multimodel-analysis
Conda / Mamba
Anaconda · Miniconda · Mambaforge

Conda automatically resolves C-level dependencies like OpenMP, BLAS, and MKL — critical for XGBoost and Scikit-learn performance on all platforms.

Conda Environment Setup
(base) $ conda create -n ml-env python=3.11 -y
(base) $ conda activate ml-env
(ml-env) $ pip install multimodel-analysis
Mamba (Faster Alternative)
(base) $ mamba create -n ml-env python=3.11 -y
(base) $ mamba activate ml-env
(ml-env) $ pip install multimodel-analysis
System Requirements Python 3.8+ pip 21+ scikit-learn pandas numpy matplotlib

Core Execution Architecture

Engineered for zero data-leakage, fault tolerance, and developer productivity.

Automated Preprocessing

Applies StandardScaler while preserving pandas DataFrame structure, column names, and index identifiers. No more lost feature names!

Smart Label Encoding

Automatically detects and encodes string, numerical, or categorical target labels using LabelEncoder, supporting seamless multiclass analysis.

Stratified Splitting Guard

Implements intelligent stratified train-test splits with safeguards (min_count >= 2) to ensure rare target classes never crash validation.

Fault-Tolerant Parallelism

Wraps individual model fittings in isolated exception blocks. If one model fails to converge, the benchmark continues cleanly without aborting.

Exact Statistical Metrics

Computes weighted Precision, Recall, F1, and One-vs-Rest macro ROC-AUC for classification, plus R², MAE, MSE, and RMSE for regression.

Headless Exporting

Designed for CI/CD pipelines and Docker containers. Easily export tables to CSV/Excel/HTML/JSON and render charts directly to PNG files.

Interactive Benchmarking Simulator

Experience how multimodel-analysis evaluates and ranks algorithms in real time. Adjust parameters below and trigger a simulated execution.

Live Model Benchmarker Simulated Sandbox
Customer Churn Prediction
Target Unit: Binary ("Yes" / "No")

Predicting customer attrition in telecom/SaaS. Feature matrix (X) includes customer tenure, monthly billing, and support ticket frequency. Evaluated under realistic 80/20 class distribution.

Fitting models in parallel (n_jobs=-1)... 0%
Rank Estimator Name Accuracy Precision Recall F1 Score ROC-AUC

Evaluation Metric Formula Sandbox

Interact with the statistical formulas powering multimodel-analysis. Adjust the confusion matrix cell values below to see real-time metric recalculations.

Confusion Matrix Parameters

True Positives (TP): 150
False Positives (FP): 15
False Negatives (FN): 10
True Negatives (TN): 325
Accuracy
95.0%
(TP + TN) / Total
Precision
90.9%
TP / (TP + FP)
Recall (Sensitivity)
93.8%
TP / (TP + FN)
F1 Score (Harmonic)
92.3%
2·(Prec·Rec)/(Prec+Rec)

Quickstart Code Showcase

Copy and paste these production-ready templates directly into your Jupyter Notebook or Python IDE.

import pandas as pd
from sklearn.datasets import load_breast_cancer
from multimodel_analysis import MultiModelClassifier, save_report

# 1. Load benchmark dataset & prepare feature matrix
data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

# 2. Instantiate pipeline with stratification and feature scaling
clf = MultiModelClassifier(
    X=X, y=y, test_size=0.2, scaled_data=True, random_state=42, stratify=True
)

# 3. Train & evaluate all 8 built-in classifiers in parallel
results = clf.run_all_models(random_state=42)

# 4. Display tabular leaderboard & export report to CSV
df_report = clf.show_tabular_report(return_df=True)
clf.save_report("breast_cancer_metrics.csv")

# 5. One-line diagnostic summary & figure exports
clf.get_summary(save_prefix="breast_cancer", show_plot=False)
import pandas as pd
from sklearn.datasets import fetch_california_housing
from multimodel_analysis import MultiModelRegressor

# 1. Load continuous target dataset & prepare feature matrix
data = fetch_california_housing()
X = pd.DataFrame(data.data, columns=data.feature_names).iloc[:500]  # Subset for fast execution
y = data.target[:500]

# 2. Instantiate regression pipeline with scaling
reg = MultiModelRegressor(
    X=X, y=y, test_size=0.2, scaled_data=True, random_state=42
)

# 3. Train & evaluate all 7 built-in regression models
results = reg.run_all_models(random_state=42)

# 4. Display tabular evaluation leaderboard
reg.show_tabular_report()

# 5. Export summary reports & diagnostic scatter plots
reg.get_summary(save_prefix="california_housing", show_plot=False)
import pandas as pd
from xgboost import XGBClassifier
from lightgbm import LGBMClassifier
from multimodel_analysis import MultiModelClassifier, save_report

# 1. Prepare dataset for multi-estimator benchmarking
df = pd.read_csv("credit_risk.csv")
X, y = df.drop(columns=["Default"]), df["Default"]

# 2. Instantiate core MultiModelClassifier instance
clf = MultiModelClassifier(X=X, y=y, test_size=0.20, scaled_data=True)

# 3. Define custom third-party gradient boosting models
custom_models = {
    "XGBoost Classifier": XGBClassifier(n_estimators=200, learning_rate=0.05, random_state=42),
    "LightGBM Classifier": LGBMClassifier(n_estimators=200, learning_rate=0.05, random_state=42)
}

# 4. Benchmark built-in estimators ALONGSIDE XGBoost & LightGBM!
results = clf.run_all_models(custom_models=custom_models)

# 5. Export unified leaderboard comparing standard & boosting models
df_report = clf.show_tabular_report(results, return_df=True)
save_report(df_report, "custom_boosting_benchmark.csv")
clf.plot_comparison(results, save_path="custom_model_comparison.png", show_plot=False)
import pandas as pd
from multimodel_analysis import MultiModelClassifier, save_report

# 1. Execute benchmarking suite to produce evaluation results
df = pd.read_csv("medical_diagnosis.csv")
clf = MultiModelClassifier(X=df.drop(columns=["Target"]), y=df["Target"])
results = clf.run_all_models()

# 2. Capture tabular evaluation metrics DataFrame
df_report = clf.show_tabular_report(results, return_df=True)

# 3. Export to multiple publication & pipeline formats in 1 line each!
save_report(df_report, "exports/benchmark_results.csv")   # CSV Spreadsheet
save_report(df_report, "exports/benchmark_results.xlsx")  # Excel Workbook
save_report(df_report, "exports/benchmark_results.html")  # Styled Web Table
save_report(df_report, "exports/benchmark_results.json")  # REST API JSON Payload

# 4. Generate all diagnostic charts directly into PNG image files
clf.plot_confusion_matrices(results, save_path="exports/confusion_matrix.png", show_plot=False)
clf.plot_roc_curves(results, save_path="exports/roc_curves.png", show_plot=False)
clf.plot_comparison(results, save_path="exports/metrics_summary.png", show_plot=False)

Complete API Reference

Exhaustive specification of methods, arguments, and return types across all modules.

💡
Interactive API Explorer
Click on any function or method name in the index below (or hover over any row) to open runnable Python code examples, parameter details, and usage snippets in an interactive modal dialog.
Methods & Utilities Index ⚡ Click any function to see example code
Class / Module Method Name (Click to view code example) Parameters Return Type Description
multimodel_analysis df: DataFrame = None, filepath: str = "report.csv" None Exports report DataFrame to disk (.csv, .xlsx, .xls, .html, .json) based on file extension.
MultiModelClassifier random_state: int = None, max_iter: int = 1000, **kwargs tuple Trains a Logistic Regression classifier and computes evaluation metrics.
MultiModelClassifier random_state: int = None, kernel: str = 'linear', probability: bool = True, **kwargs tuple Trains a Support Vector Classifier (SVC) with probability estimation for ROC-AUC.
MultiModelClassifier random_state: int = None, **kwargs tuple Trains a Decision Tree Classifier and evaluates split accuracy metrics.
MultiModelClassifier n_neighbors: int = None, **kwargs tuple Trains K-Nearest Neighbors Classifier. Auto-calculates optimal n_neighbors if omitted.
MultiModelClassifier **kwargs tuple Trains a Gaussian Naive Bayes Classifier with automatic random state handling.
MultiModelClassifier n_estimators: int = 100, random_state: int = None, **kwargs tuple Trains an ensemble Random Forest Classifier across parallel CPU cores.
MultiModelClassifier n_estimators: int = 100, random_state: int = None, **kwargs tuple Trains a Gradient Boosting Classifier and evaluates stage-wise residual loss.
MultiModelClassifier n_estimators: int = 50, random_state: int = None, **kwargs tuple Trains an AdaBoost Classifier using decision stump base estimators.
MultiModelClassifier custom_models: dict = None, random_state: int = None list[tuple] Fits all 8 built-in classifiers plus optional custom estimators. Returns evaluation tuples.
MultiModelClassifier model: estimator, X_test: array = None, y_true: array = None tuple Evaluates a single fitted classifier model on test set data (Report, Matrix, Acc, Prec, Rec, F1, ROC-AUC).
MultiModelClassifier models: list = None, return_df: bool = False pd.DataFrame | None Displays formatted console leaderboard sorted by Accuracy and recommends the optimal classifier.
MultiModelClassifier models: list = None, save_path: str = None, show_plot: bool = True None Generates colorful confusion matrix heatmaps with original decoded class labels.
MultiModelClassifier models: list = None, save_path: str = None, show_plot: bool = True None Plots Receiver Operating Characteristic (ROC) curves with AUC metrics for all models.
MultiModelClassifier models: list = None, save_path: str = None, show_plot: bool = True None Renders grouped bar charts comparing Accuracy, Precision, Recall, and F1 Score across models.
MultiModelClassifier models: list = None, save_prefix: str = None, show_plot: bool = True None Runs full reporting and plotting suite in one call with automated file exporting.
MultiModelClassifier df_or_filepath: DataFrame|str = None, filepath: str = None None Saves the classifier's latest tabular performance report to disk.
MultiModelRegressor **kwargs tuple Trains an Ordinary Least Squares Linear Regression model.
MultiModelRegressor alpha: float = 0.1, random_state: int = None, **kwargs tuple Trains a Lasso (L1 Regularized) Regression model.
MultiModelRegressor alpha: float = 1.0, random_state: int = None, **kwargs tuple Trains a Ridge (L2 Regularized) Regression model.
MultiModelRegressor kernel: str = 'rbf', **kwargs tuple Trains a Support Vector Regressor (SVR).
MultiModelRegressor random_state: int = None, **kwargs tuple Trains a Decision Tree Regressor.
MultiModelRegressor n_estimators: int = 100, random_state: int = None, **kwargs tuple Trains an ensemble Random Forest Regressor across parallel cores.
MultiModelRegressor n_estimators: int = 100, random_state: int = None, **kwargs tuple Trains a Gradient Boosting Regressor.
MultiModelRegressor n_estimators: int = 50, random_state: int = None, **kwargs tuple Trains an AdaBoost Regressor.
MultiModelRegressor custom_models: dict = None, random_state: int = None list[tuple] Fits all 7 built-in regressors plus optional custom estimators. Caches results in regressor.models_.
MultiModelRegressor model: estimator, X_test: array = None, y_true: array = None tuple Evaluates a single fitted regressor model on test set data (MAE, MSE, RMSE, R2, y_pred).
MultiModelRegressor models: list = None, return_df: bool = False pd.DataFrame | None Displays formatted leaderboard sorted descending by R² Score (includes MAE, MSE, RMSE).
MultiModelRegressor models: list = None, save_path: str = None, show_plot: bool = True None Generates a grid scatter plot of True vs. Predicted target values with identity (y = x) reference lines.
MultiModelRegressor models: list = None, save_path: str = None, show_plot: bool = True None Renders a comparative bar chart of R² Scores across all evaluated regressor models.
MultiModelRegressor models: list = None, save_prefix: str = None, show_plot: bool = True None Runs full evaluation and visualization pipeline for regressors (prints report, plots scatter & bar charts).
MultiModelRegressor df_or_filepath: DataFrame|str = None, filepath: str = None None Saves the regressor's latest tabular performance report to disk.
MultiModelRegressior Same as MultiModelRegressor Class Backward-compatibility class alias mapping directly to MultiModelRegressor.

Diagnostic Visualizations Gallery

Generate publication-quality seaborn and matplotlib charts with zero boilerplate plotting code.

Best Practices & Execution Guide

Pro tips for getting maximum performance and reliability out of your model benchmarks.

1. Handling Class Imbalance

When rare classes represent less than 10% of your dataset, ensure stratify=True remains enabled (the default). For severe imbalance, inspect weighted F1 and Recall rather than Accuracy alone.

2. DataFrame Target Slice Formatting

When separating target columns from pandas DataFrames, pass a 1D Series (df['target']) rather than a 2D DataFrame slice (df[['target']]) to avoid scikit-learn data conversion warnings.

3. Headless CI/CD & Server Exporting

In Docker containers, automated GitHub Actions, or SSH servers without an X11 graphical display, pass show_plot=False and specify a save_path to generate PNG charts directly to disk without GUI errors.

4. CPU Multi-Threading Allocation

For tabular datasets exceeding 50,000 rows, set n_jobs=-1 in constructor initialization to parallelize tree construction across Random Forest and Extra Trees algorithms across all available CPU cores.