Machine Learning

The FTIRdataanalysis class provides comprehensive machine learning capabilities for FTIR plastic classification, including model evaluation, hyperparameter tuning, and explainability analysis.

Overview

The machine learning workflow includes:

  1. Data preparation: Train/test split and scaling

  2. Model evaluation: Test 20+ classification algorithms

  3. Model comparison: Visualize performance metrics

  4. Hyperparameter tuning: Optimize top models

  5. Model interpretation: SHAP explainability analysis

from xpectrass import FTIRdataanalysis

# Initialize with preprocessed data
analysis = FTIRdataanalysis(processed_df, label_column="type", random_state=42)

Data Preparation

Train/Test Split

# Prepare data for machine learning
analysis.ml_prepare_data(
    test_size=0.2           # 20% for testing
)

print(f"Training samples: {len(analysis.y_train)}")
print(f"Test samples: {len(analysis.y_test)}")
print(f"Features: {analysis.x_train_scaled.shape[1]}")
print(f"Classes: {analysis.class_names}")

Attributes created:

  • x_train_scaled, x_test_scaled: Standardized feature matrices

  • x_train_raw, x_test_raw: Raw (unscaled) feature matrices

  • y_train, y_test: Labels

  • class_names: Unique class labels

  • scaler: Fitted StandardScaler

  • label_encoder: Fitted LabelEncoder

  • wavenumbers: Array of wavenumber values

  • dir_: Dictionary containing all of the above

Available Models

View All Models

# See available classification models
models = analysis.available_models()
print(f"Total models: {len(models)}")
for model_name in models:
    print(f"  - {model_name}")

Model Categories

The library includes 20+ classification algorithms across multiple families:

Linear Models:

  • Logistic Regression

  • Ridge Classifier

  • SGD Classifier

Tree-Based Models:

  • Decision Tree

  • Random Forest

  • Extra Trees

  • AdaBoost

  • Gradient Boosting

Ensemble Models:

  • XGBoost (multiple configs)

  • LightGBM (multiple configs)

Naive Bayes:

  • Gaussian Naive Bayes

  • Multinomial Naive Bayes

Support Vector Machines:

  • Linear SVM

  • RBF SVM

  • Poly SVM

Nearest Neighbors:

  • K-Nearest Neighbors (multiple K values)

Neural Networks:

  • Multi-Layer Perceptron (multiple architectures)

Discriminant Analysis:

  • Linear Discriminant Analysis

  • Quadratic Discriminant Analysis

Running Models

Run a Single Model

# Run specific model
results = analysis.run_a_model(
    model_name='XGBoost (100)',
    model=None,
    cv_folds=5,
    plot_confusion=True,
    save_plot_path=None,
    print_test_result=True
)
overall, per_class, confusion_matrix = results
print(f"Accuracy: {overall['accuracy']:.3f}")
print(f"F1 (weighted): {overall['f1_weighted']:.3f}")

Run All Models

Evaluate all available models with cross-validation:

# Run all models
results = analysis.run_all_models(
    test_size=0.2,
    plot_comparison=True,
    accuracy_threshold=0.9,
    top_n_methods=20,
    save_plot_path=None
)

# View results sorted by F1 score
print(results.sort_values('test_f1', ascending=False))

# Save results
analysis.results_all = results  # Stored for later use

Results DataFrame includes:

  • model_name: Model name

  • test_accuracy: Test set accuracy

  • test_precision: Weighted precision

  • test_recall: Weighted recall

  • test_f1: Weighted F1 score

  • y_pred: Predicted labels

  • y_proba: Predicted probabilities

  • train_accuracy: Training accuracy

  • cv_mean: Mean cross-validation score

  • cv_std: CV standard deviation

  • overfit_gap: Difference between train and test accuracy

  • train_time: Training time (seconds)

  • pred_time: Prediction time (seconds)

View Top Models

# Get top 10 models by F1 score
top_models = results.nlargest(10, 'test_f1')
print(top_models[['model_name', 'test_accuracy', 'test_f1', 'cv_mean']])

Model Comparison Visualization

When plot_comparison=True (the default), run_all_models() automatically generates four visualization plots:

Model Comparison Plot

  • Bar plot of model performance sorted by test accuracy

  • Shows top N models (controlled by top_n_methods parameter)

  • Error bars showing CV standard deviation

  • Color-coded by performance

Family Comparison Plot

  • Average performance per model family (e.g., tree-based vs ensemble vs linear)

  • Violin plots showing distribution within each family

  • Best model in each family highlighted

Efficiency Analysis Plot

  • Scatter plot: Training time vs F1 score

  • Helps identify fast, accurate models

  • Models above the accuracy_threshold are highlighted

Overfitting Analysis Plot

  • Train score vs test score comparison

  • Diagonal line = perfect generalization

  • Points above line indicate overfitting

  • Shows top N models (controlled by top_n_methods parameter)

Hyperparameter Tuning

Tune Top Models

Optimize hyperparameters for best-performing models:

# Tune top 2 models (must run run_all_models() first)
results = analysis.run_all_models()
tuned_results = analysis.model_parameter_tuning(
    number_of_models=2       # Number of top models to tune
)

Note: You must call run_all_models() before model_parameter_tuning(), as it relies on self.results_all and self.dir_ being set.

Tuning Search Spaces:

Each model has a predefined search space covering important hyperparameters:

  • Random Forest: n_estimators, max_depth, min_samples_split, max_features

  • XGBoost: learning_rate, max_depth, n_estimators, subsample, colsample_bytree

  • LightGBM: learning_rate, num_leaves, max_depth, feature_fraction

  • SVM: C, gamma, kernel

  • KNN: n_neighbors, weights, metric

Model Interpretation

SHAP Explainability

Understand which spectral features drive predictions:

# Explain model predictions with SHAP
shap_results = analysis.explain_by_shap(
    model_name='XGBoost (100)',  # Model to explain
    max_display=20,              # Max features to display in SHAP plots
    sample_size=100,             # Background samples for SHAP values
    test_size=0.2,               # Train/test split ratio
    cv_folds=5,                  # Cross-validation folds
    save_plot_path=None          # Path to save SHAP plots
)

# Results are stored in analysis.shap_results

Plots generated:

  1. Summary plot: Shows global feature importance

  2. Beeswarm plot: Feature importance with value distributions

Local SHAP Interpretation

Explain individual predictions:

# Plot decision plot for a single test sample
# (must call explain_by_shap() first)
analysis.local_shap_plot(
    sample_index=0,              # Index of test sample to explain
    figsize=(10, 8),
    save_plot_path=None
)

Decision plot shows:

  • How features push prediction from base value to final prediction

  • Feature contributions for specific samples

  • Comparison across multiple samples

Feature Importance by Wavenumber

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Get feature importance as DataFrame
importance_df = pd.DataFrame({
    'wavenumber': analysis.wavenumbers,
    'importance': np.abs(shap_results['shap_values']).mean(0)
})

# Plot top important wavenumbers
top_features = importance_df.nlargest(20, 'importance')
plt.figure(figsize=(10, 6))
plt.bar(top_features['wavenumber'], top_features['importance'])
plt.xlabel('Wavenumber (cm⁻¹)')
plt.ylabel('Mean |SHAP value|')
plt.title('Top 20 Discriminative Wavenumbers')
plt.show()

Complete Machine Learning Workflow

from xpectrass import FTIRdataprocessing, FTIRdataanalysis
from xpectrass.data import load_jung_2018

# 1. Load and preprocess data
print("Loading and preprocessing data...")
df = load_jung_2018()
ftir = FTIRdataprocessing(df, label_column="type")
ftir.run()
processed_df = ftir.df_norm

# 2. Initialize analysis
analysis = FTIRdataanalysis(
    processed_df,
    dataset_name="Jung_2018",
    label_column="type",
    random_state=42
)

# 3. Run all models (automatically prepares data)
print("\nEvaluating all models...")
results = analysis.run_all_models(
    test_size=0.2,
    plot_comparison=True,
    accuracy_threshold=0.9,
    top_n_methods=20
)

# 4. Display top models
print("\n" + "="*60)
print("TOP 10 MODELS")
print("="*60)
top10 = results.nlargest(10, 'test_f1')
print(top10[['model_name', 'test_accuracy', 'test_f1', 'cv_mean', 'train_time']])

# 5. Tune top models
print("\nTuning top 2 models...")
tuned = analysis.model_parameter_tuning(number_of_models=2)

# 6. Explain best model with SHAP
print("\nExplaining with SHAP...")
analysis.explain_by_shap(
    model_name='XGBoost (100)',
    max_display=20,
    sample_size=100
)

# 7. Local explanations
print("\nGenerating local explanations...")
analysis.local_shap_plot(sample_index=0)

# 8. Save results
results.to_csv("model_comparison_results.csv", index=False)

print("\n✓ Machine learning workflow complete!")

Cross-Dataset Validation

Test model generalization across different datasets:

from xpectrass.data import load_jung_2018, load_frond_2021

# Train on one dataset
df_train = load_jung_2018()
# ... preprocess ...
analysis_train = FTIRdataanalysis(df_train, label_column="type")
analysis_train.ml_prepare_data(test_size=0.2)

# Test on another dataset
df_test = load_frond_2021()
# ... preprocess with same pipeline ...
analysis_test = FTIRdataanalysis(df_test, label_column="type")

# Evaluate cross-dataset performance
# (Note: Requires manual model training and prediction)

Tips and Best Practices

Data Preparation

  1. Always preprocess first: Baseline correction, denoising, normalization

  2. Use stratified split: Maintains class balance in train/test sets

  3. Set random_state: For reproducible results

  4. Check class balance: Imbalanced classes may require special handling

Model Selection

  1. Start with all models: Run run_all_models() to get baseline

  2. Consider speed vs accuracy: Use efficiency analysis plot

  3. Check cross-validation scores: Models with low CV std are more stable

  4. Don’t overfit: Monitor overfitting analysis plot

Hyperparameter Tuning

  1. Tune top 3-5 models: No need to tune everything

  2. Use cross-validation: Prevents overfitting to test set

  3. Increase n_iter for better results: 50-100 iterations recommended

  4. Be patient: Tuning can take time for complex models

Model Interpretation

  1. Use SHAP for final model: Understand what features matter

  2. Check if important wavenumbers make sense: Should align with chemistry

  3. Validate with domain knowledge: Peak assignments should be reasonable

  4. Use local explanations: Understand individual predictions

Performance Metrics

Choose metrics appropriate for your problem:

  • Accuracy: Overall correctness (good for balanced datasets)

  • F1 Score: Harmonic mean of precision and recall (better for imbalanced data)

  • Precision: Minimize false positives

  • Recall: Minimize false negatives

Common Issues and Solutions

Issue: Poor Model Performance

Solutions:

  1. Check preprocessing: Baseline correction, normalization

  2. Remove outliers: Use data validation

  3. Try feature selection: Remove noisy wavenumber regions

  4. Increase training data: Combine multiple datasets

  5. Check class balance: Use class weights or resampling

Issue: Overfitting

Solutions:

  1. Use cross-validation consistently

  2. Reduce model complexity

  3. Increase training data

  4. Apply regularization

  5. Use ensemble methods

Issue: Slow Training

Solutions:

  1. Use n_jobs=-1 for parallelization

  2. Reduce sample size for initial testing

  3. Start with fast models (Logistic Regression, Linear SVM)

  4. Reduce n_iter for hyperparameter tuning

Issue: SHAP Takes Too Long

Solutions:

  1. Reduce sample_size parameter (50-100 is usually sufficient)

  2. Use TreeExplainer for tree-based models (faster)

  3. Use KernelExplainer sample size parameter

  4. Explain fewer samples

Model Export and Deployment

Save Trained Model

import joblib

# Train your best model
best_model_name = tuned.iloc[0]['model']
# ... train model ...

# Save model
joblib.dump(model, 'best_ftir_classifier.pkl')

# Save preprocessing parameters
joblib.dump({
    'scaler': analysis.scaler,
    'class_names': analysis.class_names,
    'wavenumbers': analysis.wavenumbers
}, 'preprocessing_params.pkl')

Load and Use Model

import joblib
import pandas as pd

# Load model and parameters
model = joblib.load('best_ftir_classifier.pkl')
params = joblib.load('preprocessing_params.pkl')

# Preprocess new data
# ... apply same preprocessing pipeline ...

# Predict
predictions = model.predict(new_data_scaled)
probabilities = model.predict_proba(new_data_scaled)

print(f"Predictions: {predictions}")
print(f"Probabilities: {probabilities}")

Next Steps