Data Analysis and Visualization

The FTIRdataanalysis class provides comprehensive tools for exploratory data analysis, statistical testing, dimensionality reduction, and clustering of preprocessed FTIR spectra.

Overview

After preprocessing your FTIR data with FTIRdataprocessing, use FTIRdataanalysis to:

  • Visualize spectral patterns and trends

  • Perform statistical analysis

  • Reduce dimensionality for visualization

  • Cluster samples based on spectral similarity

  • Prepare data for machine learning

from xpectrass import FTIRdataanalysis

# Initialize with preprocessed data
analysis = FTIRdataanalysis(
    df=processed_df,
    dataset_name="MyDataset",
    label_column="type",
    random_state=42,
    n_jobs=-1
)

Initialization Parameters

FTIRdataanalysis.init()

FTIRdataanalysis(
    df,                      # Preprocessed DataFrame
    dataset_name=None,       # Dataset identifier for plots
    label_column="type",     # Label column name
    sample_id_column="sample_id",  # Sample ID column name
    exclude_columns=None,    # Additional non-spectral columns
    start_wn=None,           # Minimum wavenumber (not yet implemented)
    end_wn=None,             # Maximum wavenumber (not yet implemented)
    drop_region=None,        # Wavenumber region to drop (not yet implemented)
    random_state=None,       # Random seed for reproducibility
    n_jobs=-1                # Parallel processing cores
)

Spectral Visualization

Plot Mean Spectra by Class

Visualize average spectra for each polymer type:

# Plot mean spectra for all classes
analysis.plot_mean_spectra(
    title="Mean Spectra by Type",  # Plot title
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Output:

  • Mean spectrum for each class with different colors

  • Optional standard deviation shading

  • Legend showing all polymer types

Plot Overlay of Mean Spectra

Compare mean spectra across classes:

# Overlay all class means
analysis.plot_overlay_spectra(
    title="Mean Spectra overlay",  # Plot title
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Plot Spectral Heatmap

Visualize all spectra as a heatmap:

# Create heatmap ordered by class
analysis.plot_heatmap(
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Features:

  • Samples as rows, wavenumbers as columns

  • Color intensity represents absorbance

  • Optional hierarchical clustering

  • Class annotations

Plot Coefficient of Variation

Identify variable and stable spectral regions:

# Plot CV by class
analysis.plot_cv(
    title="Spectral Variability by Type",  # Plot title
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Interpretation:

  • High CV = high variability (potential noise or real variation)

  • Low CV = stable peaks (good for classification)

Statistical Analysis

ANOVA Analysis

Test for significant differences between classes:

# Perform ANOVA at each wavenumber
analysis.perform_anova(
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Plot shows:

  • -log10(p-value) across spectrum

  • Significance threshold line

  • Regions where classes differ significantly

Correlation Matrix

Visualize correlations between wavenumbers:

# Plot correlation matrix
analysis.plot_correlation(
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Use cases:

  • Identify correlated spectral regions

  • Detect redundancy in features

  • Guide feature selection

Dimensionality Reduction

Principal Component Analysis (PCA)

# Perform PCA
analysis.plot_pca(
    standardize=True,        # Standardize features before PCA
    handle_missing="zero",   # How to handle missing values
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Plots generated:

  1. 2D scatter: PC1 vs PC2 colored by class

  2. Explained variance plot: Variance explained by each component

Interpretation:

  • Well-separated clusters = good class discrimination

  • Explained variance indicates information retention

t-SNE (t-Distributed Stochastic Neighbor Embedding)

Non-linear dimensionality reduction for visualization:

# Perform t-SNE
analysis.plot_tsne(
    pca_components=20,       # Number of PCA components for pre-reduction
    standardize=True,        # Standardize features
    handle_missing="zero",   # How to handle missing values
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Parameters:

  • perplexity: Higher values focus on global structure

  • Typical range: 5-50

  • Recommended: 30 for most datasets

Best for:

  • Visualizing complex, non-linear patterns

  • Revealing cluster structure

  • Publication-quality figures

UMAP (Uniform Manifold Approximation and Projection)

Modern non-linear dimensionality reduction:

# Perform UMAP
analysis.plot_umap(
    pca_components=20,       # Number of PCA components for pre-reduction
    standardize=True,        # Standardize features
    handle_missing="zero",   # How to handle missing values
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Parameters:

  • n_neighbors: Controls balance between local and global structure

    • Low (5-15): Emphasizes local structure

    • High (50-100): Emphasizes global structure

  • min_dist: Controls cluster tightness

    • Low (0.0-0.1): Tight clusters

    • High (0.5-0.99): Spread out clusters

Advantages over t-SNE:

  • Faster computation

  • Better preserves global structure

  • More consistent results

  • Supports supervised projections

PLS-DA (Partial Least Squares Discriminant Analysis)

Supervised dimensionality reduction:

# Perform PLS-DA
analysis.plot_plsda(
    n_components=20,         # Number of latent variables
    standardize=True,        # Standardize features
    handle_missing="zero",   # How to handle missing values
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Best for:

  • Supervised classification and dimensionality reduction

  • Feature importance analysis

  • Maximizing class separation

OPLS-DA (Orthogonal PLS-DA)

Enhanced PLS-DA with orthogonal signal correction:

# Perform OPLS-DA
analysis.plot_oplsda(
    n_components=1,          # Predictive components
    n_orthogonal=2,          # Orthogonal components
    standardize=True,        # Standardize features
    handle_missing="zero",   # How to handle missing values
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Advantages:

  • Separates predictive and orthogonal variation

  • Easier interpretation than PLS-DA

  • Better for biomarker discovery

Clustering Analysis

K-means Clustering

Partition spectra into K clusters:

# Perform K-means clustering
analysis.plot_kmeans_clus(
    n_components_clustering=10,  # PCA components for clustering
    k_range=(2, 11),         # Range of K values to evaluate
    standardize=True,        # Standardize features
    handle_missing="zero",   # How to handle missing values
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Plots:

  1. Cluster scatter: PCA projection colored by cluster

  2. Elbow plot: Helps choose optimal K

  3. Silhouette analysis: Shows cluster quality

Choosing K:

  • Look for “elbow” in inertia plot

  • Check silhouette scores

  • Consider domain knowledge

Hierarchical Clustering

Build dendrogram showing sample relationships:

# Perform hierarchical clustering
analysis.plot_hierarchical_clus(
    n_samples_dendro=100,    # Max samples in dendrogram
    standardize=True,        # Standardize features
    handle_missing="zero",   # How to handle missing values
    figsize=(16, 12),
    save_plot=False,
    save_path=None
)

Linkage methods:

  • ward: Minimizes variance (recommended)

  • average: Average linkage

  • complete: Maximum linkage

  • single: Minimum linkage

Plots:

  1. Dendrogram: Tree showing sample relationships

  2. Clustered heatmap: Spectra reordered by clustering

Complete Analysis Example

from xpectrass import FTIRdataprocessing, FTIRdataanalysis
from xpectrass.data import load_jung_2018

# 1. Load and preprocess data
df = load_jung_2018()
ftir = FTIRdataprocessing(df, label_column="type")
ftir.run()
processed_df = ftir.df_norm

# 2. Initialize analysis
analysis = FTIRdataanalysis(
    processed_df,
    dataset_name="Jung_2018",
    label_column="type",
    random_state=42
)

# 3. Exploratory visualization
print("Plotting mean spectra...")
analysis.plot_mean_spectra()

print("Creating spectral heatmap...")
analysis.plot_heatmap()

print("Calculating coefficient of variation...")
analysis.plot_cv()

# 4. Statistical analysis
print("\nPerforming ANOVA...")
analysis.perform_anova()

# 5. Dimensionality reduction
print("\nDimensionality reduction analysis:")

print("  - PCA...")
analysis.plot_pca()

print("  - t-SNE...")
analysis.plot_tsne()

print("  - UMAP...")
analysis.plot_umap()

print("  - PLS-DA...")
analysis.plot_plsda()

# 6. Clustering
print("\nClustering analysis:")

print("  - K-means...")
analysis.plot_kmeans_clus()

print("  - Hierarchical...")
analysis.plot_hierarchical_clus()

print("\n✓ Analysis complete!")

Saving Figures

All plotting methods support saving:

# Save individual plots
analysis.plot_pca(
    save_plot=True,
    save_path="figures/pca_analysis.png"
)

# Save with custom format
analysis.plot_mean_spectra(
    save_plot=True,
    save_path="figures/mean_spectra.pdf"  # Supports: png, pdf, svg, eps
)

Tips and Best Practices

  1. Always visualize first: Start with mean spectra and heatmaps to understand your data

  2. Check statistical significance: Use ANOVA to identify discriminative wavenumbers

  3. Try multiple methods: Compare PCA, t-SNE, and UMAP for different perspectives

  4. Use appropriate parameters:

    • t-SNE perplexity: 5-50 (typically 30)

    • UMAP n_neighbors: 5-100 (typically 15)

    • K-means: Use elbow plot to choose K

  5. Save your figures: Use high DPI (300) for publication-quality images

  6. Cross-validate: Use PLS-DA cross-validation to assess model quality

  7. Interpret loadings: PCA/PLS loadings show which peaks drive separation

Next Steps