Skip to content

Latest commit

 

History

History
197 lines (120 loc) · 9.39 KB

ExperimentsReproducibility.md

File metadata and controls

197 lines (120 loc) · 9.39 KB

Experiment Reproducibility Guide

This guide maps each section of the paper to the corresponding code, datasets, and model outputs, so you can reproduce the results as described.

We conducted experiments on three datasets: Healthy Heart (HH), Autism Spectrum Disorder (ASD), and Acute Myeloid Leukemia (AML). For each dataset, we provide:

    1. Datasets
    • Preprocessing and 5-fold cross-validation scripts to split the data into train/val/test.
    • Alternatively, we provide the precomputed data splits via scMEDAL's Figshare. See Datasets.
    1. Scripts for model training for our scMEDAL API models (Autoencoder, scMEDAL variants, Mixed Effects Classifier) and comparable models: scVI, scANVI, Harmony, Scanorama, and SAUCIE.
    1. Analysis notebooks: Once you have trained your models you need to run the analysis notebooks that retrieve clustering scores (Figures 2-4 and 8), Genomaps (Figures 5-7), UMAPs (Figures 2-4 and 8) and extract the Random Forest classifier outputs reported in Table 1. Use these notebooks to reproduce the paper results once latent spaces and reconstruction outputs are available.

We also provide in scMEDAL's Figshare:

  • Latent space outputs and a sample count matrix with 300-cell projections so you can reproduce the Genomaps exactly as in the paper.

Default paths are defined in utils/defaults.py: inputs load from ./data/ and outputs save to ./outputs/.

To set up your project path, edit utils/defaults.py:

# utils/defaults.py
ROOT_PATH = "/your/project/path"

Before running Python scripts or notebooks, you can optionally point the working directory to your project path:

# In a script or notebook cell
import os
os.chdir("/your/project/path")  

If your project uses different locations, update the configuration accordingly. See Outputs and analysis folders.

You have two options:

A. Reproduce analyses from provided model outputs

Download our latent spaces and the multibatch projection of 300-cell sample matrix, place them in the specified folders (see below), and run the analysis notebooks to regenerate paper figures. For example, for AML latent space.

  1. Place the AML latent space under:

      
      /outputs/AML/latent_space/log_transformed_2916hvggenes
      
  2. Place the multibatch projection of 300-cell sample matrix used for Genomaps under:

    
      outputs/AML/compare_models/log_transformed_2916hvggenes/AML_default/genomap/CMmultibatch_300_cells_per_batch_19batches_Mono_Mono-like_with_2fe_input

After these paths are set, run the Analysis notebooks to reproduce the paper figures.

B. Run models from scratch

Train the models yourself and once you have latent space and reconstruction outputs, run the analysis notebooks.

scMEDAL API model scripts

To run all models from our API ($AE$, $AEC_{ct}$, scMEDAL-FE, scMEDAL-RE, and scMEDAL-FEC), use 1-run_scMEDAL_alldatasets.py and make sure your scMEDAL environment is activated. See scMEDAL Installation.

python 1-run_scMEDAL_alldatasets.py

The $AEC_{batch}$ experiment can be run with:

python 1-run_scMEDAL_alldatasets_AECbatch.py

Comparable Models Scripts

Each comparable model has its own script and environment requirements. Configurations vary by dataset. Below we list the AML entry points; for HH and ASD, use the corresponding dataset paths/scripts.

All comparable-model scripts produce the same output file structure as our API models. See Outputs & analysis folders.

SAUCIE
scVI
scanorama
harmony
scANVI

Note: For scVI, scANVI, Harmony, Scanorama, and SAUCIE, use the Comparables conda environment. You can create it using comparables_requirements.txt

MEC Classifier Scripts

After running the models, you may need to update paths to match your own output directories. See Outputs and analysis folders. Once the output folders are updated, you can train the Mixed Effects Classifier (MEC):

ASD

AML

Healthy Heart


Analysis Notebooks

Use the following notebooks to reproduce the analyses and figures:

Additional analysis notebooks for $AEC_{batch}$ are in /scripts/3-analysis_AECbatch. Use these to obtain the $AEC_{batch}$ scores shown in Figure 2c and Supplementary Figures S11–S13.

Download/setup the required model outputs (or train from scratch), update paths where necessary, and run these notebooks to regenerate the paper results. If you train models yourself, you may need to update folder names to match your own output directories. See Outputs and analysis folders

Note: Due to variability in TensorFlow, model outputs may differ slightly across runs. To account for this, we report 95% confidence intervals (CI) as an estimate of variability.


References

  • scMEDAL's Figshare: Andrade, Aixa X.; Nguyen, Son; Montillo, Albert (2026). scMEDAL: Interpretable Single-Cell Transcriptomics Analysis with Batch Effect Visualization via Deep Mixed-Effects Autoencoder. figshare. Dataset. https://doi.org/10.6084/m9.figshare.28414367
  • Litvinukova, M. et al. Cells of the adult human heart. Nature 588, 466-472 (2020).
  • van Galen, P. et al. Single-Cell RNA-Seq Reveals AML Hierarchies Relevant to Disease Progression and Immunity. Cell 176, 1265-1281.e24 (2019).
  • Velmeshev, D. et al. Single-cell genomics identifies cell type-specific molecular changes in autism. Science 364, 685-689 (2019).
  • Speir, M. L. et al. UCSC Cell Browser: visualize your single-cell data. Bioinformatics 37, 4578-4580 (2021).
  • Yu, X., Xu, X., Zhang, J., & Li, X. Batch alignment of single-cell transcriptomics data using deep metric learning. Nat Commun 14, 960 (2023).
  • Yu, X., Xu, X., Zhang, J., & Li, X. Batch alignment of single-cell transcriptomics data using deep metric learning. figshare https://doi.org/10.6084/m9.figshare.20499630.v2 (2023).
  • Lopez, R. et al. Deep generative modeling for single-cell transcriptomics. Nat Methods 15, 1053-1058 (2018).
  • Xu, C. et al. Probabilistic harmonization and annotation of single-cell transcriptomics data with deep generative models. Mol Syst Biol 17, e9620 (2021).
  • Amodio, M. et al. Exploring single-cell data with deep multitasking neural networks. Nat Methods 16, 1139-1145 (2019).
  • Hie, B. et al. Efficient integration of heterogeneous single-cell transcriptomes using Scanorama. Nat Biotechnol 37, 685-691 (2019).
  • Korsunsky, I. et al. Fast, sensitive and accurate integration of single-cell data with Harmony. Nat Methods 16, 1289-1296 (2019).
  • Gayoso, A. et al. A Python library for probabilistic analysis of single-cell omics data. Nat Biotechnol 40, 163?166 (2022). https://doi.org/10.1038/s41587-021-01206-w
  • Islam, M. T. & Xing, L. Cartography of Genomic Interactions Enables Deep Analysis of Single- Cell Expression Data. Nat. Commun. 14, 679 (2023).