Scientific Machine Learning

Problems and mathematical structure

• Solution learning - Approximates a PDE solution from equations, boundary/initial conditions and observations. The original PINN formulation fits a coordinate-to-state function; solving a new problem can require another optimization. Raissi et al. 2017.
• Operator learning - Learns maps between function spaces, such as coefficients to solution fields. Shared parameters across discretizations distinguish the operator formulation from a fixed-grid predictor; approximation and numerical discretization remain separate errors. Kovachki et al. 2023.
• Hybrid models - Combines known differential equations with trainable missing terms. The supplied physics restricts what must be learned; fitting observed trajectories does not uniquely identify an unknown mechanism. Rackauckas et al. 2020.

Neural solvers and equation constraints

• Physics-informed neural networks - Uses automatic differentiation to penalize differential-equation residuals alongside data and boundary losses. Residual penalties enforce equations approximately at sampled points; they are not exact conservation constraints. Raissi et al. 2017.
• Deep BSDE solvers - Recasts suitable parabolic PDEs as backward stochastic differential equations. Neural networks approximate solution gradients along simulated paths; terminal-condition mismatch supplies the training loss. E et al. 2017.
• Deep Ritz - Minimizes a variational energy over neural trial functions. Unlike strong-form residual fitting, it uses the problem's variational formulation; the chosen functional and admissible boundary treatment determine the solution sought. E and Yu 2018.
• Deep Galerkin Method - Trains a neural solution on randomly sampled space-time points using PDE, initial and boundary residuals. Avoids a tensor-product mesh; reported high-dimensional examples do not imply dimension-independent accuracy for arbitrary PDEs. Sirignano and Spiliopoulos 2018.
• hp-VPINNs - Combines neural trial functions with local polynomial test functions and domain decomposition. Weak residuals and h/p refinement connect physics-informed training to variational numerical methods. Kharazmi et al. 2021.
• Residual-adaptive refinement - Adds collocation points where PDE residuals are large. DeepXDE uses this to focus PINN training; a small residual still needs a stability argument to imply small solution error. Lu et al. 2021.

Training failures and error analysis

• PINN failure modes - Convection, reaction and diffusion examples expose optimization failures despite sufficient network expressivity. Curriculum regularization and sequential time training improve the tested cases; adding a PDE penalty alone is insufficient. Krishnapriyan et al. 2021.
• Unbalanced loss gradients - Wang et al. trace training failures to stiffness and unequal backpropagated gradients across loss terms. Gradient-statistics-based balancing improves their examples; the equation and boundary losses need compatible optimization scales. Wang et al. 2021.
• PINN neural tangent kernels - Infinite-width analysis reveals unequal convergence rates for residual and boundary losses. Eigenvalue-based reweighting addresses this imbalance in experiments; the limiting-kernel theory is not a finite-network convergence guarantee. Wang et al. 2022.
• Time-ordered PINN training - Weights later-time residuals according to earlier-time errors. This respects the order of an initial-value problem and improves chaotic-system benchmarks; "causal" here refers to temporal evolution, not causal discovery. Wang et al. 2022.
• Residual-to-solution error bounds - De Ryck et al. bound Navier-Stokes PINN errors using training error, network size and quadrature. Regularity and PDE-specific stability enter the result; approximation capacity alone does not guarantee successful optimization. De Ryck et al. 2023.
• PINN loss curvature - Links differential-operator conditioning to loss curvature. Adam+L-BFGS beats either alone; Nyström-preconditioned Newton-CG improves stalled solutions in the tested PDEs. This compares PINN optimizers, not neural and classical solvers. Rathore et al. 2024.

Learning operators and numerical updates

• Reduced-order operator inference - Fits linear or low-order polynomial dynamics by least squares on projected trajectories, without full solver operators. A reduced state basis limits the model; estimating dynamics adds information beyond compressing snapshots. Peherstorfer and Willcox 2016.
• Learned discretizations - Learns spatial-derivative approximations inside a numerical time integrator. Bar-Sinai et al. obtain accurate coarse-grid dynamics on tested 1D equations; the learned component augments a solver rather than replacing its entire structure. Bar-Sinai et al. 2019.
• Solver-in-the-loop correction - Backpropagates through solver rollouts to train state corrections. Earlier corrections change later inputs, unlike fixed-trajectory training. Gains depend on dynamics: the forced-Burgers experiment shows only a small advantage. Um et al. 2020.
• Particle graph simulators - Represents particles as graph nodes and learns interactions by message passing. Training with perturbed states reduces accumulated rollout error in the reported simulations; long-horizon performance differs from one-step fit. Sanchez-Gonzalez et al. 2020.
• DeepONet - A branch network encodes input-function samples; a trunk network encodes output locations. Their combination predicts an output function. The operator approximation theorem does not bound optimization or generalization error. Lu et al. 2021.
• Fourier Neural Operator - Learns integral kernels through Fourier-mode transformations. Demonstrates resolution transfer on Burgers, Darcy and Navier-Stokes tasks; shared weights across grids do not guarantee accuracy on unseen physical regimes. Li et al. 2021.
  • Physics-Informed Neural Operator - Combines operator training on coarse data with PDE constraints evaluated at finer resolution. Connects FNO-style amortized prediction with equation-based refinement; separate pretraining cost from per-instance inference. Li et al. 2024.
• MeshGraphNets - Passes messages over simulation meshes to predict physical updates, with adaptive meshing in studied systems. Evaluated on cloth, solids and fluids; rollout accuracy and remeshing behavior matter beyond one-step prediction. Pfaff et al. 2021.
• PDE pretraining: Poseidon - Pretrains a multiscale operator transformer on fluid equations, then adapts to 15 tasks. Uses time conditioning and semigroup-based data pairing; transfer to new PDEs includes downstream training, not universal zero-shot solving. Herde et al. 2024.
• Sparse molecular Hessians - Derives a local potential's Hessian sparsity from its interaction graph, then uses coloring and Hessian-vector products for exact recovery. Linear time and storage require fixed cutoff, fixed message-passing depth and bounded local density. Thiede et al. 2026.
• Meshfree exterior calculus - Builds conservative operators on point clouds and learns a local flux law. Transfer bounds separate discretization and flux error, but require controlled stability constants and new edge-feature coverage; one solution need not identify the flux. Shaffer et al. 2026.

Dynamics, conservation and discovery

• SINDy - Selects a sparse set of candidate functions to explain observed state derivatives. Compact equations depend on the chosen variables, function library, derivative estimates and excitation in the observed trajectories. Brunton et al. 2016.
  • PDE-FIND - Extends sparse equation discovery to spatial derivatives and nonlinear terms. Multiple initial conditions can distinguish equations that explain the same single trajectory; low regression error alone need not identify a unique PDE. Rudy et al. 2017.
• Neural ODEs - Parameterizes a continuous-time vector field and integrates it with an ODE solver. The solver, tolerances and gradient calculation become part of the learned system; a neural vector field need not preserve physical invariants. Chen et al. 2018.
  • Latent ODEs - Uses continuous latent dynamics and an ODE-RNN inference network for irregular observations. Can model observation times with a Poisson process; irregular sampling support does not itself identify the underlying physical state. Rubanova et al. 2019.
• Hamiltonian neural networks - Learns a scalar Hamiltonian and derives dynamics through canonical derivatives. Conservation follows for the learned continuous system; numerical integration can still introduce energy drift. Greydanus et al. 2019.
• Lagrangian neural networks - Learns a Lagrangian and obtains dynamics from Euler-Lagrange equations. Uses positions and velocities without requiring canonical momenta; the learned dynamics and numerical solver still need validation. Cranmer et al. 2020.
• Reservoir dynamics models - Fixed recurrent states and delayed polynomial features provide alternatives to fitting a neural vector field. See reservoir computing for the methods and chaotic-system evidence.

Data assimilation and digital twins

• Adjoint variational assimilation - Fits a dynamically constrained trajectory to observations across time. A forward model run and backward adjoint compute gradients with respect to the initial state, avoiding one perturbation run per control variable. Le Dimet and Talagrand 1986.
• Ensemble Kalman filter - Propagates an ensemble through the dynamics and uses its sample covariances to update states from observations. The analysis approximates nonlinear Bayesian filtering; finite ensembles also introduce sampling error. Evensen 1994, Evensen 2003.
• Filtering connections - See Kalman filtering for linear state estimation and Sequential Monte Carlo for particle filtering and smoothing. Ensemble covariance updates and weighted-particle approximations retain different information about uncertainty.
• Ensemble Kalman inversion - Repurposes ensemble updates for derivative-free inverse-problem optimization. Estimates remain in the span of the initial ensemble, making its choice a restriction on recoverable parameters or fields. Iglesias et al. 2013.
• Probabilistic digital twins - Uses a dynamic decision network to update uncertain asset states and guide control. The UAV study calibrates on experimental data; in-flight damage tracking and planning use synthetic observations. Kapteyn et al. 2021.

Inverse problems and uncertainty

• Simulation-based inference - Uses simulations when likelihood evaluation is unavailable, including learned likelihoods, likelihood ratios and posteriors. Inference concerns parameters under a simulator; posterior confidence does not validate the simulator itself. Cranmer et al. 2020.
• Bayesian PINNs - Combines equation constraints and noisy observations in Bayesian inference. HMC gives better uncertainty estimates than tested variational/dropout alternatives in the paper; calibration remains specific to likelihood, prior and model adequacy. Yang et al. 2021.

Sequence-to-function genomics

• Basenji - Uses dilated convolutions to predict quantitative chromatin and transcription profiles from DNA. Enlarging the receptive field improves held-out prediction; variant scores align with measured expression associations across 19 tissues. Kelley et al. 2018; code.
• Enformer - Uses attention to integrate regulatory DNA up to 100,000 bases away. Across held-out genes, mean correlation with measured transcription rises from 0.81 with Basenji2 to 0.85; enhancer prioritization also improves against CRISPR perturbation data. Avsec et al. 2021; code.
• Borzoi - Extends Enformer to predict RNA-seq coverage from 524,000-base DNA windows. Coverage supports scoring effects on expression, splicing and RNA processing; adding chromatin-accessibility training improves expression-variant and enhancer-linking predictions. Linder et al. 2025; code.
• AlphaGenome - Jointly predicts expression, splicing and chromatin features from one million DNA bases, with outputs up to single-base resolution. Matches or exceeds the strongest tested external models on 25 of 26 variant-effect benchmarks. Avsec et al. 2026; code.
  • Variant-effect underestimation - On selected GTEx whole-blood genes, AlphaGenome often gives candidate causal variants weak effects relative to neighboring variants. Underestimation persists in a 1-Mb input check. Drusinsky and Pollard 2026.

Comparative evidence

• PDEBench - Compares forward and inverse PDE tasks with FNO, U-Net and PINN baselines. Metrics include boundary, conservation and frequency-dependent errors; a single aggregate field error can conceal physically relevant failures. Takamoto et al. 2022.
• Personal gene-expression prediction - Across 421 people, Enformer and three other models predict differences between genes but have near-zero mean correlation for differences between individuals. Some genes show reversed predicted effects. Huang et al. 2023.
• PINNs versus finite elements - FEM was generally faster at equal or better accuracy in the tested PDEs; 1D Schrödinger results were inconclusive. Faster PINN point evaluation appeared only for 3D Poisson, after training. Grossmann et al. 2024.
• Mass-balance audit - In two steady-state 1D groundwater cases, PINNs selected from 10,800 configurations each fit hydraulic heads well but retain larger mass-balance errors than finite volumes. Accurate fields did not ensure local or global conservation. Mamud et al. 2024.
• The Well - Provides 16 diverse simulation datasets with common interfaces. Baselines expose differences across physical systems; evaluate rollouts and dataset-specific dynamics rather than treating performance on one fluid equation as general simulation ability. Ohana et al. 2024.
• Perturbation-response prediction - Tested deep models did not outperform simple baselines on gene perturbations. A cell-line follow-up finds gains with metrics focused on perturbation effects. Additive baselines remain strong for near-additive double perturbations. Ahlmann-Eltze et al. 2025, Miller et al. 2026.
• Neural-solver breakeven - Counts solves needed to repay data-generation and task-training cost against an error-matched classical solver. Some tested periodic PDE/model pairs need hundreds of thousands of calls; others never break even. Foundation-model pretraining is excluded. Zhang et al. 2026.

Software

• DeepXDE - Implements PINNs, operator networks and adaptive sampling for differential equations. Supports forward and inverse problems; equation residuals, boundary conditions and sampling rules remain user-specified. docs.

Related topics

• Geometric deep learning
• Symbolic regression
• Model predictive control
• Bayesian inference
• Neural nets
• Evaluation