Applications of Ordinary and Partial Differential Equations in Machine Learning and Data Science
Differential equations have emerged as fundamental mathematical tools in machine learning and data science, providing both theoretical foundations and practical methodologies for solving complex problems. This comprehensive review explores the applications of ordinary differential equations (ODEs) and partial differential equations (PDEs) across machine learning domains, including neural network optimization, generative modeling, and physics-informed learning. We discuss continuous-time perspectives on gradient descent, Neural Ordinary Differential Equations as continuous-depth models, diffusion-based generative modeling through stochastic differential equations, Physics-Informed Neural Networks for scientific computing, and flow-based models. This paper demonstrates how differential equations bridge classical mathematics and modern machine learning, providing powerful theoretical insights and practical improvements in model development and training.
Keywords
Ordinary differential equations, Partial differential equations, Neural networks, Optimization, Physics-informed learning, Generative models, Stochastic differential equations
[1] R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, “Neural ordinary differential equations,” in Advances in Neural Information Processing Systems, 2018, pp. 6571–6583.
[2] L. Ruthotto and E. Haber, “Deep neural networks motivated by partial differential equations,” arXiv preprint arXiv:1804.04272, 2018.
[3] W. Su, S. Boyd, and E. J. Candes, “A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights,” J. Mach. Learn. Res., vol. 17, no. 153, pp. 1–43, 2016.
[4] T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, “Neural ordinary differential equations,” in Advances in Neural Information Processing Systems, vol. 31, 2018, pp. 6571–6583.
[5] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” in Proc. Int. Conf. Learn. Represent., 2021, pp. 11237–11307.
[6] M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” J. Comput. Phys., vol. 378, pp. 686–707, 2019.
[7] B. Polyak, “Introduction to optimization,” in Optimization Software Publications, 1987.
[8] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, no. 6088, pp. 533–536, 1986.
[9] H. K. Khalil, Nonlinear Systems, 3rd ed. Upper Saddle River, NJ: Prentice Hall, 2002.
[10] A. Lyapunov, The General Problem of the Stability of Motion. Taylor & Francis, 1992.
[11] C. Jin, P. Netrapalli, and M. I. Jordan, “Accelerated gradient descent escapes saddle points faster than gradient descent,” in Proc. 31st Conf. Learn. Theory, vol. 75, 2018, pp. 1042–1085.
[12] A. Wibisono, A. C. Wilson, and M. I. Jordan, “A variational perspective on accelerated methods in optimization,” Proc. Natl. Acad. Sci., vol. 113, no. 47, pp. E7351–E7358, 2016.
[13] L. Chizat and F. Bach, “On the global convergence of (fast) incremental expectation maximization methods,” in Adv. Neural Inf. Process. Syst., 2018, pp. 4040–4048.
[14] J. M. Welling and Y. W. Teh, “Bayesian learning via stochastic gradient Langevin dynamics,” in Proc. Int. Conf. Mach. Learn., 2011, pp. 681–688.
[15] S. Raginsky, A. Rakhlin, and M. Telgarsky, “Non-convex learning via stochastic gradient descent: a nonasymptotic analysis,” in Proc. 30th Conf. Learn. Theory, vol. 49, 2017, pp. 1674–1703.
[16] P. Chaudhari, A. Choromanska, S. Soatto, and Y. LeCun, “Entropy-SGD: Biasing gradient descent into wide minima,” arXiv preprint arXiv:1611.01838, 2016.
[17] A. Nesterov, “Introductory lectures on convex optimization: A basic course,” in Applied Optimization, vol. 87. Springer, 2004.
[18] W. Su, S. Boyd, and E. J. Candes, “A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights,” J. Mach. Learn. Res., vol. 17, no. 153, pp. 1–43, 2016.
[19] Y. Nesterov, “A method for unconstrained convex minimization problem with the rate of convergence O(1/k^2),” Doklady SSSR, vol. 269, pp. 543–547, 1983.
[20] C. D. Dang and G. Lan, “Stochastic first-and zeroth-order methods for nonconvex stochastic optimization,” SIAM J. Optim., vol. 28, no. 4, pp. 2896–2916, 2018.
[21] Y. Rubanova, R. T. Chen, and D. K. Duvenaud, “Latent ODEs for irregularly-sampled time series,” in Advances in Neural Information Processing Systems, vol. 32, 2019, pp. 6571–6583.
[22] J. Kelly, J. Bettencourt, M. T. Johnson, and D. K. Duvenaud, “Learning differential equations that are easy to solve,” in Advances in Neural Information Processing Systems, 2020, pp. 4370–4380.
[23] G. Grathwohl, R. T. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud, “FFJORD: Free-form continuous dynamics for scalable reversible generative models,” in Advances in Neural Information Processing Systems, 2019, pp. 12400–12409.
[24] P. Kidger, J. Morrill, J. Foster, and T. Lyons, “Neural controlled differential equations for irregular time series,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 6696–6707.
[25] Y. Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” in Advances in Neural Information Processing Systems, vol. 32, 2019, pp. 11895–11907.
[26] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” arXiv preprint arXiv:2011.13456, 2021.
[27] M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics informed deep learning (Part I): Data-driven solutions of nonlinear partial differential equations,” arXiv preprint arXiv:1711.10561, 2017.
[28] S. Wang, X. Yu, and P. Perdikaris, “When and why PINNs fail to train: A neural tangent kernel perspective,” J. Comput. Phys., vol. 449, p. 110768, 2022.
[29] S. Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli, “Scientific machine learning through physics-informed neural networks: Where we are and what’s next,” J. Sci. Comput., vol. 92, no. 3, p. 88, 2022.
[30] Y. Shin, J. Darbon, and G. E. Karniadakis, “On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs,” Commun. Comput. Phys., vol. 28, no. 5, pp. 2146–2186, 2020.
[31] D. Rezende and S. Mohamed, “Variational inference with normalizing flows,” in Proc. Int. Conf. Mach. Learn., 2015, pp. 1530–1538.
[32] D. Cordero-Erausquin, B. Nazaret, and C. Villani, “A mass-transportation approach to sharp Sobolev and Gagliardo-Nirenberg inequalities,” Adv. Math., vol. 182, no. 2, pp. 307–332, 2004.
[33] D. Kingma and P. Dhariwal, “Glow: Generative flow for invertible 1×1 convolutions,” in Advances in Neural Information Processing Systems, 2018, pp. 10215–10224.
[34] B. D. O. Anderson, “Reverse-time diffusion equation models,” Stochastic Process. Appl., vol. 12, no. 3, pp. 313–326, 1982.
[35] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar, “Fourier neural operator for parametric partial differential equations,” in Proc. Int. Conf. Learn. Represent., 2021, pp. 5866–5879.
[36] L. Lu, P. Jin, and G. E. Karniadakis, “DeepONet: Learning nonlinear operators for identification and control of dynamical systems,” arXiv preprint arXiv:1910.03193, 2019.
[37] P. Goyal and P. Benner, “Robust optimal control with uncertainty quantification for stochastic dynamical systems,” Nonlinear Dyn., vol. 95, pp. 495–511, 2019.
[38] J. Pathak et al., “FourCastNet: A global data-driven weather forecasting model,” arXiv preprint arXiv:2202.11575, 2022.
[39] H. Zhang, X. Gao, J. Unterman, and T. Arodz, “Learning latent dynamics for efficient probabilistic consensus,” in Proc. Int. Conf. Mach. Learn., 2021, pp. 12305–12315.
[40] B. M. de Silva, K. Champion, M. Quade, J.-L. Loiseau, J. N. Kutz, and S. L. Brunton, “PySINDy: A Python package for the sparse identification of nonlinear dynamical systems from data,” J. Open Source Softw., vol. 5, no. 49, p. 2104, 2020.
[41] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 6840–6851.
[42] P. Marion, A. Barbier, and N. P. Stucki, “Generalization bounds for neural ordinary differential equations and deep residual networks,” arXiv preprint arXiv:2305.06648, 2023.
[43] P. Marion, “Generalization bounds for neural ordinary differential equations and deep residual networks,” in Proc. Int. Conf. Learn. Represent., 2023.
[44] H. Zhang, M. Cai, and D. K. Duvenaud, “Neural network-based score estimation in diffusion models: Optimization and generalization,” arXiv preprint arXiv:2401.15604, 2024.
[45] H. Zhang, Z. Zhang, M. Caldana, and E. Haber, “Neural ordinary differential equations for model order reduction of stiff systems,” arXiv preprint arXiv:2408.06073, 2024.
[46] Z. Hong, L. Zhang, and Z. Shen, “Semi-implicit neural ordinary differential equations,” arXiv preprint arXiv:2412.11301, 2024.
[47] L. Ruthotto and E. Haber, “Differential equations for continuous-time deep learning,” arXiv preprint arXiv:2401.03965, 2024.
[48] S. Ermon and Y. Song, “Improved techniques for training score-based generative models,” in Advances in Neural Information Processing Systems, vol. 33, 2020.
[49] A. D. Jagtap and G. E. Karniadakis, “Extended physics-informed neural networks (XPINNs): A deep learning framework for surrogate modeling and parameter discovery,” Commun. Comput. Phys., vol. 28, no. 5, pp. 2002–2041, 2020.
[50] D. Zhang, L. Guo, and G. E. Karniadakis, “Learning the solution operator of parametric partial differential equations with physics-informed DeepONets,” Sci. Adv., vol. 7, no. 40, p. eabi8605, 2021.
[51] P. Kidger, R. T. Chen, and T. Lyons, “Hey, that’s not an ODE: Faster ODE Adjoints via Incorporeal Backpropagation,” in Proc. Int. Conf. Mach. Learn., 2021, pp. 5443–5452.
A. Gupta, “Applications of ordinary and partial differential equations in machine learning and data science,” IPEM Journal of Computer Application & Research, vol. 10, pp. 114–121, Dec. 2025doi: