The Limitations of Machine Learning in Replacing Physical Laws: Expanding the Critique

Tue, 22 Apr 2025 00:00:00 +0000

Before diving into my analysis, I want to acknowledge the insightful blog by Mehmet Süzen that inspired this post, which eloquently discusses the fallacy of replacing physical laws with machine-learned inference systems. Having read it, I felt compelled to share my own perspective and expand on these critical arguments with additional examples from recent literature and research.

The Fundamental Problem of Circular Reasoning

The original blog brilliantly identifies the circular reasoning inherent in claiming that machine learning systems can discover or replace physical laws. This point deserves further emphasis: when a neural network is trained on data generated by known physical principles, it cannot be said to “discover” those same principles through inference.

Consider recent work in fluid dynamics, where physics-informed neural networks (PINNs) have gained popularity. The paper “Discovery of Physics From Data: Universal Laws and Discrepancies” highlights that “the naive application of ML/AI will generally be insufficient to infer universal physical laws without further modification”. The authors demonstrate this by examining falling objects, showing that measurement noise and secondary mechanisms (like fluid drag) obscure the underlying law of gravitation, leading to erroneous models that might suggest an Aristotelian theory where objects fall at speeds related to their mass, rather than identifying the true universal gravitational constant.

This illustrates perfectly how ML systems trained on physical data will incorporate all the complexities and noise present in that data, rather than abstracting to the elegant, universal laws that human scientists have carefully identified through theoretical reasoning and controlled experimentation.

Beyond Narrow Applications: The Generalization Problem

The original blog correctly identifies the problem of faulty generalization. Machine learning algorithms excel at computational acceleration within narrowly defined parameter spaces, but struggle with broader generalization.

A fascinating discussion on Reddit highlights this limitation: “In addition, the ‘marginally-better SOTA’-esque papers with no novel methods or aspects besides some parameter tuning or adding extra layers to the DNN are also tiring to read. The wall of math then exists only to provide a sense of rigor and novelty, obscuring the iterative nature lacking novelty”. This reflects how ML approaches in physics often claim breakthroughs that are actually just incremental improvements in limited domains.

Another illustrative example comes from the field of symbolic regression. While the AbdusSalam et al. paper in Physical Review D demonstrates how symbolic regression can help derive analytical expressions for physics beyond the Standard Model, the authors position it as a tool to assist numerical studies, not as a replacement for physical theory. The expressions derived still rely on the underlying physics-based model (the constrained minimal supersymmetric Standard Model) and serve primarily to accelerate computation, not to discover new physical laws.

The Irreplaceable Role of Scientists in Establishing Causality

Perhaps the most important point from the original blog is that causality still requires scientists. Machine learning excels at finding correlations but struggles with identifying true causal relationships.

The Amazon Science blog on physics-constrained machine learning notes that “the predictions of deep-learning models trained on physical data typically ignore fundamental physical principles. Such models might, for instance, violate system conservation laws” (see here). This highlights why human scientists remain essential - they understand that physical laws must adhere to conservation principles, symmetries, and other fundamental constraints that ML systems don’t inherently respect.

A conversation on Reddit about Physics Informed Neural Networks (PINNs) further illuminates this issue. One commenter precisely notes: “The point of including a physical loss function, in addition to a data-driven loss, is to impose inductive bias into the training process”. This human-guided approach to incorporating physics into ML demonstrates that we’re not replacing physics with ML, but rather using our understanding of physics to guide ML - the exact opposite of what some overenthusiastic claims suggest.

The Scientific Machine Learning Fallacy: A Deeper Look

The term “Scientific Machine Learning Fallacy” coined in the original blog deserves broader recognition. Claims of “machine scientists” or “automated scientific discovery” fundamentally misunderstand the nature of scientific inquiry.

A recent paper on “Scientific machine learning for closure models in multiscale problems” acknowledges that “the generalizability and interpretability of learned models is a major issue that needs to be addressed further”. This admission from researchers in the field underscores the gap between current ML capabilities and true scientific discovery.

The Conversation article about an “AI scientist” further reveals the limits of these approaches. While Sakana AI Labs claims their system can “make scientific discoveries in the area of machine learning in a fully automated way,” the article questions whether such a system can produce truly “interesting” scientific papers, noting that “good science requires novelty”. The ability to generate papers that look like scientific literature doesn’t equate to generating novel scientific insights or laws.

The AutoML Misnomer and Meta-Scientific Work

I strongly agree with the original blog that “AutoML” is a misnomer in scientific contexts. These systems don’t replace scientists but rather change the nature of scientific work.

The paper on “Combining physical modeling and machine learning for micro-scale modeling of a fuel cell electrode” demonstrates this well. It describes a “comprehensive transition from white-box models, characterized by their reliance on physical laws, to black-box models exemplified by neural networks”. Yet the core contribution isn’t replacing physics but creating a “synergistic integration” where neural networks complement physical modeling.

This represents what the original blog aptly calls “MetaML” - a transformation of scientific workflows rather than a replacement of scientific thinking.

The Proper Role: Augmentation, Not Replacement

To conclude, I believe the most productive path forward is viewing machine learning as an augmentation to physical sciences, not a replacement. The paper on “Learning physical laws: the case of micron size particles in dielectric fluid” demonstrates this approach well, noting that “representation structure is key in learning generalizable models”. The authors use “the port-Hamiltonian formalism as a high level model structure” that is “continuously refined based on our understanding of the physical process.” This integration of physics understanding with machine learning represents the right approach.

Similarly, the work on “Discovering Artificial Viscosity Models for Discontinuous Galerkin Approximation of Conservation Laws” shows how physics-informed machine learning can automate the discovery of models - but within a physics-informed framework, not replacing it

In summary, while machine learning offers powerful tools for scientific research, the fallacy of replacing physical laws with learned models deserves continued critical attention. True scientific progress will come from the thoughtful integration of machine learning with physical understanding, not from claims that ML can autonomously discover or replace the fundamental laws of nature. The original blog’s warning about circular reasoning, faulty generalization, and the continued need for human scientists remains prescient and worthy of expansion as these technologies continue to develop.

Human Thinking, Not Machine Imitation

This philosophical depth reminds us that true scientific thinking involves not just pattern recognition and prediction, but deep conceptual understanding that may not be reducible to computational processes. When we forget this, we risk confusing the map (our mathematical models and computational simulations) with the territory (physical reality itself).

The Hitchhiker’s Guide to Rebranding Machine Learning (And a Shoutout to Geoff Hinton!)

Thu, 12 Dec 2024 00:00:00 +0000

Ever feel like the world of machine learning is just a little too… techy? Well, let’s give that jargon a fresh coat of paint; one with a physics twist.

Welcome to my Hitchhiker’s Guide to Rebranding Machine Learning! 🚀

Let’s face it: sometimes, explaining ML to your friends sounds like you’re reading from a sci-fi manual. So, why not embrace it? Here’s my tongue-in-cheek translation table for the next time you want to sound like a quantum physicist at a data science meetup:

Machine learning? Nah, let’s call it statistical mechanics.
Loss function? That’s just an energy functional.
Optimize the model? We’re really just minimizing free energy.
Trained model? You’ve simply reached equilibrium distribution.
KL divergence? That’s the free energy difference, obviously.
Gaussian noise? Those are just random thermal fluctuations.
Random step? Welcome to Brownian motion.
SGD? Think of it as directional Brownian motion.
GPU? Basically a simulated particle accelerator.
Diffusion models? That’s Langevin dynamics for you.
LLM (Large Language Model)? Try high-order discrete Markov chain on for size.
NLP? Let’s call it “string theory” (because why not?).
Reinforcement learning? That’s just control theory.
Robotics? It’s all physical computation.
Audio learning? 1D signal processing.
Image learning? 2D signal processing.
Video learning? You guessed it: 3D signal processing.
Multimodal models? We’re talking multidimensional signal processing.

You’re welcome. 😉

A Quick Geek-Out: Congrats to Geoff Hinton!

Before I wrap up, I have to give a huge shoutout to Geoff Hinton, who just became the second person ever to win both the Turing Award and a Nobel Prize! (Herbert A. Simon was the first, winning in Economics.) Legends, both of them.

So, the next time someone asks what you do, feel free to say you’re working on “minimizing free energy in high-order discrete Markov chains using simulated particle accelerators.” If nothing else, you’ll sound like you’re saving the universe.

Physics | Mahyar's world 🌏

The Limitations of Machine Learning in Replacing Physical Laws: Expanding the Critique

The Fundamental Problem of Circular Reasoning

Beyond Narrow Applications: The Generalization Problem

The Irreplaceable Role of Scientists in Establishing Causality

The Scientific Machine Learning Fallacy: A Deeper Look

The AutoML Misnomer and Meta-Scientific Work

The Proper Role: Augmentation, Not Replacement

Human Thinking, Not Machine Imitation

The Hitchhiker’s Guide to Rebranding Machine Learning (And a Shoutout to Geoff Hinton!)

Welcome to my Hitchhiker’s Guide to Rebranding Machine Learning! 🚀

Machine learning? Nah, let’s call it statistical mechanics.

Loss function? That’s just an energy functional.

Optimize the model? We’re really just minimizing free energy.

Trained model? You’ve simply reached equilibrium distribution.

KL divergence? That’s the free energy difference, obviously.

Gaussian noise? Those are just random thermal fluctuations.

Random step? Welcome to Brownian motion.

SGD? Think of it as directional Brownian motion.

GPU? Basically a simulated particle accelerator.

Diffusion models? That’s Langevin dynamics for you.

LLM (Large Language Model)? Try high-order discrete Markov chain on for size.

NLP? Let’s call it “string theory” (because why not?).

Reinforcement learning? That’s just control theory.

Robotics? It’s all physical computation.

Audio learning? 1D signal processing.

Image learning? 2D signal processing.

Video learning? You guessed it: 3D signal processing.

Multimodal models? We’re talking multidimensional signal processing.

A Quick Geek-Out: Congrats to Geoff Hinton!