Fig. 6: Heat flow across a more complex domain composed of three holes of
varying sizes and positions, as well as curved left and right boundaries.
Credits: Manuel Madeira / Inductiva.
## Generalizing through grid-searching (is inefficient)
To make a point (and get more awesome visualizations 😁), let's see how the
output changes by changing the diffusitivity rate $D$ for the hot hole scenario.
For that, we simply run each PDE for each $D$:
Fig. 7: Heat diffusion profile in the holed domain by ramping up the
diffusitivity from $D=0.01$ [top] to $D=0.1$ [middle] and $D=1$ [bottom]. Note
that each PINN had to be trained individually. Credits: Manuel Madeira /
Inductiva.
_Hot_! We can see that the diffusitivity allows us to disentangle both phenomena
streams we discussed (downward vs radial). Additionally, it sets the time scale
for _equilibration_, when the state becomes _stationary_ _i.e._ whenever
$\frac{\partial u}{\partial t} = 0 $
The main point to grab here though is that the output you see comes after
training each PINN **individually**! For instance, for these settings, each PINN
will take **about 5 hours**. This is completely **inefficient** and does not
allow the algorithm to organically understand how to map the IBCs to the output.
### The PINN architecture behind generalization
In order to make generalization tangible, the computing infrastructure needs to
be versatile and efficient. From a computational perspective, we should ask:
> How exactly does the PINN change to accommodate for this generalization?
The fact that the loss function differs according to the domain from where the
data domain points considered are sampled is of **huge computational
relevance**.
While a initial or boundary point is directly fit by the PINN to its target
(imposed by the initial or boundary condition itself), a point stemming from an
interior domain contributes to the fitting procedure through its PDE residue.
PINNs do not impose an upper limit to the number of IBCs or interior domains.
Each of these IBCs may have a different target and each interior domain might be
assigned to different PDEs. As you can imagine, for complex problems, PINNs have
a high chance of turning into a mess!
Let's make use of a perk from the IDRLnet library (which supported us throughout
the Heat series) --- a visual representation of the computational graph (in
terms of `IDRLnet nodes`) underneath the implementation it receives.
For this our instance, the representations obtained can be visualized as:
Fig. 8: Computational graphs considered by IDRLnet for each sampling domain
considered. If we added holes to our plate, an extra graph would be obtained
(similar to the ones from the IBCs). Credits: Manuel Madeira / Inductiva.
Note that the blue nodes are obtained by sampling the different domains
considered (DataNodes), the red nodes are computational (PDENodes or NetNodes),
and the green nodes are constraints (targets). [2]
Having a graphical representation of what is going on inside our code is always
helpful. Naturally, this tool may become extremely handy to ensure that the
problem solution is well implemented.
## Conclusion
Well, it has been quite a ride! To finish off the tutorial, we took the
opportunity to sail through seas that classical methods can not achieve (or at
least through simple procedures). The reason is that they are not scalable with
increasing parameter customization. Classical methods have underwhelming
potential in providing acceleration into solving PDEs.
We argue Neural Networks have the capability of streamlining these
high-dimensional increments by generalizing more smartly. But the complexity has
been transferred to other issues. As we alluded before, the choice and
fine-tuning of variables and parameters is not something trivial to achieve
(either through classical and DL frameworks).
To see how versatile NNs can be, we pushed a little harder and checked if adding
more complexity could be coped by Physics-Informed Neural Networks (PINNs). In a
modest setup of incremental difficulty, we sure have explored a lot of
situations and probed potential means to achieve smarter training.
This strategy of adding new parameters as simple inputs to achieve
generalization is the simplest one (but arguably not the most efficient one).
There are several alternatives --- a great example is _HyperPINNs_ [1], whose
results have been published recently. They result from the fusion of
hypernetworks and the more conventional PINNs. Most importantly, HyperPINNs have
been shown to succeed in achieving generalization --- although in a different
implementational flavor.
The outlook message to us is simple to state. The issues pointed out and run
experiments illustrate two essential aspects that will need to continue being
optimized are different in nature:
- the power that Machine/Deep Learning techniques have to propel scientific
computing to unseen new paradigms;
- the challenges in computation and algorithm architecture caused by evermore
realistic and refined systems of interest.
## References & Remarks
[[1]](https://arxiv.org/abs/2111.01008) A great introduction to HyperPINNs!