As a thought experiment, imagine an AI is trying to work out if a rooster crowing makes the sun rise or the other way around (because they are always correlated in its experience). It can do the experiment of putting the rooster in a bag and see if the sun doesn't rise. It can refine it's model after doing this experiment (all with optimization). It is more about untangling correlations that come from latent variables that the system doesn't know about yet.
Well, that sounds an awful lot like an optimization problem, and is actually reminescent of a Machine Learning technique called Active Learning: the model suggests data points for which it has low confidence (say images that resembles both a cat and dog according to its own understanding), and a human labels those (or in your case: the AI itself runs an experiment to get the ground truth) to gain the most information.
It's not quite the same as active learning. Experimentation and active learning are quite different in ways I didn't fully appreciate at the beginning of my career, and they come in the form of (conditional) independence relations, which you can get around with either structural assumptions or randomized intervention.
It's the entire reason why we randomize in experiments. In active learning, the machine doesn't know what happens when the rooster is in a bag, so decides to try it. In a randomized experiment, the machine knows what happens when the rooster is in a bag, but thinks there might be other factors at work, so it decides to try it in a way that it can be sure that all other factors are equal (at least on average).
To be considered a generalized AI, it does not just have to run experiments, it has to create hypotheses and devise experiments that will both test them and reject the null hypotheses, which brings us right back to reasoning about causality.
The thing is that "not knowing casualty" along with "lack of understanding" feels very plausible for an explanation of what current AI is missing. And while AI has made lots of progress, it certainly feels like this progress is all of a certain kind (though that could the human reflex to discount any problem solved by robot).
The problem might be that "understanding" isn't something we really have a good working definition of.
