Yeah, I think it's worth also asking whether humans /actually/ are any good at answering the 'why' with anything but bullshit. I would argue that we're pretty good at understanding causality in very limited circumstances (the window broke because that kid threw the ball), and extremely overconfident in our ability to understand causation in a much broader range of circumstances (the stock price went up because...). This overconfidence drives a lot of the decisions we make, for better or for worse.

It's an area where if we push hard on AI, we'll likely have to come to terms with how bad we are in this area, and ask ourselves whether we feel comfortable deploying 'thinking machines' with similar levels of incompetence and/or arrogance.