Although I will posit that physical exhaustion is typically more manageable, so that helps. There are certainly exceptions, but by and large in our modern, sedentary world, physical exertion can easily be limited to controlled environments where one is able to stop when they reach the point of exhaustion. It is unlikely that stepping outside means that you will have to outrun a lion, unable to pause else become lunch. Chances are these days you will only ever reach exhaustion if you purposefully push yourself to (e.g. hitting the gym). But if I have reason to worry that I am going to find myself in a situation where I cannot meaningfully rest when I reach my physical breaking point, absolutely the anxiety will be running hot.
Whereas social interaction can be harder to control. For example, reaching your social limit in the middle of the workday does not usually, for all practical purposes, enable you to remove yourself from the situation. You are expected to keep going until the end of the business day, no matter how you are feeling on the inside. Us techies might have enough individual work to slink away to, perhaps, but for many jobs there is no such luxury. Or you might come up against a Chatty Cathy who will continue pushing to keep the social engagement going even after you try to back away. There aren't all that many HN-esq situations in the real world where you can just magically pop in when you want to engage and then disappear as fast as you came. And where you might find something approximate, you are apt to find it much too limiting to be an exclusive social outlet.
