The shame is inherent to the crushing expectations put on women's appearances and the pressure to be small. It manifests in clothing sizing for the same reason it manifests standing on a scale, it's a measure of your smallness. And what makes it insidious is that the measures are juuust comparable enough across different people to make people feel bad for not having the same numbers as someone 5" shorter than you.

And my experience isn't unique in any way here and it's really hard to not see it pervasive through our culture.