My mind always jumped to the far more awkward question of "just how far does the stoneskin go.. and wouldn't that be uncomfortable?" It's like... we don't just have an 'outside' and an 'inside' with a clear-cut hard defining line, and we don't just have our skin as a distinct, discrete piece of anatomy... we've got orifices, and all of that, and the transition form exterior skin to otherwise is not exactly cut and dry... seems potentially very uncomfortable to me.
Maybe the stoneskin affects out clothing as a tacit way of saying it doesn't affect what's underneath, so the don't have to worry about those oddities...
On the other hand, if it only affects the body and not the garments, then if it's doing its job and stopping slicing blades and the like from harming you, then there's a good chance that all those unprotected clothes are getting cut up, and potentially cut into rags that won't stay on... so enjoy dealing with that issue when the spell ends and your soft places turn soft again...
These are the things I think about all the time... the problem may be with me.