I allways thought its bcs it felt odd that once you wield your shield in hand, it gives you armor ... but once you put it on your back, its just like if it dissapeared.

Well, graphically it does just disappear I guess!
I suppose I can see an argument for a reduced AC benefit for just having a shield on your back, but it doesn’t feel at all the same as wielding one, with the flexibility to move the shield to catch and block attacks that you can see coming.