Ogre Magi aren't the same kind of creatures as Ogres (unless their design was changed in 5e). They're more based on Japanese Oni spirits than anything else, they just share the ogre part in the name.
Though that said I also find them a lot closer to the real Swedish folklore trolls than what they call trolls in-game...
