Unless you think 4 hp with no armour is as good as 5hp with heavy armour you've got to see that getting full access to the wizard spellbook is something you've got to pay for. A cleric 8 wizard 4 with 8 wis and int focus is just strictly better than a wizard 12 with the same stat spread, even if they never touch the cleric spellbook. There are a bunch of other best of both worlds you can get if spell advancement isn't slowed. Sorceror or Wizard? Why not a Sorwizard who has almost full access to metamagic while having a full sized wizard spellbook and no downsides? For the cost of a feat you can be a Clerwizeror who has almost full metamagic, full wizard spells, full cleric spells and heavy armour.
