Spells from items use the last spell casting stat so any scrolls/ staffs etc cast with your Intel
If you added a bard, sorcerer or warlock level then from that point on your charisma will be what scrolls use
Yes taking another cleric slows down your wizard but it delays when a feat becomes available and what maximum spell slot i.e you won't have max level spells in the late game unless you cast from a scroll but your actual DC should be fine
