If I recall correctly, it takes the spell casting stat of the last class you took a level with.

So if you want it to use Charisma, respec and get a Sorcerer level after your 3 Cleric levels.