THat depends on you entirely. If you get the 3 signs you can have the doll restored and you lose it. Or, if you want to keep the doll as a mule just forget the restoring part and go on... there will be no problem for you, just a loss of a little exp.
I chose to keep the Doll for use as a mule and to mess around with later after the game.
