I’m abstaining on this one for now, though possibly someone might say something that makes my mind up one way or the other. I personally don’t have any problem with him hanging round my camp minding his own business, and do tend to use his services. Though I don’t know why I hoard the scrolls when I know I’ll not need them all! I’m also looking forward to finding out more about Jergal’s connection to the story.
But even if Withers does have an important story role to play, and I’m not sure if he’ll appear even if you don’t visit his temple in act 1, it still feels as though you could be given an option to ask him to go and he’ll say his “we’ll meet again in another time and place” line and just pop up again later when he has some lines.
I struggle to think how persuading him to hide somehow would feel realistic, though, and I don’t think we should be given a means to kill him unless that’s actually part of/a branch of the story anyway.