You can refuse to help him and eventually it'll give you a choice to part ways permanently. Alternatively, have him wait in camp by talking to him. Lil him if you're mad and just don't res him.