You can not get into that camp before going to the castle; a subsequent quest opens the gates.

Clear the Rivertown sewers first, and it will make getting the teleporter stones back easier after you get out of the castle.