I've got a similar bug right now. I entered the bandit camp and started a couple quests there, but I can't seem to leave. The door to exit is missing, there is just a stone wall. Thought it may have been some part of a quest and loaded my save from before I entered and went back to town only to find all the doors missing in town as well.
I agree that this seems like it could be a really good game, but these bugs are really dragging the game down. Hopefully there will be a patch to fix them soon.
