I even did it accidentially. So I had to flee sometimes and either come back better prepared or sneak if I desparately wanted to go somewhere I was not supposed to go with the level of my party. No problem at all.

Regards,
Thorsten