There is a detour after you leave the castle, but you can get back to anything you have left to do. As Stabbey said being at a higher level is better, but level 16 is not too low. You will meet the imp historian again and get the teleporter stones back (after you get out of the next area, you will receive a quest to rescue Zandalor; after you do so, he will teleport you to a new area when you are ready, and you can find the imp there after you clear it).
I did most of the quests in Rivertown and Verdistis and cleared the Rivertown sewers before heading to the castle (clearing the sewers is easier when you still have your teleporer stones, and having the sewer clear makes it faster to rescue Zandalor).
You may want to put any loot and miscellaneous stuff from your inventory in a chest and toss it out the front gate before finishing off the castle, even if you stashed stuff at your base before entering the castle. You will probably get a fair bit of loot in the next area, and if you don't clear out unnecessary stuff you would be more likely to become encumbered (though you can always stash stuff in a barrel, put a flag on your map and come back for it later).
The only 'point of no return' in the game is the blessing ceremony, which requires a few quests you will get from Zandalor.