Yes, it would likely help to use the map a little more, for awhile (you can place custom map markers with notes). Once you walk through a few times you'll become more familiar with the town, and should not have that problem.
You can use the WASD or arrow keys to move the screen focus where you want to go, and then click on the ground there to move your characters.
The Home key will recentre the camera on the lead character, and return it to the default angle of rotation. That may help you get your bearings when checking the map.
