Worst part so far: Traveling on the map + resting system + encumbrance. Anytime you want to go anywhere, you have to travel on a map with lots of random encounters. You also often need to rest when traveling on the map, and of course, resting has random encounters, too! This is all before you can even start a quest. You can build some fast travel teleportation circles, but they take forever to build in multiple places and also have a cooldown. Once when I was leaving an area, it said I couldn't leave unless I dropped some loot!
Sounds very much like Kingmaker. I think I wouldn't mind world map that much in that game, if locations and quests were more meaty. It was a lot of loadscreens and slogging back and forth on the world map in that game.
Give everyone an animal companion and the weight frustrations go away. I give Seelah a level of beast rider right away.
Building teleportation circles reduces frustrations about map travel