It would be nice if the traveling to the camp was a gameplay element. Something Gale could fix you up, for example, if you find him a few items he needs for it. So he would make a teleport item you would carry with you and use, maybe with limited number of uses over some period or uses needing to be recharged. Additionally limited to be usable only from open world areas and locations.
I would also remove the fast travel option and tie it only to those waypoints. Maybe even have the ability to create them myself - also Gales invention - item.
Using the waypoints being tied to Gale, or a Wizard class, in general, is something that came to my mind as well, especially because Gale pops up from within it, which teaches the player these waypoints can even be used as such. It's unclear to me if utilizing these waypoints is a magical feat, or that it's something every class or character would be able to do. I'm guessing it isn't, because otherwise the Goblins from the Blighted Town or Goblin Camp outside of Shattered Sanctum, would probably have used the waypoint to travel near the Druid Grove in almost an instant.
So let's say it actually IS tied to a Wizard-like class and without such a class, you just can't use the Waypoints. I think, that in itself, is a reasonable restriction, without resorting to the need for items.
I wouldn't agree with the ideas about the inventory, backpacks and trading items.
That just creates even more inventory management. And imposes limits that dont make much sense considering how the world is built. The current design is made for ease of use and does not create such large breaks of immersion that it should be changed.
To do it properly the whole world approach to items and loot would need to be remade and thats just not going to happen.
At the very least, they should then have a reasonable explanation as to why they can magically send items to their traveling chest and carry around ridiculous amounts of gear to begin with. I would argue that they use waypoint-like magic to do such a thing and that for this, they need a wizard-like class in their party.
That being said, I would still prefer just not being able to take with you ridiculous amounts of items. Adding a backpack just for the sake of potions and small trinket-like items wouldn't be that hard to implement and increasing the rewards for completed quests would remove the need for hording items, only for sale. I think that would probably be the easiest and still immersive option.