The camp seems to be a mixture of Dragon Age Origins camp which is only accessible from the world map, and Baldur’s Gate rest system which you can do anywhere except in towns.

It looks a bit odd as is, because it appears that you are supposed to setting up for the night wherever you press the long rest button. That might potentially be even more strange once you get camp followers who could seem like they appear from nowhere.

Perhaps what you’re really doing is going back to wherever your camp is on that map when you say you want a long rest, and then returning?
Or maybe that’s what happens once you start getting camp followers? Camp could actually be a place on the map (near where you start each one), and a long rest just fast travels you there and (perhaps optionally) back again?

It’s clear that the camp is going to be a significant part of the game, so it’s definitely here to stay. Between long and short rests, I’m sure it’s possible to have it make sense, possibly with a few tweaks from early access feedback.