I've played a lot and noticed that things are definitely happening if you long rest, there are consequences but it seems like a consequence only happens if you long rest after making some progress - so if you decide to long rest a bunch in a row then it's seemingly not doing anything but if you make a bit of progress and then rest, you get some camp progress and it also affects the world in various ways. Doing a playthrough avoiding resting and one that rests after basically any event has shown several really cool differences.
Free swap is definitely something I see as an issue, especially if later you get more magic items - you could end up feeling the need to swap to everything defensive then back every turn. I think it would be tedious to only get to swap one item for a bonus action so maybe if you dig into your bag it will cost 1 bonus action but give you unlimited item movements, so for one bonus action you can throw all your barrels on the floor or stored boxes to make a quick ladder and swap all your armour,etc.
That way the cost is small enough that we can get creative but not so free that some players feel compelled to min-max every single turn.
It's interesting how you noticed the time factoring into the world in different ways. But I mean, besides that, maybe there should be the occasional wandering monsters that could attack the camp - that is classic in D&D - who's gonna keep watch? lol Most of the time, I felt like long resting was pretty easy and with no real consequences.
As for the swap, I think that is a good suggestion, 1 bonus action for all item movements.
