You have definitely traveled the Hollow Marshes a bit. The marshes are level 5 and 6 and will be rough for an undergeared level 4 group. You definitely sound undergeared. By the time you leave Fort Joy proper, none of your characters should be in rags and all should have at least level 3 weapons. You have definitely missed some early stuff. At the first way point you unlock in Fort Joy, there is a spot where you can teleport a character to and explore. No fights, but a barrel and chest to loot as well as find a unique sword that is one of the better weapons you will find in Fort Joy. If I recall properly, the sword is call "The Viper" and gives a +1 to Strength and +1 to Warfare, making it great for the Red Prince. The Viper is only level 3, so it will start being a bit underwhelming when you are ready to leave Fort Joy, but should serve you extremely well until then. FYI : If you keep an eye on the minimap, you will be able to tell which areas of the map your characters can get to, it is just a matter of figuring out how to get to some of them.

A good way to tell if you have explored the majority of the island of Fort Joy before leaving is by the collection of Crafting Bible books you have found. On the island of Fort Joy you can find Bibles : 1, 3, 4, 5, 8, 9, 10, 11, 12, 13 and 14. Number 14 will be loot in the ultimate fight on the island.

You can still complete the Teleporter Quest and I recommend doing so for a bit of extra XP. However, don't do it right away. Save it until your getting ready to leave the island of Fort Joy. The NPC (Gawin) that gave the quest is an Aerothurge vendor. You can loot him for Aerothurge spells. However, if you have the gold to purchase them it is easier to buy the skill books and the loot the gold. You can just combine the stolen gold with other gold you have and nobody will ever be able to tell you stole that gold from them, even if they catch you and search you, as long as you have joined the stolen gold to some other gold prior to the search.

Yes, we are talking about the same noxious bulbs. The heads on the sticks can be fun to talk to if you haven't. You can also get a bit of useful info from one of them. However, having opened the chest, it is too late if you haven't already. From there, continue keeping to the right side of things. This will take you in basically a U turn and into the second set of just two noxious bulbs. Past the two bulbs is Amadia's Sanctuary. With the teleporter gloves, you can simply teleport these two bulbs out of the way. With the teleport gloves there is a short cut to Amadia's Sanctuary just before the noxious bulbs with the chest. However, that is something you can look for that on a subsequent play through. By the way even after 4 play throughs I can still find things I haven't noticed before.

Actually Scholar and Mystic is great. There is one spot in chapter two (Driftwood) in which the combo will be invaluable. Don't get worried about not having the soldier trait.

Overall, it sounds like you are doing well. If your Spidey Sense went off on the chest in the noxious bulb ring, it should serve you well through the game. Teleporting the chest out of the ring is great adaptive thinking. You should do well in the game. As for equipment, here are my recommendations. For basic (white) gear try to keep it within two levels of your characters. Green and blue gear should be kept within 3 levels of your characters. Purple and unique (gold) can be as far as 4 to 5 levels below your character's level, but try to keep it a bit closer. One character focusing the Lucky Charm ability will allow you to find loot more frequently in the chests, barrels, etc. that you open up. Have one character, preferably a human due to the bonus +1 Bartering, focus on Bartering. Worst case, if the loot you find doesn't have the stats you want and is of no use to your group, at least with higher Bartering you can get more gold when selling it at a vendor.