It sounds like your gear isn't what it needs to be. Also, given your description, you found everything you could in Fort Joy, but there is likely quite a bit you haven't done yet. It takes everyone a few play throughs to find most things. That being said, if your difficulty setting is classic, then you should be fine, yet.
If you haven't done anything with the other origin characters yet, things are easily salvageable. The biggest issue will be getting to Amadia's Sanctuary. From the boat, hug the coast leaving Fort Joy behind. The first bit will be a beach area. Once you get off the beach, you will go through some noxious bulbs. In the bulbs are heads on pikes that will talk/mumble as you go by. Ignore them and don't touch anything. Keep to the right side of things, this should allow you to sneak past the voidwoken encounter where the oil barrels are. Past that, there are two bulbs you need to get through. If you don't have the teleport spell to get them out of the way, you will have to use Armor of Frost and then just run through hoping you have enough magic armor to make it through. The goal will be to either get a character with the Soldier trait or your character with the highest persuasion past the bulbs. Past the bulbs is a bell to ring, which will get you a lizard pointing a crossbow right at you. You will have to convince her, you mean her no harm. A Soldier character has a great dialog option to do this, otherwise a couple points in persuasion should be enough. Once convinced you are no threat, the lizard will lower a rope for you to climb up. This next part is key, once you get to the top of the bridge, you are on the clock. There is no rush, you have roughly 15-20 minutes. Go into the sanctuary and you should find the kid (Han) you took the boat with to get out of Fort Joy. He will be with another lizard. The lizard will will ask you about Gareth. The dialog is short and shouldn't cost you much time. After the dialog is over, this lizard will note Han is injured. Hint. Hint. Wink. Wink. Heal Han and he will tell you others need to be healed as well. The others are the ones with the clock ticking. If you take more than 30 minutes to get to them, they will die and you will miss out on a nice chunk of free XP. After healing the injured, run around the sanctuary and talk to everyone. This should also unlock the quick travel. You can then dismiss your crew. Quick travel back to the fort and pick up the origin characters you haven't used yet. Pick them up and make them Shadow Blades so they have Thieving abilities. Using that thieving, rob the vendors in Fort Joy. This should get you some gold to upgrade your gear and see if there are any skills you want to add to your group. With that done, you can quick travel back to the sanctuary, dismiss your Shadow Blades and pick up your crew again. You will have to run around the sanctuary to find them, but it shouldn't take long. Once they have the better gear and any extra skills, you should be in a much better spot to start wandering the marshes looking for appropriate level encounters.
FYI : If you have decent gear and solid builds, you should be able to handle the crocs for the teleport quest at level 3. The problem is that you are starting for basically having nothing, so at levels 1 and 2 in Fort Joy you should only pick fights where you have a good numbers advantage. Also, on your first play through, Pet Pal talent is extremely recommend for your main character and have your main character put a couple points into persuasion. This should help find and complete a few extra quests. The traits I would recommend for your main character on the first play through are Soldier and Scholar. These are the two most useful traits.
I hope this helps. Best of luck to you.
