Because BG3 is based on D&D 5e it might be useful to read up on that. Though bear in mind the implementation in BG3 isn’t exactly as per 5e.
Roughly, though, when you attack with a weapon you (1) roll to determine if you hit, and (2) if you hit, roll again to determine damage.
For your to-hit roll, you roll a 20 sided die (d20) then add/subtract modifiers. For many melee weapons, that will be determined by strength and whether you are proficient in the weapon. Weapons that are tagged as “Finesse” use dexterity instead of strength. Ranged weapons also use dexterity. If you open your inventory and look just under your weapon slots you should see some numbers representing what number should be added to your d20 for melee or ranged attacks and hover over them to see how these numbers are made up. You will hit if the total number after modifiers is equal to or greater than your enemy’s armour class (AC).
There are all sorts of other factors that can change your chance to hit or not, but the basics are to make sure you’re wielding a weapon you’re proficient in and that your strength (or dexterity, if you’re using finesse or ranged weapons) is decently high. Usually you want one or the other to be 16 on character creation if your character is primarily going to be using weapon attacks.
Damage will be determined by the specific weapon, rolling one or more dice of a variety of sizes and again (usually) adding strength or dexterity modifiers. With respect to your specific examples:
1d10(1d8)+5: I assume this is from a weapon with the “Versatile” property. For these weapons, the first number is the damage if wielded two-handed, the second of one-handed (ie with a shield or other weapon in the other hand). If you are wielding such a weapon and have nothing in your off hand, you roll a d10 (10 sided die) then add 5. If you have a shield, for example, you roll a d8 and add 5.
2d6+3: Roll two d6s and add the numbers together, then add 3.
Again, there are other factors that can affect damage, but those are the basics.
If you want to take a closer look at what numbers are being rolled to help understand why you’re not hitting or what your damage is, you can expand the combat log at the right of the screen and try hovering over various elements.
