No need to "avoid magic" with an archer.
Some magic skills align perfectly with an archer.
Furthermore, all skills can be combined with all other skills - as i just said above the game is made for any kind of jack of all trades - ACTUALLY.
Additionally, you dont need to have any skill points in the Marksman skill to use a bow and those special arrows.
If a character has enough dexterity to wield a bow - that character can use it and the special arrows.
Archery - Marksman skill is not necessary for that at all.
It isn't a matter of ability points, it's a matter of attribute points. You'd need enough Dexterity for the bow, and enough Intelligence for the spells... and not much left over for Constitution, Speed, or Perception.
I guess it's worth pointing out that I'm the sort who switched the difficulty setting to "Hard" as soon as I noticed it existed. The Hard penalty is less health (from +15% to -25%, so about one third of Normal health), which makes Constitution and Speed more important, the latter due to the reduced viability of the Glass Cannon talent (-75% health is brutal). Actually, Initiative in general becomes more important too; if you strike first, you have better odds of survival.