I don't think you lose too much from not investing heavily in skill trees tbh
the damage difference is 3-4 pts per investment from what I've seen, maybe it makes a bigger difference at higher levels with bigger stat investments, but the utility gain heavily outweighs any potential damage loss
maybe invest 2-3 pts in each of those and focus on one as your primary stat, as skillbooks (seem) to cap at 5, and that's for source skills only, with the next lowest being 3pts, but that's only if you really want those skills and can't find the appropriate gear to boost you from 2-3 or higher.
Currently rocking hydro/aero/summoner and it's working fantastically, about to drop 1-2 points into necro for another skill and a bit more utility (& debuffs) but I'm primarily pumping summoner bc it benefits the most
