You're correct about those being the highest level skills for those two groups.
However, it's worth it to get Marksman 5 in order to unlock the Quickdraw talent; Elemental Ranger is kind of meh, but you can get that too.
The only benefit for getting Scoundrel 5 is access to the Headstrong talent, which is pretty strong but not quite on the level of Quickdraw. That one's more of a "worth it sometimes" deal.
