When you say skills, you mean Scoundrel, Marksman, Man-At-Arms, and only those types of Abilities?

It can definitely work, I'm just not sure how much fun it will be.

It's going to be a pain, at low level, when you need those combat abilities more. And at high level I feel like it's going to get tedious without any abilities of your own to spice things up.