I think you need to do the side quests to build up your character. Also, the side quests are a lot of fun and really contribute to immersion in the game world. It's all part of a fairly unified whole, although you may not have to do the side quests. I think you'll miss out by going that way.
