Spending 5 skill points to get 1 is a TERRIBLE investment.
It's not about how many extra levels you get. For me it's about being able to get to new levels faster, which means less time spent wandering around looking for bad guys to kill. Plus, there are also three different wisdom books you can find to boost the stat, if I remember correctly.
