The difficulty level only effects combat and determines how many skill points you start with, if any.