I think that the most important problem is, that exp is often the ONLY significant reward in an RPG.
Money: In most RPGs (including the larian games I played DD, D2 and D:OS) you have tons of money very soon. After some point, getting a bit more money from a quest does not make any difference.
items: Usually you can get the best items from bosses, as quest rewards or from certain locations (like the trapped and hidden treasure chamber of a dungeon. Once you have some basic equipment, killing villagers or "trash mobs" will not give you something useful.
EXP are always useful unless you have reached the highest possible level (assuming you use a level system) or you have maxed out all skills (if you can spend the points to raise your char directly). EXP are also very addictive because you get an reward every time you perform an action (kill something, finish a quest, . . .). In some games the game became more boring when I reached the max level because I did not get a reward for every action (arcanum, fallout 3, even mass effect 1)
There was an interesting mechanic in Divinity 2: Mind Reading. You get an exp penalty for reading minds, but it could give you better skills, new quests or new ways to solve old quests, access to some places or just some very funny lines. This gave me an idea:
suggestion:
When you accept a quest, you can select if you want just the basic quest or if you select extra goals. if you select an extra goal, you get an exp penalty. If you have success in your extra goal, you get a bigger reward (more exp, a new skill, a powerful item or something that outwights the penalty). But if you fail the extra goal you get the normal reward and no compensation for the exp penalty.
example:
quest giver says: "I want you to get me the diamond from the museum."
you can answer:
1.) "Yes, I will do so." (You get the normal reward and no penalty.)
2.) "Yes, I will get the diamond and I will not kill anyone [1000 exp penalty]." (When you say this you will need 1000 exp more to reach the next level. If you do not kill anybody and bring back the diamond you will get the normal reward plus a bonus like 2000exp or a powerful item. But if you kill somebody you get the normal reward and you have lost 1000exp. This way you could run to the diamond, grab it and run away while the guards try to attack you and you still get the bonus reward.)
3.) "Yes, I will get the diamond, I will not kill anyone, I will not be detected and I will not trigger any alarm [2000 exp penalty]." (When you succeed, you get 4000 exp, a powerful item and a skill bonus, but if you are detected you get the normal reward and lose 2000 exp.)
4.) "No, I am not a thief" (no quest, no reward)
This way the player can select the difficulty and reward he likes and you do not have to remove the exp for combat. While this sounds nice in some situations, there may be some quests where it makes no sense at all. The biggest question: Why should the quest giver give a better reward if the player chooses to make his (the players) life harder?
Prof. Dr. Dr. Mad S. Tist 
