I don't think you can avoid the skill boost on reloading, though you could try getting both skills up to level 2 yourself before reloading the game (character level 7).
Not using a sword isn't going to hurt you any. Warriors tend to have lots of spare skill points, so a boost to one you are not going to use will hardly be noticed. If you try to boost mage skills with found or purchased books, or are selective in which summoning skills you learn, you should still have skill points to spare.
