I did not put any skill points into intelligence, playing a warrior and archer. Half way through the game my hero had +4 intelligence from equipment bonuses, which became +24 by the end of the game (intelligence or mana bonus were far down the list of preferences when I was choosing equipment).
Once you get to the next level, you can find a spell book which teaches Restoration. I taught this to both main characters, primarily to top off their hit points after a fight. It takes a couple seconds to cast, so if you don't want to leave yourself vulnerable during a fight, you would have to run a bit away from melee attackers. I didn't have enough mana to cast Restoration a lot, but it was handy, and did help conserve potions (I started the game before the patch was released, when BF merchant inventories did not reset each time you entered).
I'd say ignore intelligence unless you find yourself often running low on mana. If you don't find a couple spells to try out, lack of mana may not become a problem.
There are a lot more swords around than daggers, though I never really paid much attention to daggers. In terms of damage, the long dagger compares well with most one handed swords, and the royal dagger is not too far behind the best of them. You might not have as many opportunities to upgrade, but as long as they are available you should most likely be able to use daggers throughout the game.
The weapon penalties do have an effect, but your accuracy, speed and initiative are also influenced by your stats, weapon expertise skill level and other equipment (better armour usually has larger penalties, for example). You can also find equipment with lower penalties, none, or even bonuses (especially on jewelry) that can compensate the penalties on other equipment. I had one character using a 2 handed sword, and as long as I didn't neglect agility, survival and speed I had no problems with the higher penalties on such weapons.
That said, sticking to an agility based weapon would allow you to concentrate more on that stat. Daggers don't really trail too far behind one handed swords for damage, but if you try it and start running into situation where your hero isn't pulling their weight, you could try switching to a spear, or something. Spears are mostly agility based, so shouldn't have high strength requirements. They generally have penalties higher than a dagger, but less than a sword, though the best spears are not available until later in the game. I'm not sure how easy it would be to find a spear that beats your dagger at any given point in the game, but you might want to keep an eye out for decent spears.
I only put a couple skill points into weapon expertise for my warrior, so after you go up a few more levels you should easily be able to keep skill points in reserve so you can train in different weapon types if needed. For my archer I put one skill point into accuracy and damage, a few into Point Blank and eventually maxed Reload Time.
