You really must be doing something horribly wrong. Maybe it's something you overlooked, like everyone's wearing armor they're not proficient with or something?
I'm roughly at the same spot as you, and the game really hasn't been that difficult. If you played e.g. DOS1/DOS2 or PF:Kingmaker/WOTR, all those games were much, much rougher in the beginning.
