So I went wandering around and lost interest because I had no real direction anymore and I kept revisiting the inn and talking to people but nothing is happening.
Kingmaker has bunch of moments where it is not clear what you are supposed or can do. Sometimes you need to explore, sometimes you need to sit on your ass, manage the kingdom and wait. In this case you need to finish the objective (find and defeat the bandit lord) - so just walk around, kill what you can, until you find him. If you pick it up again, I recommend playing with walkthrough. Most of its content is incoherent mess demanding metaknowledge. At the very least you will save yourself a lot of walking back and forth.
