While I've always ended up fighting him, he's never called me an "intruder". Usually there is a bunch of dialog that leads into us eventually fighting. Did you already fight in the arena and get your collar removed? Have you gone around killing every single magister and silent monk? Those are two things off the top of my head that might make his attitude towards you hostile.
