Seems reasonable to me. You broke their rules, by visiting area under invisibility. They acting like a real person could. Imagine that you are in a hostile, dangerous territory next to an army of evil forces. How would you react to someone exploring your camp while hiding under a spell?
Don't see bugs here.
