If you leave the prison or the collar comes off they will attack you on sight..It is normal, it is part of the game.

You are probably leaving the prison or finishing the collar quest too soon and are underleveled and undergeared.