From my experience, you can go along with either side if you don't have Lae'zel in the party. With Lae'zel however, you have to fignt and kill her if you decide to side with the emperor.
Since "angry alien" (quote Karlach) died before you even got to creche Ylek, you can avoid that place altogether, and there will be no encounters with hostile Githyanky patrols in act 2 and act 3, nor with Voss.
Not sure what will happen if you decide to confront Vlaakith at the creche without Lae'zel though. Will Voss still come to bring you the psionic device ? I've not gone that route.