I think the game has been rather clear. The dialogues with Halsin and Omeluum (?) reveal that there is more than physical presence in your brain that binds it to you. It is bound by magic and can not be mechanically removed. The mindflayer does mention that even death can not separate you from it.