I have 5 inert stones. It seems that the other portals didn't open up until exiting the inner chamber via its portal rather than the rift travel. So now I have a total of 5 portals open. Thanks!