All this is really to say that yes there should be some quality of life stuff included in games, because a lot of times quality of life just means "we've learned how to make better games now." But at the same time, that doesn't mean excising anything that would give games any challenge. It's not a choice between one or the other.
Yes, having unlimited fast travel doesn’t excise all challenge from the game or make it story mode. That is just people using dramatic language in their arguments against it.