I also am struggling with this decision.
I understand the thought of not trying to lose story items, but it seems like what is considered a story item is incredibly vague?
Random keys, empty alchemy bags, equipped items - all this and more being dumped in your inventory just increases the amount of inventory management we have to do whenever we want to switch our party around

I hope this addition can at least become toggleable