I struggled through the debuff in one of my playthroughs. It was very doable actually. I think we are afraid of the words "Can't" and "Reduced" a lot, while in fact those would go mostly unnoticed.
Also, there should be a way to remove it. I think there is a spell in DND that can remove curses. This can be a good way to deal with the drawback, if they implement it.
