So... if you already have disadvantage you can't use the special attacks? Otherwise every attack in dim light or whatever that gives a disadvantage will be a free cleave or bash.
I would prefer a penalty and free use over an arbitrary limitation of one per rest.
