The gloves not working is weird. As if you have disadvantage for some other reason.
Do you by chance wear boots that are medium or heavy armour and not proficient with ?