It goes away at some point, though I can't tell you when. Usually the description of the buff will tell you and most likely it's a long rest thing, it usually is.