Ok i figured it out.

Its only getting bugged from valor bard and only after lvl 5.
If i used 1d8 to improve my attack it was fine and working as intended.
But if i used it to improve my damage or improve my savings or my armor its bugged. I could fix it if i was long resting.
But if i try to inspire the same character agan the it became broken forever.
Also after lvl 5 the bonus for skill improve is unlimited and its not removing the inspired effect.