Right now, difficult terrain reduces fly distance from the fly spell, and distance you can teleport with misty step. This is a clear bug, ground terrain should not effect flight or teleport distance because you are not traveling over it. Likely a coding issue.