Aha! I found the cause. I think.
I disabled the option for "Automatically turn on game mode" in Chromecast settings and suddenly the picture became about a hundred times better. Now I'm not sure why this did this, as if I reenable it and the other settings I can't make the picture bad again.
Maybe it was just a classic case or turn it off and on again.
Glad you found the problem. Actually this might be a problem with your tv and not the Chromecast. You can go to your tv picture settings and do it there.
What Chromecast was doing is turning the game mode on in your tv. This reduces latency drastically because it turns off all the TVs image processing settings.
This feature is always recommended to be on because it can intact the latency in 20 to 100ms depends on the tv model.
But if this is causing problems for you and you can't notice a drastically increase in latency. Keeps it off.
Just remember to turn it on when you change your tv.
Btw , if you have a windows computer this game feels amazing with m&k. De developers are saying controller is not very optimized yet and are recommending m&k for now.