Yeah, higher stats at character creation will affect balance. But it won't matter all that much. D&D 5e has a concept called bounded accuracy that limits the extent to which you can stack bonuses. Even level 20 character cannot have stats higher than 20 except for very specific situations (level 20 barbarian for instance).
So, having a 20 in your prime stat at level 1 will be helpful, and it will make the first few levels easier, but tough encounters will not be trivialized. Late game balance will be affected by the fact that people can spend their ASI's on feats rather than stats, but they will not be able to push their prime stats any higher in the long run than a player with point buy. This will just make the game a little bit easier. It won't be equivalent to activating god mode or anything of that sort.
The feats will make the characters stronger, but even then bounded accuracy means that they will still miss fairly often, and they will still get hit fairly often.
Just make it so that hosts can toggle off the ability to roll stats for multiplayer (or enforce ONE roll, or something) and let the decisions for the single player game be made by the player.
