As far as I know you still need use the hide action (stealth check) vs a perception check even if you are invisible. Until you do so, you are still detectable by sound, for example. Of course, the attacks still get disadvantage.
It is still a bit tricky and in general 5e does allow a lot of freedom based on the circumstances (I would personally take into account distance, noises, the movement of the invisible character to determine how easy it is to stealth). But there are ways to detect an invisible character.
None of that has anything to do with the fact that interacting with objects should never break invisibility. You're talking about two entirely different things.
