As a "Pact of the Chain" Warlock, I keep trying to use my familiar in creative ways to interact with environmental objects (burrows, buttons, etc.), only to remember that familiars can't interact with ANY object.
This takes away almost all of the utility of the available familiars outside of scouting ahead using the Imp or Crow.
I suggest that familiars should be allowed to interact with doors, buttons, etc. I do however understand that familiars should not be able to pickup items, open inventories, etc.