Yes, you can only pick what's white/yellowish then in the library (so not the chest for example). They 'catching rats' book seems broken in that it remains red even with permission though.
If it's your first offense they will simply repremande you (with an attitude penalty), and if you stole items you would need to return them. Too many times though and yes, guards will be called.
If I try to open a container (unlocked) that has a red cursor, in front of other NPCs, but dont actually TAKE anything, will it cause any kind of negative reaction with NPCs?
Yes.
As for why pretty much everything is red in people's houses, well, it's their stuff. This isn't a RPG where the houseowners will simply stand by while you loot them (while it was funny in Baldur's Gate, not working here).