Bosses are actually very memorable and fun if done right, but you have to take some things in consideration:
1. They are a final exam of sorts. They should challenge you on the things you have learned up to now. I remember all the bosses around Cyseal and some after that, but they weren't all that challenging, maybe except the blob monster at the Lighthouse. I remember dying a few times only to him (and to the big bad in the church, but he was cheap, I will get to him later) legitimately and that was great. They should be challenging in general.
OK, so, these are actually the last examples I left to think up; and me and a friend of mine have been banging our heads against a wall for half an hour trying to think up any truly challenging bosses in an RPG. Turns out it's hard to find examples. We went through Baldur's Gate 2, DA:O, the FF games, NWN, MotB, Dark Souls and many other RPGs and we had a counter to any bosses we thought up. The main problem is that all bosses boil down to the simple mechanics it uses and working around that. You might say "But Dark Souls was really challenging!" and I'd say that it wasn't. It was mostly trial and error and keeping concentration for half an hour when you found out what kind of actions you should repeat. Kangaxx in Baldur's Gate 2 came up often and I don't know whether he belongs here or in the cheap section. His imprisonment was the problem and you didn't have access to freedom spells at that time. He also had access to the same overpowered spells you had, but that was fine. That leads me to this conclusion - the hardest bosses in games are other players. It doesn't necessarily mean PvP, dungeon master mode is an example. That is the only way for truly reactive bosses who can challenge you without feeling any artificiality and triviality. You can create the illusion of challenge like many RPGs and, without a truly spectacular AI rivaling human intelligence, that is your only option. I'm sorry that I can't provide anything more concrete than this, but that is how it is. That's not necessarily a bad thing mind you, ALL games against an AI are like this and there is nothing wrong with that. There are a plethora of good examples of this type of challenge in Baldur's Gate 2 alone.
2. They aren't random mooks with 10x the hp bar. They should be set up properly in the lore, background and/or foreground information. Having them randomly appear out of nowhere and being important just because they have a name is really contrived and jarring. Working them into the narrative is a difficult, but rewarding task. A great example are the bosses in the first Deus Ex, you knew *exactly* who they are and why they want to kill you. Not going to spoil them.
3. They shouldn't cheat. I.e. They shouldn't be able to break the game's rules or be completely different than anything else to the point of thinking whether it's a programming error *cough* Adra Dragon in PoE *cough*. They may have unique abilities and spells, but those should be within the game's context. They shouldn't have instant death spells if nothing else in the game has them. Examples include bosses who can cast more spells per round than is allowed, bosses who have unlimited movement points etc.
4. They shouldn't be cheap. This is where I talk about the big bad beneath the church in Cyseal. He had an aoe meteor strike kind of spell that he randomly cast at the start of the combat which wiped the whole party. There is no way to counter this if he casts it in the first round because he ALWAYS had first initiative. If he didn't cast it then you simply scattered your party and it wasn't that big of a problem. They also shouldn't have a bajillion hp which takes 1 hour to chip away. JRPG bosses are notorious for this. The Weapons in FF7 for example.
5. They should be rewarding. This is the most nebulous and hard to define point. It partains to the context of the boss and is a combination of the other 4. You get a sense of accomplishment when you overcome them. Either in a lore sense or purely mechanical or even a great unique item. Dragons in Baldur's Gate 2 are a good example. Though any challenging or important boss can fit.
That's basically it. It isn't easy, but nothing good in this life is. I forgot to talk about adds, but that's mainly because I don't have a problem with them. As long as they aren't annoying and gratuitous it adds a layer of complexity which is a welcome addition.
