He also says that most of them didn't even grant him a temporary bliss.
Exactly - they were victims, not lover.
I agree that they were victims, but didn't you get the sense that Astarion was saddened by something more? A sense of what could have been, maybe?
I got the feeling--and this is just my feeling--that Astarion had a connection with him. Not love, perhaps, because there wasn't enough time to build love, but a moment where he saw the possibility. But his curse forced him to betray his own emotional needs and remove any chance he could have had of exploring something more.
Granted, in time, he became numb to the sexual encounters. Which is exactly what makes Sebastian stand out.