Can you please give me description of graphical glitches you've seen?
660M is a decent card but with low memory. What OS X version are you using? How much RAM do you have?
The video at startup should not run choppy - it should have 2 problematic points where it might freeze a bit

But video play should be smooth.
I test play this thing in test lab on HD3000 - it's a very low end config and it plays decently. Resolution matters as well as shadows and of course high-end PP. Your iMac is 2560 x 1440 and that's extremely big for 660M - your resolution should vary around 1920, 1680 or 1400. You should not use Fake fullscreen as well.
That's about it. No idea on 5 minute issues, might be video memory overflow issues.