I also believe that putting a limit on the vote amount wouldn't help.
Someone could vote <img src="/ubbthreads/images/graemlins/riftrunner.gif" alt="" /> down 50 times in one hour. And then you would be stuck. Becouse you would have to wait one hour to start voting it up again. While at the same time, in that same hour, that other person can start voting <img src="/ubbthreads/images/graemlins/riftrunner.gif" alt="" /> down again.
<img src="/ubbthreads/images/graemlins/disagree.gif" alt="" /> It could make things worse instead of better. <img src="/ubbthreads/images/graemlins/sad.gif" alt="" />