Compression should have a much greater effect on file size than just reducing the resolution or frame rate, though you'll probably want to do both.
Fraps most likely writes uncompressed video by default, in order to minimize the impact it has on game performance. It may have an option to compress the video, though (either during capture or after). Alternately, you can use
VirtualDub to edit/trim video as well as compress it, or
ZS4 (have not tried the latter).