I often record gameplay clips from video games I'm playing with FRAPS. Usually in 1920x1080 resolution depending on the game and the FPS I get with FRAPS running. For transcoding the raw footage to something bearable to upload I usually use only VirtualDub.
Lately I've experimented with settings, and I think I found some good ones. Good video and sound quality with small file size were my goals. I think roughly 100 mb for a 5 min. clip and about 300mb for the full 10 min. in full 720p HD quality is awesome.
I'm still experimenting with sound compression but I think with 192 kb/s AC-3 ACM I have a good start.
Please bear in mind that the higher your resolution is the bigger your file size will be and vice versa. Those 100 mb respectively 300 mb are based on my 1920x1080 resolution!
This guide is intended for 16:10 and 16:9 resolutions. If you're playing on a 4:3 resolution, do not resize down to 1280*720p without maintaining aspect ratio, because the end-result will otherwise look over-stretched/-squeezed.
My settings for VirtualDub are in the picture below. If you want a very detailed explanation what each option does, check this out!
By the way, you have to have the H.264 codec installed to get it working. Download and install the latest VFW-version build here. PAY attention which version you're downloading, x86 (=32bit) or x86_64 (=64bit).
A warning about the Option "Multithreading" -> "Threads" don't set it higher than [# of your CPU Cores * 1.5] because then your whole system will start chugging. For example a Core 2 Duo would be 2 Cores * 1.5 = 3, a Q9550 would be 4 Cores * 1.5 = 6 and an AMD Phenom II X3 would be 3 Cores * 1.5 = 4.5 4.
This is by no means a guarantee that it will or will not work on your system. Each system, its components and its software are unique. You'll probably have to experiment a bit on your own. But those mentioned numbers are a good start.
It also might be necessary for you to adjust the audio volume to your liking. I have -10 dB, because the sound I record is terribad. My onboard sound sucks, period. Some people also experience asynchronous sound, here's how you can fix it.
Edit 30.05.2012 - FFDSHOW Resize Filter:
There's an external filter, a better one in my opinion, available to use with VirtualDub. It's calle ffdshow and it provides amongst other things resize, watermark and sharpen functions. Here's the guide.
On a side note: When you have uploaded your clip on YouTube and YouTube processed it the quality then will be lowered again. There's nothing you or I can do about that, it's up to YouTube.
The whole tutorial is also available on YouTube as a video:
A few links to my own HD clips:
Addendum 16.September 2009: Enabling GPU acceleration
Since version 1.9.4 it is possible to use the GPUs processing power (if your GPU supports it, which most modern GPUs do) as part of whole the transcoding process. This way you can speed up the transcoding process significantly (depending on your compression settings!).
On my computer this cut my transcoding time from an average 1h for a 10 min. clip down to some 40-ish minutes. Under "Options" -> "Preferences" enable "3D accel".
Addendum 23.10.2009: Crashes during encoding
There's currently a problem causing VirtualDub 1.9.4/5/6/7/8/9 to crash under Windows 7 while encoding sometimes. The culprit is 3D accel! Turn it off under "Preferences" -> "3D accel" and enjoy VirtualDub once again!
Addendum 16.05.2010: Option "3d accel" slowing down encoding
I've been getting mixed messages. Apparently using "3d accel" causes it to render slower than usual on some systems. I can confirm that. With 3d accel turned on, a 10min video takes about 1 hour 10-20 minutes, while turned off it takes about 1 hour max. You'll have to experiment on your system!
And if you are running on a multicore cpu, don't forget to enable threading via "Option" -> "Preferences" -> set "Threading" to 1!
Added link to ffdshow filter guide.