Hypercam. Grab a MJPEG codec like Morgan or Pegasus, because that will be the only codec that will keep up at quality. Also, limit your area of capture to 720 x480 for 30FPS, or increase the resolution and drop the framerate.
Remember, the screen capture utility uses CPU and Harddrive bandwidth. If the app you are trying to capture already uses up these resources, there is no way you have enough left for the capture.
If that's the case, then you have to go to a hardware solution called a framegrabber. Prepare to spend $3000.