Does it occur even if there is no activity in the session (like, for example, a video running or animations)?
What is your Mac OS X version and on what hardware?
One option you could try is switching to the MJPEG display encoding, in Server preferences, Performance tab, accessible through the tray icon.