Just one more note on this. My friend also found that doing the following got rid of the slow down on his Ubuntu machine:
FYI – I found a “Power Manager” setting on my remote Ubuntu 20.04 PC that was causing the problem!
Xfce Power Manager -> Display Power management (disable button) and all is well.
I had turned off screen save stuff before, but that is in a completely different settings widget and never noticed that the power manager settings had something for the display