Today my problem is back 🙁
Seems like “Use acceleration for display processing” doesn’t really matter: The problem was my desktop resolution.
In 1080p hardware encoding is working, but not in any higher resolution. (I have to set the resolution directly in windows, setting it in the client does not help.)
So this is probably a limitation of VCN or the GPU itself.
Unfortunately, I can’t find any information online on what resolutions are supported for H264 encoding on my APU.
Also rendering doesn’t “feel” any faster when using HW encoding compared to SW encoding, so I don’t really care.
Consider this case closed 🙂