I published the NX enterprise client via Citrix XenApp and connected to my test machine (again, h.264) and it’s very laggy.
If I understand correctly, the NoMachine client is running on the server and its output is encoded by Citrix XenApp and forwarded to the XenApp client at the remote location (through the NetScaler VPN). So (again if I understand correctly) NoMachine encoding is taking place between the XenApp server (where the NoMachine client is running) and the NoMachine Enterprise Desktop, with the NoMachine client displaying its output as if the XenApp backend was the real display. If this is the case, I’m afraid this looks more a Citrix performance problem than a NoMachine performance problem.
I read that this is not possible in your case, due to other constraints, but can you compare the performance connecting directly from the client side to the NoMachine Enterprise Desktop machine using the NoMachine client and the same NetScaler VPN link? This could be a good starting point.