Just to update this – we discussed this further offline and the developers found the issue is because PAM had been configured to enforce membership of the wheel group to allow su usage. /etc/pam.d/su contained:
auth required pam_wheel.so use_uid
The information I received was:
Our user “nx” is not in the wheel group while this PAM configuration requires
that nx is in that group to allow authentication working properly.We now ship our own /etc/pam.d/nx module which can separate PAM
configuration that is used by NoMachine from the one used by SU service.So, possible solutions are:
1) Comment out the problematic line in /etc/pam.d/su.
2) If you don’t want to change the su configuration, please replace the
/etc/pam.d/nx module with the file attached to this mail.It’s a clean copy of your “su” configuration file just with the problematic line removed.
In this way you can use a custom policy only for NoMachine and don’t need to change
your “su” configuration. Feel free to edit the nx file according to your needs.
I wanted this restriction to remain so I created the specific nx module suggested, which contained:
auth sufficient pam_rootok.so
auth include system-auth
account sufficient pam_succeed_if.so uid = 0 use_uid quiet
account include system-auth
password include system-auth
session include system-auth
session optional pam_xauth.so
Can’t fault the developers who responded. Excellent support to help troubleshoot this issue.