I ended up figuring this out.
After installing NoMachine with:
installer -pkg /Volumes/NoMachine/NoMachine.pkg -target /
You then need to manually run the install scripts to configure PAM:
/Applications/NoMachine.app/Contents/Frameworks/scripts/setup/nxnode –install
/Applications/NoMachine.app/Contents/Frameworks/scripts/setup/nxclient –install
/Applications/NoMachine.app/Contents/Frameworks/scripts/setup/nxplayer –install
/Applications/NoMachine.app/Contents/Frameworks/scripts/setup/nxserver –install
If this is done correctly, /etc/pam.d/nx should exist, and login will work correctly.