ED25519 and ECDSA for NX protocol produce “Authentication Failed” error

[neatchee]

Due to recent further degradation in the security of RSA, I have been migrating to more modern cryptography wherever possible. To that end, I attempted to upgrade my NoMachine instances to utilize ED25519 keys for authentication (and ECDSA when that failed)

However, I am unable to connect to *any* of my NoMachine instances using NX protocol + non-RSA SSH key auth

This has been tested in multiple configurations, all using NoMachine 8.14.2 Free Server + NoMachine 8.14.2 Enterprise Client

Combinations tested:

Windows 11 as Server (Free) + Windows 11 as Client
Ubuntu 22.04 as Server (Free) + Windows 11 as Client
Windows 11 as Server (Free) + Ubuntu 22.04 as Client

All scenarios were tested using ED25519 *and* ECDSA keys in both modern and legacy (PEM) formats:

ssh-keygen -t ed25519 -f <output_path> -C “<comment>”
ssh-keygen -t ecdsa -b 521 -f <output_path> -C “<comment>”
ssh-keygen -t ed25519 -m PEM -f <output_path> -C “<comment>”
ssh-keygen -t ecdsa -b 521 -m PEM -f <output_path> -C “<comment>”

Additionally, key generation was attempted using multiple version of OpenSSH including, 7.5p1, 8.9p1 and 9.5p1

In all cases, the client reports “Authentication failed” with no further information shown in the server or error logs

Does NX protocol only support RSA still?

[Britgirl]

Hi,

RSA, ED25519 and ECDSA are all supported, for both NX and SSH connections. The only exception is with web-based sessions where only RSA keytype is supported (web-based sessions are not available in the free version anyway). Are you importing the keys in the NXS files? Does the problem happen when the keys are not imported?

[neatchee]

I make a point not to embed my SSH keys in the connection file whenever possible. I have just tried importing the key anyway and received the same “Authentication Failed” result I’ve generated a fresh ED25519 key using the first commands listed above, added the pubkey to the “authorized.crt” file in %USERPROFILE%/.nx on the host Windows machine, and tried again. Same failure. I’ve removed the pubkey from authorized.crt and attached both the public and private key here. This keypair is not used anywhere else, and is only for the purpose of this troubleshooting process. The key password is “testme123.

The only non-default configuration options in server.cfg are: NXTCPPort NXUDPPort EnablePasswordDB 0 AcceptedAuthenticationMethods NX-private-key Is there a way for me to get additional logging? Where can I see the reason for the authorization failure?

Attachments:

1. testkeyfornomachine.zip

[neatchee]

Ugh, apologies for the terrible formatting. Apparently your forum software removes all linebreaks from text input when a filetype is rejected for upload.

[Author]

Posts