ED25519 and ECDSA for NX protocol produce “Authentication Failed” error

[neatchee]

Due to recent further degradation in the security of RSA, I have been migrating to more modern cryptography wherever possible. To that end, I attempted to upgrade my NoMachine instances to utilize ED25519 keys for authentication (and ECDSA when that failed)

However, I am unable to connect to *any* of my NoMachine instances using NX protocol + non-RSA SSH key auth

This has been tested in multiple configurations, all using NoMachine 8.14.2 Free Server + NoMachine 8.14.2 Enterprise Client

Combinations tested:

Windows 11 as Server (Free) + Windows 11 as Client
Ubuntu 22.04 as Server (Free) + Windows 11 as Client
Windows 11 as Server (Free) + Ubuntu 22.04 as Client

All scenarios were tested using ED25519 *and* ECDSA keys in both modern and legacy (PEM) formats:

ssh-keygen -t ed25519 -f <output_path> -C “<comment>”
ssh-keygen -t ecdsa -b 521 -f <output_path> -C “<comment>”
ssh-keygen -t ed25519 -m PEM -f <output_path> -C “<comment>”
ssh-keygen -t ecdsa -b 521 -m PEM -f <output_path> -C “<comment>”

Additionally, key generation was attempted using multiple version of OpenSSH including, 7.5p1, 8.9p1 and 9.5p1

In all cases, the client reports “Authentication failed” with no further information shown in the server or error logs

Does NX protocol only support RSA still?

[Britgirl]

Hi,

RSA, ED25519 and ECDSA are all supported, for both NX and SSH connections. The only exception is with web-based sessions where only RSA keytype is supported (web-based sessions are not available in the free version anyway). Are you importing the keys in the NXS files? Does the problem happen when the keys are not imported?

[neatchee]

I make a point not to embed my SSH keys in the connection file whenever possible. I have just tried importing the key anyway and received the same “Authentication Failed” result I’ve generated a fresh ED25519 key using the first commands listed above, added the pubkey to the “authorized.crt” file in %USERPROFILE%/.nx on the host Windows machine, and tried again. Same failure. I’ve removed the pubkey from authorized.crt and attached both the public and private key here. This keypair is not used anywhere else, and is only for the purpose of this troubleshooting process. The key password is “testme123.

The only non-default configuration options in server.cfg are: NXTCPPort NXUDPPort EnablePasswordDB 0 AcceptedAuthenticationMethods NX-private-key Is there a way for me to get additional logging? Where can I see the reason for the authorization failure?

Attachments:

1. testkeyfornomachine.zip

[neatchee]

Ugh, apologies for the terrible formatting. Apparently your forum software removes all linebreaks from text input when a filetype is rejected for upload.

[Britgirl]

You wrote:

added the pubkey to the “authorized.crt” file in %USERPROFILE%/.nx on the host Windows machine

it should be .nx/config. Maybe this is the problem? Can you follow the instructions for How to set up key based authentication with NX protocol in this article https://kb.nomachine.com/AR02L00785?

Is it possible there is some trivial error like a typo in the file name, or even wrong permission?

You can also send us the logs from both player and server machines. You can extract them using the instructions here: https://kb.nomachine.com/DT07S00243. Send them to forum[at]nomachine[dot]com. Please use the title of this topic as the subject of your email. Thanks!

[neatchee]

Sorry, I meant that it was in .nx/config; just didn’t type the entire thing out haha.

While sanitizing logs before sending, I noted that the client logs showed that it was failing to load the ED25519 certificate with a decoder error.

Further investigation down this track determined that the issue is limited specifically to the NoMachine Enterprise Client and does not occur when using the player that is bundled with the full NoMachine Server installation

That is a sufficient solution for me at this time – it will at least allow me to sunset my RSA key – though I’d rather not have to install the full version on machines that do not need to be accessed remotely.

Can you please try to reproduce this issue using the NoMachine Enterprise Client specifically on Windows?

[Steve92]

Hello,

A few weeks ago, I had some problems too with ED25519 algorithm to generate keys and I thought it was not supported (hence my question https://forum.nomachine.com/topic/ed25519-algorithm-for-ssh-nx-keys ).

I’ve just done a test in full Linux environment, all is OK. (following https://kb.nomachine.com/AR02L00785)

On “!M Client” 8.14 side:

$ ssh-keygen -t ed25519               (-b is useless since fixed length key)

I kept default key names and added a passphrase.

The server is “!M Enterprise Cloud Server” 8.14.

FYI, ssh version :

$ ssh -V

OpenSSH_9.2p1
OpenSSL 3.0.14   4/6/2024

O/S:

Debian 12 Bookworm
Linux antix1 6.1.105
(super light Linux, perfect for testing !M in live VMs)

Good luck ! 🙂

NB: RSA 4096-bit key is still strong enough (even 3072-bit for common usage) !

Regards,

Steve.

[Steve92]

Hi,

I did a quick successful test closer to your need. 🙂

# !M Client 8.14.2 installed on:
Microsoft Windows 11 Enterprise Evaluation (expired from a few months)
Version 10.0.22621 Build 22621
VM under “VMWare Player 17” on “Debian 11”: 4 CPU 8 Go RAM

> ssh -V:
OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3

> ssh-keygen -t ed25519
(default path+filenames+a passphrase entered)

Public key
C:\Users\User\.ssh\id_ed25519.pub
transferred to Linux remote server and added to
/home/my-user/.nx/config/authorized.crt

# Remote server:
“!M Enterprise Desktop 8.14.2” (evaluation)
VPS “Debian 12”, 1 vCPU, 2 Go !
LXDE

# In “!M Client” on W11 VM
For “My Enterprise Desktop” created connection
Edit/Configuration
x Use key-based auth. with a key you provide
[ Modify ]
C:\Users\User\.ssh\id_ed25519

That works like a charm !

Good luck and go ahead ! 😉

Steve.

 

 

[Author]

Posts