ED25519 and ECDSA for NX protocol produce “Authentication Failed” error

[neatchee]

Due to recent further degradation in the security of RSA, I have been migrating to more modern cryptography wherever possible. To that end, I attempted to upgrade my NoMachine instances to utilize ED25519 keys for authentication (and ECDSA when that failed)

However, I am unable to connect to *any* of my NoMachine instances using NX protocol + non-RSA SSH key auth

This has been tested in multiple configurations, all using NoMachine 8.14.2 Free Server + NoMachine 8.14.2 Enterprise Client

Combinations tested:

Windows 11 as Server (Free) + Windows 11 as Client
Ubuntu 22.04 as Server (Free) + Windows 11 as Client
Windows 11 as Server (Free) + Ubuntu 22.04 as Client

All scenarios were tested using ED25519 *and* ECDSA keys in both modern and legacy (PEM) formats:

ssh-keygen -t ed25519 -f <output_path> -C “<comment>”
ssh-keygen -t ecdsa -b 521 -f <output_path> -C “<comment>”
ssh-keygen -t ed25519 -m PEM -f <output_path> -C “<comment>”
ssh-keygen -t ecdsa -b 521 -m PEM -f <output_path> -C “<comment>”

Additionally, key generation was attempted using multiple version of OpenSSH including, 7.5p1, 8.9p1 and 9.5p1

In all cases, the client reports “Authentication failed” with no further information shown in the server or error logs

Does NX protocol only support RSA still?

[Britgirl]

Hi,

RSA, ED25519 and ECDSA are all supported, for both NX and SSH connections. The only exception is with web-based sessions where only RSA keytype is supported (web-based sessions are not available in the free version anyway). Are you importing the keys in the NXS files? Does the problem happen when the keys are not imported?

[neatchee]

I make a point not to embed my SSH keys in the connection file whenever possible. I have just tried importing the key anyway and received the same “Authentication Failed” result I’ve generated a fresh ED25519 key using the first commands listed above, added the pubkey to the “authorized.crt” file in %USERPROFILE%/.nx on the host Windows machine, and tried again. Same failure. I’ve removed the pubkey from authorized.crt and attached both the public and private key here. This keypair is not used anywhere else, and is only for the purpose of this troubleshooting process. The key password is “testme123.

The only non-default configuration options in server.cfg are: NXTCPPort NXUDPPort EnablePasswordDB 0 AcceptedAuthenticationMethods NX-private-key Is there a way for me to get additional logging? Where can I see the reason for the authorization failure?

Attachments:

1. testkeyfornomachine.zip

[neatchee]

Ugh, apologies for the terrible formatting. Apparently your forum software removes all linebreaks from text input when a filetype is rejected for upload.

[Britgirl]

You wrote:

added the pubkey to the “authorized.crt” file in %USERPROFILE%/.nx on the host Windows machine

it should be .nx/config. Maybe this is the problem? Can you follow the instructions for How to set up key based authentication with NX protocol in this article https://kb.nomachine.com/AR02L00785?

Is it possible there is some trivial error like a typo in the file name, or even wrong permission?

You can also send us the logs from both player and server machines. You can extract them using the instructions here: https://kb.nomachine.com/DT07S00243. Send them to forum[at]nomachine[dot]com. Please use the title of this topic as the subject of your email. Thanks!

[neatchee]

Sorry, I meant that it was in .nx/config; just didn’t type the entire thing out haha.

While sanitizing logs before sending, I noted that the client logs showed that it was failing to load the ED25519 certificate with a decoder error.

Further investigation down this track determined that the issue is limited specifically to the NoMachine Enterprise Client and does not occur when using the player that is bundled with the full NoMachine Server installation

That is a sufficient solution for me at this time – it will at least allow me to sunset my RSA key – though I’d rather not have to install the full version on machines that do not need to be accessed remotely.

Can you please try to reproduce this issue using the NoMachine Enterprise Client specifically on Windows?

[Steve92]

Hello,

A few weeks ago, I had some problems too with ED25519 algorithm to generate keys and I thought it was not supported (hence my question https://forum.nomachine.com/topic/ed25519-algorithm-for-ssh-nx-keys ).

I’ve just done a test in full Linux environment, all is OK. (following https://kb.nomachine.com/AR02L00785)

On “!M Client” 8.14 side:

$ ssh-keygen -t ed25519               (-b is useless since fixed length key)

I kept default key names and added a passphrase.

The server is “!M Enterprise Cloud Server” 8.14.

FYI, ssh version :

$ ssh -V

OpenSSH_9.2p1
OpenSSL 3.0.14   4/6/2024

O/S:

Debian 12 Bookworm
Linux antix1 6.1.105
(super light Linux, perfect for testing !M in live VMs)

Good luck ! 🙂

NB: RSA 4096-bit key is still strong enough (even 3072-bit for common usage) !

Regards,

Steve.

[Steve92]

Hi,

I did a quick successful test closer to your need. 🙂

# !M Client 8.14.2 installed on:
Microsoft Windows 11 Enterprise Evaluation (expired from a few months)
Version 10.0.22621 Build 22621
VM under “VMWare Player 17” on “Debian 11”: 4 CPU 8 Go RAM

> ssh -V:
OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3

> ssh-keygen -t ed25519
(default path+filenames+a passphrase entered)

Public key
C:\Users\User\.ssh\id_ed25519.pub
transferred to Linux remote server and added to
/home/my-user/.nx/config/authorized.crt

# Remote server:
“!M Enterprise Desktop 8.14.2” (evaluation)
VPS “Debian 12”, 1 vCPU, 2 Go !
LXDE

# In “!M Client” on W11 VM
For “My Enterprise Desktop” created connection
Edit/Configuration
x Use key-based auth. with a key you provide
[ Modify ]
C:\Users\User\.ssh\id_ed25519

That works like a charm !

Good luck and go ahead ! 😉

Steve.

 

 

[neatchee]

This issue persists. The instructions provided by Steve92 do not resolve the problem.

To reiterate the issue: ED25519 keys cannot be read by the standalone Enterprise Client or by the Android Client

Detailed steps to reproduce the issue:

1. Install !M Free Edition 8.14.2 server on a Windows 11 device
2. Generate an ED25519 SSH key using any recent version of OpenSSH’s ssh-keygen
   ❯ ssh -V
   OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2
   ❯ ssh-keygen -t ed25519 -f ./testkeyfornomachine -C “TestKeyForNoMachine”
3. Add the generated public key to the authorized.crt file on the Windows 11 device
   $env:USERPROFILE\.nx\config\authorized.crt
4. On a separate Windows 11 device, install the
5. Launch the !M standalone Enterprise Client v8.14.2 and add a ‘Machine’ for the Windows 11 host, setting authentication to use the generated ED25519 SSH private key
6. Attempt to connect to the Windows 11 host from the Windows standalone enterprise client using the newly added ‘Machine’ entry
7. On any Android device, install the !M Android client
8. Launch the !M Android client and add a ‘Machine’ for the Windows 11 host, setting authentication to use the generated ED25519 SSH private key
9. Attempt to connect to the Windows 11 host from the Android device using the newly added ‘Machine’ entry

Expected results:
Both the Windows standalone Enterprise client and the Android client are able to successfully connect to the Windows 11 host using the ED25519 SSH key

Actual results:
Both the Windows standalone Enterprise client and the Android client are not able to connect, showing “Authentication failed, please try again”. Inspecting client logs shows errors like:
2024-12-08 16:32:02 278.476 Encryptor/Encryptable: ERROR! Failed to read private key.
Error: Failed to read private key.

Reproduction rate:
5/5

Notes:
To reiterate, this issue does not happen with the server’s installation included client. This only happens for the standalone Enterprise Client and the Android Client

[neatchee]

Not sure why links were stripped from that post, but it should reference the following two URLs:

!M ENTERPRISE CLIENT (STANDALONE) – https://www.nomachine.com/product&p=NoMachine%20Enterprise%20Client

!M ANDROID CLIENT – https://play.google.com/store/apps/details?id=com.nomachine.nxplayer

[Britgirl]

Thanks for clarifying the links. We are not quite sure why you are encountering this difference when connecting from the standalone Enterprise Client/Android app but not when connecting from the Free Edition because the implementation is the same code shipped in all cases. 

Can you tell us if you are importing the key in the connection file when connecting from the EC and mobile clients?

There is a Trouble Report already open which affects importing the private key, the error shown is “Failed to read private key. Error: Failed to read private key.”

https://www.nomachine.com/TR05S10271

Try not importing the key at all inside the connection file, but instead placing the key on the device. Sometimes also generating new keys is helpful, as the problem is due to the characters used by base64 encoding for padding, breaking XML parsing. A new key has a different base64 encoding and thus can just work.

 

[neatchee]

Hi Britgirl,

No, I am NOT importing the key into the connection file. It is being read from disk at runtime.

I have tried regenerating a new key with no luck. However, if it’s a matter of random luck regarding characters that interfere with encoding/decoding, I will try a few more times to see if it magically works eventually

[neatchee]

I have now tried with 10 different freshly generated ED25519 keys. None of them worked.

I have also tried with both importing the key into the connection file, and with reading from disk for all 10 freshly generated keys. None of them worked.

Can you give me some guidance on exactly what base64 encoded values interfere with the XML parsing? If I know this, I might be able to repeatedly generate keys until I know I have one without the padding characters that interfere with XML parsing.

As it stands, I can only conclude that that there is something very subtly different about the Android and Standalone Enterprise clients build process, despite using the same codebase, that is causing ED25519 keys to fail to read in these cases.

[Author]

Posts