Performance issue –> reboot necessary + CredentialsStoring Bug?

Forum / NoMachine Terminal Server Products / Performance issue –> reboot necessary + CredentialsStoring Bug?

Viewing 10 posts - 1 through 10 (of 10 total)
  • Author
    Posts
  • #30710
    termmachine
    Participant

    Hello,
    my Terminal Server Version is: NoMachine Terminal Server Subscription – Version 6.9.2
    and I am running it on Suse Linux Enterprise Server 12 SP5 which is virtualized in VMWare ESXi 6.7.0 Update 3.

    In our company we use the Terminal Server to offer our employees access to other systems via RDP and XDM and (x)VNC.
    Our employees are using the webplayer and our admins are using the installed Enterprise Client player.

    1)
    I have some big performance issues after around 1 week of runtime. I ran into some ressource problems, where no more session can be generated and the weirdest things happens (Error 11 no threads can be generated)…. Also the webplayer show the weirdest things to my employees in this error-state.
    Only rebooting the virtual machine + ESXi host helps.
    I send all logfiles + config files and ESXi settings screenshots to your support email.
    I also set all /etc/security/limits.conf parameters to unlimited where it is possible and the number of open files to the maximum allowed. Did not help.
    I saw on a fresh reboot the number of processes are around 340. During crash it is about 580 processes, but with no or few users logged in. Seems that they aren’t terminated correctly and so the number increases.
    What is the problem in my case?

    2)
    Moreover I have set the “EnableCredentialsStoring” parameter to “none”, but in webplayer and player users still can save their credentials…

    Kindly waiting for your help, thank you!

    #30718
    termmachine
    Participant

    I did another Test, seems I have a problem with getting ma /etc/security/limits.conf working…
    ATORNX002:~ # ps -ef | grep “nxserver.bin –daemon”
    nx        1751     1  1 14:54 ?        00:00:01 /usr/NX/bin/nxserver.bin –daemon
    root      2403  2339  0 14:55 pts/0    00:00:00 grep –color=auto nxserver.bin –daemon
    ATORNX002:~ # cat /proc/1751/limits
    Limit                     Soft Limit           Hard Limit           Units
    Max cpu time              unlimited            unlimited            seconds
    Max file size             unlimited            unlimited            bytes
    Max data size             unlimited            unlimited            bytes
    Max stack size            8388608              unlimited            bytes
    Max core file size        unlimited            unlimited            bytes
    Max resident set          unlimited            unlimited            bytes
    Max processes             71327                71327                processes
    Max open files            4096                 4096                 files
    Max locked memory         65536                65536                bytes
    Max address space         unlimited            unlimited            bytes
    Max file locks            unlimited            unlimited            locks
    Max pending signals       71327                71327                signals
    Max msgqueue size         819200               819200               bytes
    Max nice priority         0                    0
    Max realtime priority     0                    0
    Max realtime timeout      unlimited            unlimited            us

    In my conf file I have set the following:
    nx              soft    nproc           unlimited
    nx              hard    nproc           unlimited
    nx              soft    as              unlimited
    nx              hard    as              unlimited
    nx              soft    nofile          1048576
    nx              hard    nofile          1048576
    nx              soft    sigpending      unlimited
    nx              hard    sigpending      unlimited
    nx              soft    stack           unlimited
    nx              hard    stack           unlimited
    nx              soft    msgqueue        unlimited
    nx              hard    msgqueue        unlimited

    I also did a reboot, but settings dont work.

    #30721
    Britgirl
    Keymaster

    What has changed recently on the system in the past week?

    Have you already checked this article which is for similar issues and followed the advice there:

    Troubleshooting issues caused by system resources exceeded on Linux
    https://www.nomachine.com/AR04O00927

    Also the webplayer show the weirdest things to my employees in this error-state.

    What things? Can you submit screenshots of an example of what users see? What browser(s) are they using?

    Moreover I have set the “EnableCredentialsStoring” parameter to “none”, but in webplayer and player users still can save their credentials…

    Provide a screenshot of what users are seeing when they are at the authentication stage.
    Was access via the browser working correctly beforehand? Was the  EnableCredentialsStoring key working correctly before this? (we checked that it’s working in the latest version by the way). That key is set on the server, and setting to None will mean users won’t see the checkbox at the authentication stage.

    #30725
    termmachine
    Participant

    Hello,
    the only difference in the past was, that my system was used only by 2-3 users and I allowed now more users to use the systems (up to 20-30 simultaneously). Last week in config files I disabled some unnecessary (for our cases) features like balloon messages, monitor win, pulseaudio and so on to save resources.

    I have already checked your linked article, this was the reason I edited the /etc/security/limits.conf.
    But I don’t understand, why my set limits don’t work for nx user.
    For every other user my set limits works….

    The credentialstoring “none” worked in the past, but I cannot say if it was the ressource problem crash, or some other parameter in my config file, that it doesn’t work anymore.
    I also saw that the authentication page in webplayer is “English” now (with checkbox for saving credentials) and after that it is “German” again, as set in config files.

    Attachments:
    #30735
    termmachine
    Participant

    Hi, I got an answer from one of our webplayer users. During this ressource-error-state users saw the message “Node was not retrieved” (see screenshot).
    Moreover they saw another message like: “Service gesperrt” (German), I think in English it should be like “Service locked” (unfortunately no screenshot available).

    #30758
    Britgirl
    Keymaster

    The logs show several errors and carrying out a thorough debug of the affected server is not something we can do via the forums. You should consider opening a support ticket.

    But overall I would say that the machine is overloaded.

    You wrote:

    the only difference in the past was, that my system was used only by 2-3 users and I allowed now more users to use the systems (up to 20-30 simultaneously).

    If there is 16G ram tjhis could be the problem if there are many users are there. See our test comparisons here:
    https://www.nomachine.com/AR02P00966 for example. In a light environment (centos 6 with old Gnome which doesn’t consume resources like more modern systems)…

    …Number of concurrent sessions actively used without performance penalties:

    i) 13 users can surf the internet with Firefox with 8 GB of memory usage and 87% overall CPU usage.
    ii) 5 users working on a 3D cad app (FreeCad) with 5 GB of total memory usage and 82% overall CPU usage.
    iii) 32 users using a text editor (LibreOffice) with 12 GB of total memory usage and 86% overal CPU usage

    on NoMachine Terminal Server (2.40 GHz, 6 Cores, 20 GB RAM)

    Try upgrading the hardware of the TS host.

    #30831
    termmachine
    Participant

    Thank you for your input.
    Generally I don’t believe, that the machine is overloaded, because on ESXi interface CPU usage is never above 4% utilization and the virtual machine has 18GB RAM assigned and the maximum usage was 10GB (when all 30-40 user are logged in and use the custom RDP and VNC sessions).

    May I ask you if you can answer these 3 questions for me.

    1) I think I have to look, why /etc/security/limits.conf don’t work properly for the NX user, but works great with all other users.
    Have you ever seen this before? Do you know, if I have to configure the limits for NX user on another place?

    2) Can there be some limits in htd.cfg (because all my employees use the webplayer)?
    For example in this parameters:
    #
    # All these directives control the management of NXHTD’s
    # subprocesses for handling individual HTTP requests.
    #
    StartServers         1
    ThreadLimit        255
    ThreadsPerChild    240
    ServerLimit         16
    MaxRequestWorkers 1920
    MinSpareThreads     25
    MaxSpareThreads     75

    3) How can I reset the server.cfg and node.cfg to defaults?
    Because of the bug in “EnableCredentialsStoring” I want to start again from zero, and reconfig my needs.

    Thank you!

    #30839
    brotech
    Participant

    Hello,

    in case of limits per user, questions are for kernel and operating systems. NX user and any other users are under kernel control. I think processes
    are not simply counted as they can have many threads, and every thread is counted as well.

    Resource limit is controlled by kernel, so when you have issue try to find which process has a lot of files in use, which one has a lot of threads etc
    There are system tools which can help as lsof or ps.

    We have a guide available at:
    https://www.nomachine.com/AR04O00927

    which contains for example answers for your question regarding where limits are set for user nx if not in limits.conf
    […]
    Systemd limits are set in the following files:
    /etc/systemd/system.conf
    /etc/systemd/user.conf
    […]

    For default configuration file, during installation, setup is checking what kind of system it is, what tools are available among those required etc so there is no one default config key.
    If you need clear one, uninstall nx, remove .cfg and install again.

    #30847
    termmachine
    Participant

    Hello,
    thank you for your input, seems that /etc/systemd/system.conf is the solution. (Sorry I totally missed that part in your guide)

    I edited the following parameter in system.conf:
    DefaultTasksMax=infinity
    DefaultLimitSTACK=infinity
    DefaultLimitNOFILE=1048576
    DefaultLimitNPROC=infinity
    DefaultLimitMEMLOCK=infinity
    DefaultLimitSIGPENDING=infinity
    DefaultLimitMSGQUEUE=infinity

    Before editing limits for NX user was:
    DefaultTasksMax=512
    DefaultLimitSTACK=8388608
    DefaultLimitNOFILE=4096
    DefaultLimitNPROC=71327
    DefaultLimitMEMLOCK=65536
    DefaultLimitSIGPENDING=71327
    DefaultLimitMSGQUEUE=819200

    Now:
    ATORNX002:~ # ps -ef | grep nxserver
    nx        1721     1  3 12:15 ?        00:00:01 /usr/NX/bin/nxserver.bin –daemon
    root      2384  2331  0 12:15 pts/0    00:00:00 grep –color=auto nxserver
    ATORNX002:~ # cat /proc/1721/limits
    Limit                     Soft Limit           Hard Limit           Units
    Max cpu time              unlimited            unlimited            seconds
    Max file size             unlimited            unlimited            bytes
    Max data size             unlimited            unlimited            bytes
    Max stack size            unlimited            unlimited            bytes
    Max core file size        unlimited            unlimited            bytes
    Max resident set          unlimited            unlimited            bytes
    Max processes             unlimited            unlimited            processes
    Max open files            1048576              1048576              files
    Max locked memory         unlimited            unlimited            bytes
    Max address space         unlimited            unlimited            bytes
    Max file locks            unlimited            unlimited            locks
    Max pending signals       unlimited            unlimited            signals
    Max msgqueue size         unlimited            unlimited            bytes
    Max nice priority         0                    0
    Max realtime priority     0                    0
    Max realtime timeout      unlimited            unlimited            us

    All the infinity limits should be no problem, because no of my employees can do anything on my machine, except of using the webplayer with preconfigured session-files.

    Or do you see any reason not to do that?

    Looks good, now I have to wait and see, if all my resource problems are solved.
    Maybe you can leave that topic open for some time 🙂

    Thanks,
    Termmachine

    #32080
    Britgirl
    Keymaster

    No further feedback has been submitted by the user so we can consider this solved.

Viewing 10 posts - 1 through 10 (of 10 total)

This topic was marked as solved, you can't post.