Virtual memory question

[rtsoliday]

When I run “top” I am seeing that nxnode.bin is listed as using 47.7g of memory:
PID       PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
19618       0 -20   47.7g 159648  18868 S   0.0  0.5   1:50.74 nxnode.bin
This is a RHEL7 system with version 6.4.6 installed. If I kill the process, it is automatically restarted and only uses a tiny bit of memory. After I connect remotely from home and come in to work the next day, this is what I see. Since it is only virtual memory, do I even have to worry about it?

[graywolf]

Would run pmap to explore memory allocated by nxnode.bin? That is:
pmap -p NNNN -X > nxnode.pmap
(put nxnode.bin process ID in place of NNNN).

[rtsoliday]

Here is is before I connect:
https://ops.aps.anl.gov/downloads/nxnode.pmap.initial.txt

And here it is after I connect and disconnect:
https://ops.aps.anl.gov/downloads/nxnode.pmap.txt

You can see a huge memory chunk at:
0000000200000000 46137344K —–   [ anon ]

[graywolf]

As we know the memory chunk size, you could try to catch the allocation, using gdb:

1. Restart NoMachine to have a fresh nxnode.bin process
2. Attach to nxnode.bin (replace NNNN with process ID):
# gdb -p NNNN
3. In gdb, Insert breakpoint:
(gdb) b mmap64 if $rsi=47244640256
4. Let process run:
(gdb) c
5. Connect to server using NoMachine client. gdb will hit the breakpoint and start waiting for commands: ask for backtrace:
(gdb) bt
6. Print the id of file descriptor that is going to be mapped. At this point it should be in register R8:
(gdb) p $r8
7. Keep process suspended. Without closing gdb (use a different terminal) query file descriptors with lsof and inspect results searching for the one found in $r8 with gdb:
# lsof -p NNNN
8. After you gathered info, you can quit gdb:
(gdb) q

[rtsoliday]

(gdb) b mmap64 if $rsi=47244640256
Breakpoint 1 at 0x7f29b2f2c330
(gdb) c
Continuing.
[New Thread 0x7f2980ff9700 (LWP 2786)]
[Switching to Thread 0x7f29a37fe700 (LWP 29781)]

Breakpoint 1, 0x00007f29b2f2c330 in mmap64 () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f29b2f2c330 in mmap64 () from /lib64/libc.so.6
#1  0x00007f29b2eb5ca2 in sysmalloc () from /lib64/libc.so.6
#2  0x00007f29b2eb6aba in _int_malloc () from /lib64/libc.so.6
#3  0x00007f29b2eb91ac in malloc () from /lib64/libc.so.6
#4  0x00007f29b19a59f8 in operator new(unsigned long) () from /usr/NX/lib/libstdc++.so.6
#5  0x00007f29b17161fb in NXShadowCreate () from /usr/NX/lib/libnxdimi.so
#6  0x00007f29b3c16fc0 in nxagentShadowInit () from /usr/NX/lib/libnxdiag.so
#7  0x00007f29b3bf7a5b in nxagentShadowClientCallback () from /usr/NX/lib/libnxdiag.so
#8  0x00007f29b14045dc in _CallCallbacks () from /usr/NX/lib/libnxdifb.so
#9  0x00007f29b13ff1df in NextAvailableClient () from /usr/NX/lib/libnxdifb.so
#10 0x00007f29b146fced in ?? () from /usr/NX/lib/libnxdifb.so
#11 0x00007f29b1470d7e in EstablishNewConnections () from /usr/NX/lib/libnxdifb.so
#12 0x00007f29b1404101 in ProcessWorkQueue () from /usr/NX/lib/libnxdifb.so
#13 0x00007f29b146a898 in WaitForSomething () from /usr/NX/lib/libnxdifb.so
#14 0x00007f29b13ff677 in Dispatch () from /usr/NX/lib/libnxdifb.so
#15 0x00007f29b13ecb1b in _NXDisplayMain () from /usr/NX/lib/libnxdifb.so
#16 0x00007f29b2424844 in ?? () from /usr/NX/lib/libnxc.so
#17 0x00007f29b3208dd5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f29b2f31ead in clone () from /lib64/libc.so.6

(gdb) p $rsi
$1 = 47244640256
(gdb) p $rdx
$2 = 3
(gdb) p $rcx
$3 = 34
(gdb) p $r8
$4 = 4294967295

I didn’t see anything related to 4294967295 in the lsof output, but here it is:
https://ops.aps.anl.gov/downloads/nxnode.lsof.txt

[graywolf]

Thanks, that is very useful.

Would you send the log file at /usr/NX/var/log/node/C-challenger.aps.anl.gov-1001-53D4418C3D3EDA98900CF407B5AD065D/session ?

Do you mind retrying with a debug library in order to gather more information? In that case, please write to forum (at) nomachine (dot) com to receive instructions.

[rtsoliday]

The session file and the request for the debugging version has been sent to the email. Thanks.

[graywolf]

Would you remove symlink /usr/NX/lib/libstdc++.so.6, restart the server and see if something changed?

Would you provide the output of commands

sudo dpkg -l libstdc++\*
sudo dpkg -L libstdc++6

and the file /usr/NX/var/log/nxinstall.log ?

Thank you.

[Author]

Posts