Wrong used memory in SLES12

Alex · July 2019

Hi.
I have wrong used memory in my SLES12SP4 server.
Jul 3 13:37:01 2019 version 6.01-0:|::::mem:::1056788356:1033513692:23274664

31266936:1002246756:
in_use_work is 31G, but HTOP show

I find info (https://stackoverflow.com/questions/41224738/how-to-calculate-system-memory-usage-from-proc-meminfo-like-htop) and want correct algorithm memory calculate. I do this, but not see result.

Please, ask in wich strings in /opt/lpar2rrd-agent/lpar2rrd-agent.pl i can make changes?
I make changes in section ###Linux memory###, but cant see result, as ignored it.

[root@stor2rrd new]# cat lpar2rrd-agent.pl | grep -A 10 "#A#"

#A#

my $sreclaim = 0;

--

#A# add $sreclaim section

if ( $line =~ m/^SReclaimable/ ) {

( undef, $sreclaim ) = split( / +/, $line );

}

if ( $line =~ m/^Cached:/ ) {

#A#

##( undef, $in_use_clnt ) = split( / +/, $line );

##$in_use_clnt = $in_use_clnt + $buffers;

### Buffers must be added to Cached

( undef, $in_use_clnt ) = split( / +/, $line );

$in_use_clnt = $in_use_clnt + $sreclaim - $shared;

}

--

#A#

##$inuse = $size - $free - $in_use_clnt;

##$in_use_work = $inuse; # used mem without the FS cache - FS cache is not here!

##$inuse = $inuse + $in_use_clnt;

$inuse = $size - $free;

$in_use_work = $inuse - $in_use_clnt; # FS cache is here

P.S. you use $buffers (in $in_use_clnt) before define his value. In this secton (###Linux memory###).

Alex · July 2019

Please, one a moment. I see, you scan output /proc/meminfo string by sting.
I need time to make algorinthm with this specific.

Alex · July 2019

OK, i make this changes and see result as HTOP (maked changes above section ###Linux memory###):

testdb:/opt/lpar2rrd-agent # cat lpar2rrd-agent.pl | grep -A 10 "#A#"

#A# add 2 vars

my $pcached = 0;

my $sreclaim = 0;

--

#A# add if{}

if ( $line =~ m/^SReclaimable:/ ) {

( undef, $sreclaim ) = split( / +/, $line );

}

if ( $line =~ m/^Cached:/ ) {

#A#

##( undef, $in_use_clnt ) = split( / +/, $line );

##$in_use_clnt = $in_use_clnt + $buffers;

### Buffers must be added to Cached

( undef, $pcached ) = split( / +/, $line );

}

--

#A#

##$inuse = $size - $free - $in_use_clnt;

##$in_use_work = $inuse; # used mem without the FS cache - FS cache is not here!

##$inuse = $inuse + $in_use_clnt;

$inuse = $size - $free;

$in_use_clnt = $pcached + $sreclaim - $shared;

$in_use_work = $inuse - $in_use_clnt; # buffers is here

Alex · July 2019

Why i need do this? In my server 1TiB RAM and many Oracle instances, and his can't use only 31GiB memory <span>

</span>
I compare output
# cat /proc/meminfo; free
and read article (link at top).

Can you make changes in Agent Lapr2RRD in future verions?

Alex · July 2019

P.S.
Need correct code:

if ( $line =~ m/^Shmem/ ) {

( undef, $shared ) = split( / +/, $line );

TO

if ( $line =~ m/^Shmem:/ ) {

( undef, $shared ) = split( / +/, $line );

Because in /proc/meminfo has:
Shmem

ShmemHugePages

ShmemPmdMapped

Our need only "Shmem:"

Pavel · July 2019

Hi,

your right, fixed, do you want to get a rpm with that fix?

Thanks!

Alex · July 2019

Hi, Pavel
Yes, if it posible, i want RPM with fix.
For easy update, i have many Linux servers.
Thanks!

Pavel · July 2019

http://www.lpar2rrd.com/download/lpar2rrd-agent-6.01-8.noarch.rpm

Alex · July 2019

Pavel, not for a long time of you, see this articles:
http://calimeroteknik.free.fr/blag/?article20/really-used-memory-on-gnu-linux
http://http//nopipi.hatenablog.com/entry/2015/09/13/181026 (translate to you lang)

Unfortunately, NMON have small of data about memory, and i need modify to $in_use_work value "active". (In my case Oracle RDBMS used Shmem)

Pavel · July 2019

I do not understand

Alex · July 2019

Hi, Pavel.
This topic i made when see, what LPAR Agent incorrect display used memory on my server.
On my server used Oracle RDBMS without HugePages, is used SharedMemory.
But current algorithm LPAR Agent do not use Shmem in calculation. Why? I search some articles, and find, what any programm calc used memory by any methods.
More simply it in pictures in articles, what i write links above.

For Linux, around string "###Linux memory###" LPAR Agent calc:
UsedMemory = MemTotal - MemFree - Cached - Buffers.

In articles you can see calc:
UsedMemory = MemTotal - MemFree - Cached - Buffers - SReclaimable + Shmem.

Can you analise this, and if it`s good, make changes in LPAR Agent script?

Alex · July 2019

P.S. about NMon part script of LPAR Agent.

If my application on the server (SLES12) use SharedMemory, NMon too incorect display used memory. And NMon have very small parameters to right calculate UsedMemory.

In my case, i need use Active parameter, as UsedMemory:

( undef, undef, $size, undef, undef, undef, $free, undef, undef, undef, undef, $in_use_clnt, $nactive ) = split( /,/, $mem );
...

$inuse = $size - $free;

$in_use_work = $nactive * 1024;

$in_use_clnt = $inuse - $in_use_work;

How make right calculating UsedMemory by NMon data (if used Shared Memory), i dont know now.

Alex · February 2020

I update Linux agent to version 6.15 and make changes to calc SharedMemory as Used (like algorithm in Htop program, actual for Oracle RDBMS)

[root@stor2rrd 1]# diff /1/lpar2rrd-agent.pl /2/lpar2rrd-agent.pl

126c126

< my $version = "6.15-0";

---

> my $version = "6.15-0a";

1497a1498,1500

> ##add 2 vars

> my $pcached = 0;

> my $sreclaim = 0;

1637a1641,1644

> ##add IF

> if ( $line =~ m/^SReclaimable:/ ) {

> ( undef, $sreclaim ) = split( / +/, $line );

> }

1639,1641c1646,1649

< ( undef, $in_use_clnt ) = split( / +/, $line );

< $in_use_clnt = $in_use_clnt + $buffers;

< # Buffers must be added to Cached

---

> #( undef, $in_use_clnt ) = split( / +/, $line );

> #$in_use_clnt = $in_use_clnt + $buffers;

> ## Buffers must be added to Cached

> ( undef, $pcached ) = split( / +/, $line );

1650,1652c1658,1663

< $inuse = $size - $free - $in_use_clnt;

< $in_use_work = $inuse; # used mem without the FS cache - FS cache is not here!

< $inuse = $inuse + $in_use_clnt;

---

> #$inuse = $size - $free - $in_use_clnt;

> #$in_use_work = $inuse; # used mem without the FS cache - FS cache is not here!

> #$inuse = $inuse + $in_use_clnt;

> $inuse = $size - $free;

> $in_use_clnt = $pcached + $sreclaim - $shared;

> $in_use_work = $inuse - $in_use_clnt; # buffers is here

Pavel · February 2020

Hi,

can you show some example what was wrong and what is a solution? graph, command line etc.

Alex · February 2020

testdb3:~ # free

total used free shared buffers cached

Mem: 528313444 211444856 316868588 158341632 1588816 185538936

-/+ buffers/cache: 24317104 503996340

Swap: 12585980 0 12585980

Image: https://forum.xorux.com/uploads/editor/qq/hk4plwh4axv5.png

Image: https://forum.xorux.com/uploads/editor/26/jq0vch69fa6p.png

Alex · February 2020

This changes based by article:
https://stackoverflow.com/questions/41224738/how-to-calculate-system-memory-usage-from-proc-meminfo-like-htop

Pavel · February 2020

shared is already part of used as per me, look below, then you are wrong as per me

# free
             total       used       free     shared    buffers     cached
Mem:     528313444 211444856 316868588 158341632    1588816 185538936
-/+ buffers/cache:   24317104 503996340
Swap:     12585980          0   12585980

used+free+buffers ~= total
211444856+316868588+1588816=529902260 =~ 528313444

Alex · March 2020

Hi, Pavel.

Ok, but some correction, you use out not "free", but "cat /proc/meminfo" as:

testdb3:~ # free

total used free shared buffers cached

Mem: 528313444 519418200 8895244 168357348 1491496 494083384

-/+ buffers/cache: 23843320 504470124

Swap: 12585980 0 12585980

testdb3:~ # cat /proc/meminfo

MemTotal: 528313444 kB

MemFree: 8833812 kB

MemAvailable: 331718428 kB

Buffers: 1491512 kB

Cached: 480176436 kB

SwapCached: 0 kB

Active: 146629136 kB

Inactive: 350354912 kB

Active(anon): 132600260 kB

Inactive(anon): 51080076 kB

Active(file): 14028876 kB

Inactive(file): 299274836 kB

Unevictable: 40540 kB

Mlocked: 40540 kB

SwapTotal: 12585980 kB

SwapFree: 12585980 kB

Dirty: 200 kB

Writeback: 0 kB

AnonPages: 15355576 kB

Mapped: 102649480 kB

Shmem: 168357480 kB

Slab: 14865832 kB

SReclaimable: 13907736 kB

SUnreclaim: 958096 kB

KernelStack: 78448 kB

PageTables: 6734328 kB

NFS_Unstable: 0 kB

Bounce: 0 kB

WritebackTmp: 0 kB

CommitLimit: 488068076 kB

Committed_AS: 292903228 kB

VmallocTotal: 34359738367 kB

VmallocUsed: 0 kB

VmallocChunk: 0 kB

HardwareCorrupted: 0 kB

AnonHugePages: 0 kB

ShmemHugePages: 0 kB

ShmemPmdMapped: 0 kB

HugePages_Total: 0

HugePages_Free: 0

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 2048 kB

DirectMap4k: 6617004 kB

DirectMap2M: 294266880 kB

DirectMap1G: 238026752 kB

and define values:

MemTotal: as $size

MemFree: as $free

Shmem: as $shared

Cached:+$buffers(at monet as 0) $in_use_clnt+$buffers

MemAvailable: as $available

Buffers: as $buffers

final calc:

$inuse = $size - $free - $in_use_clnt; (MemTotal - MemFree - Cached)

$in_use_work = $inse; (MemTotal - MemFree - Cached)

$inuse = $inuse + $in_use_clnt; (MemTotal - MemFree - Cached) + Cached = (MemTotal - MemFree)

*) you DO NOT USE in calc: $shared, $buffers and $available. WHY?

So, in numbers:

MemTotal = 528 313 444

MemFree = 8 833 812

$in_use_work = 528313444 - 8833812 - 480176436 = 39 303 196

$inuse = 528313444 - 8833812 = 519 479 632

In your draw "FS cache" is ($inuse - $in_use_work) - is Cached - is 480176436.

Command HTOP say: "Mem:504G used:184G buffers:1.42G cache:311G"

Why HTOP used:184G but lpar2rrd in_use_work:39G ?

Because, HTOP see "Shmem" and "SReclaimable".

(https://stackoverflow.com/questions/41224738/how-to-calculate-system-memory-usage-from-proc-meminfo-like-htop)

Total used memory = MemTotal - MemFree

Non cache/buffer memory (green) = Total used memory - (Buffers + Cached memory)

Buffers (blue) = Buffers

Cached memory (yellow) = Cached + SReclaimable - Shmem

Swap = SwapTotal - SwapFree

So, HTOP calc "FS cache" = Buffers + (Cached + SReclaimable - Shmem).

Calculate usage memory it not simple. I find article, online translate it, and see pictures (http://nopipi.hatenablog.com/entry/2015/09/13/181026)

I think, calc used memory as HTOP more right. And my Oracle RDBMS let my see it.

Wrong used memory in SLES12

Comments

Howdy, Stranger!

Categories

In this Discussion