lpar2rrd vmware dashboards appear empty -nan-

mahmut

Hello,
lpar2rrd vmware dashboards appear empty,
we are having this problem almost all vm dashboards
I guess something happened at 5 March 2:30 
adding screenshot, could you help please?
 

Jindra

send us actual logs

Note a short problem description in the text field of the upload form.

cd /home/lpar2rrd/lpar2rrd  # or where is your LPAR2RRD working dir

tar cvhf logs.tar logs tmp/*txt

gzip -9 logs.tar

Send us logs.tar.gz via https://upload.lpar2rrd.com

mahmut

I have uploaded the logs, thanks

Jindra

this info pls:

ps -ef|grep vmware_push_perf

mahmut

command output: 
ps -ef|grep vmware_push_perf

lpar2rrd  23413      1  0 Mar28 ?        00:00:00 /bin/bash /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.sh

lpar2rrd  23428  23413  0 Mar28 ?        00:00:01 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.pl

Jindra

those processes are blocked, kill them

kill 23413,23428

wait about 2 hours and check gui (heatmap)

mahmut

still not working, heatmap and other dashboards empty no data

Jindra

pls again

command output: 
ps -ef|grep vmware_push_perf

mahmut

output:

lpar2rrd  48362      1  0 Apr06 ?        00:00:00 /bin/bash /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.sh

lpar2rrd  48379  48362  0 Apr06 ?        00:06:16 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.pl

Jindra

ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | head

ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | tail

ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | wc -l

mahmut

]# ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | head

total 41434492

-rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 00:01 1646427670_..txt.tar

-rw-r--r-- 1 lpar2rrd lpar2rrd  7936000 Mar  5 00:01 1646427692_..txt.tar

-rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 00:31 1646429468_..txt.tar

-rw-r--r-- 1 lpar2rrd lpar2rrd  7895040 Mar  5 00:31 1646429489_..txt.tar

-rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 01:01 1646431271_..txt.tar

-rw-r--r-- 1 lpar2rrd lpar2rrd  8028160 Mar  5 01:01 1646431290_..txt.tar

-rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 01:31 1646433072_..txt.tar

-rw-r--r-- 1 lpar2rrd lpar2rrd  8314880 Mar  5 01:31 1646433090_..txt.tar

-rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 02:01 1646434870_..txt.tar

# ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | tail               

-rw-r--r-- 1 lpar2rrd lpar2rrd  9808980 Apr  7 08:31 1649309484_..txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16914947 Apr  7 08:31 1649309509_..txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9818069 Apr  7 09:01 1649311289_..txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16922326 Apr  7 09:01 1649311311_..txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9824479 Apr  7 09:31 1649313083_..txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16972821 Apr  7 09:31 1649313108_..txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9829121 Apr  7 10:01 1649314883_..txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16975638 Apr  7 10:01 1649314909_..txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9851012 Apr  7 10:31 1649316685_..txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16980435 Apr  7 10:31 1649316709_..txt

# ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | wc -l

3213

Jindra

these ones yet

ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | head

ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | tail

mahmut

-rw-r--r-- 1 lpar2rrd lpar2rrd 16566605 Mar  6 06:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646535702_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9748424 Mar  6 06:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646537478_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16705286 Mar  6 06:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646537500_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9748369 Mar  6 07:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646539281_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16529399 Mar  6 07:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646539304_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9749625 Mar  6 07:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646541083_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16570496 Mar  6 07:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646541101_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9744184 Mar  6 08:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646542880_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16576499 Mar  6 08:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646542903_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9741649 Mar  6 08:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646544685_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9824479 Apr  7 09:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649313083_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16972821 Apr  7 09:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649313108_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9829121 Apr  7 10:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649314883_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16975638 Apr  7 10:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649314909_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9851012 Apr  7 10:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649316685_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16980435 Apr  7 10:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649316709_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9866635 Apr  7 11:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649318485_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16963741 Apr  7 11:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649318514_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9862063 Apr  7 11:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649320287_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16984186 Apr  7 11:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649320316_.txt

Jindra

As you can see,those files are continuously pushed to RRD files, and the up to data state will be probably reached tomorrow, check those files by

ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | head

and you can see they are disappearing

mahmut

Yes but dashboards are still empty

Jindra

seems it again stucks

again this:

ps -ef|grep vmware_push_perf

ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | head

ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | tail

mahmut

lpar2rrd  48362      1  0 Apr06 ?        00:00:00 /bin/bash /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.sh

lpar2rrd  48379  48362  0 Apr06 ?        00:06:18 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.pl

mahmut

forgot these

-rw-r--r-- 1 lpar2rrd lpar2rrd 16566605 Mar  6 06:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646535702_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9748424 Mar  6 06:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646537478_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16705286 Mar  6 06:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646537500_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9748369 Mar  6 07:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646539281_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16529399 Mar  6 07:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646539304_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9749625 Mar  6 07:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646541083_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16570496 Mar  6 07:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646541101_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9744184 Mar  6 08:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646542880_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16576499 Mar  6 08:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646542903_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9741649 Mar  6 08:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646544685_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9808688 Apr  8 09:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649399478_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16925619 Apr  8 09:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649399502_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9813485 Apr  8 10:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649401286_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16948545 Apr  8 10:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649401307_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9832450 Apr  8 10:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649403092_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16942053 Apr  8 10:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649403117_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9841041 Apr  8 11:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649404891_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16953722 Apr  8 11:02 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649404920_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd  9838115 Apr  8 11:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649406687_.txt

-rw-r--r-- 1 lpar2rrd lpar2rrd 16978185 Apr  8 11:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649406712_.txt

Jindra

seems the vmware_push_perf_data_to_rrd.pl stucks

kill the jobs

kill 48362,48379 

check after some hours either in graphs legend - "Updated. time" should change

or

ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | head

the files should start to disappear

mahmut

updated time changed but dashboard still has no values, all of them empty

Pavel

Hi,

upgrade to the latest available version 7.40 and let us know if the probvlem persits.

mahmut

Hi,

upgraded 7.2 to 7.4  but graphs are still empty

mahmut

Hi

we renamed a vmware cluster, then graphs are created for that cluster on lpar2rrd,
do we need some data cleaning?
how can we solve this for other clusters? 

Jindra

Hi,

you mean, dashboard graphs for that renamed cluster are OK? and other graphs not?

no cleaning is necessary

send us againg logs as above (April 5 message)

mahmut

Uploaded logs,
yes dashboard graphs for that renamed cluster are OK, and other graphs not,
thanks