lpar2rrd vmware dashboards appear empty -nan-

Hello,
lpar2rrd vmware dashboards appear empty,
we are having this problem almost all vm dashboards
I guess something happened at 5 March 2:30 
adding screenshot, could you help please?
 

Comments

  • send us actual logs

    Note a short problem description in the text field of the upload form.

    cd /home/lpar2rrd/lpar2rrd  # or where is your LPAR2RRD working dir

    tar cvhf logs.tar logs tmp/*txt

    gzip -9 logs.tar

    Send us logs.tar.gz via https://upload.lpar2rrd.com

  • I have uploaded the logs, thanks
  • this info pls:
    ps -ef|grep vmware_push_perf

  • command output: 
    ps -ef|grep vmware_push_perf
    lpar2rrd  23413      1  0 Mar28 ?        00:00:00 /bin/bash /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.sh
    lpar2rrd  23428  23413  0 Mar28 ?        00:00:01 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.pl
  • those processes are blocked, kill them
    kill 23413,23428
    wait about 2 hours and check gui (heatmap)

  • still not working, heatmap and other dashboards empty no data
  • pls again
    command output: 
    ps -ef|grep vmware_push_perf

  • output:

    lpar2rrd  48362      1  0 Apr06 ?        00:00:00 /bin/bash /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.sh
    lpar2rrd  48379  48362  0 Apr06 ?        00:06:16 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.pl

  • ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | head
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | tail
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | wc -l



  • ]# ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | head
    total 41434492
    -rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 00:01 1646427670_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  7936000 Mar  5 00:01 1646427692_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 00:31 1646429468_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  7895040 Mar  5 00:31 1646429489_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 01:01 1646431271_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  8028160 Mar  5 01:01 1646431290_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 01:31 1646433072_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  8314880 Mar  5 01:31 1646433090_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 02:01 1646434870_..txt.tar
    # ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | tail               
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9808980 Apr  7 08:31 1649309484_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16914947 Apr  7 08:31 1649309509_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9818069 Apr  7 09:01 1649311289_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16922326 Apr  7 09:01 1649311311_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9824479 Apr  7 09:31 1649313083_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16972821 Apr  7 09:31 1649313108_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9829121 Apr  7 10:01 1649314883_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16975638 Apr  7 10:01 1649314909_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9851012 Apr  7 10:31 1649316685_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16980435 Apr  7 10:31 1649316709_..txt
    # ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | wc -l
    3213

  • these ones yet
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | head
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | tail


  • -rw-r--r-- 1 lpar2rrd lpar2rrd 16566605 Mar  6 06:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646535702_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9748424 Mar  6 06:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646537478_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16705286 Mar  6 06:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646537500_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9748369 Mar  6 07:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646539281_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16529399 Mar  6 07:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646539304_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9749625 Mar  6 07:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646541083_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16570496 Mar  6 07:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646541101_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9744184 Mar  6 08:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646542880_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16576499 Mar  6 08:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646542903_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9741649 Mar  6 08:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646544685_.txt


    -rw-r--r-- 1 lpar2rrd lpar2rrd  9824479 Apr  7 09:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649313083_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16972821 Apr  7 09:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649313108_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9829121 Apr  7 10:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649314883_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16975638 Apr  7 10:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649314909_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9851012 Apr  7 10:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649316685_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16980435 Apr  7 10:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649316709_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9866635 Apr  7 11:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649318485_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16963741 Apr  7 11:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649318514_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9862063 Apr  7 11:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649320287_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16984186 Apr  7 11:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649320316_.txt
  • As you can see,those files are continuously pushed to RRD files, and the up to data state will be probably reached tomorrow, check those files by
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | head
    and you can see they are disappearing

  • Yes but dashboards are still empty
  • seems it again stucks
    again this:

    ps -ef|grep vmware_push_perf
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | head
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | tail



  • lpar2rrd  48362      1  0 Apr06 ?        00:00:00 /bin/bash /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.sh
    lpar2rrd  48379  48362  0 Apr06 ?        00:06:18 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.pl

  • forgot these
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16566605 Mar  6 06:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646535702_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9748424 Mar  6 06:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646537478_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16705286 Mar  6 06:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646537500_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9748369 Mar  6 07:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646539281_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16529399 Mar  6 07:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646539304_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9749625 Mar  6 07:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646541083_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16570496 Mar  6 07:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646541101_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9744184 Mar  6 08:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646542880_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16576499 Mar  6 08:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646542903_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9741649 Mar  6 08:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646544685_.txt



    -rw-r--r-- 1 lpar2rrd lpar2rrd  9808688 Apr  8 09:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649399478_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16925619 Apr  8 09:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649399502_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9813485 Apr  8 10:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649401286_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16948545 Apr  8 10:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649401307_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9832450 Apr  8 10:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649403092_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16942053 Apr  8 10:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649403117_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9841041 Apr  8 11:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649404891_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16953722 Apr  8 11:02 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649404920_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9838115 Apr  8 11:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649406687_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16978185 Apr  8 11:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649406712_.txt
  • seems the vmware_push_perf_data_to_rrd.pl stucks
    kill the jobs
    kill 48362,48379 

    check after some hours either in graphs legend - "Updated. time" should change
    or
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | head
    the files should start to disappear




  • updated time changed but dashboard still has no values, all of them empty
  • Hi,

    upgrade to the latest available version 7.40 and let us know if the probvlem persits.
  • Hi,

    upgraded 7.2 to 7.4  but graphs are still empty
  • Hi

    we renamed a vmware cluster, then graphs are created for that cluster on lpar2rrd,
    do we need some data cleaning?
    how can we solve this for other clusters? 
  • Hi,
    you mean, dashboard graphs for that renamed cluster are OK? and other graphs not?
    no cleaning is necessary
    send us againg logs as above (April 5 message)
  • Uploaded logs,
    yes dashboard graphs for that renamed cluster are OK, and other graphs not,
    thanks
Sign In or Register to comment.