lpar2rrd vmware dashboards appear empty -nan-

mahmut
mahmut
in VMware
Hello,
lpar2rrd vmware dashboards appear empty,
we are having this problem almost all vm dashboards
I guess something happened at 5 March 2:30 
adding screenshot, could you help please?
 

Comments

  • Jindra
    Jindra
    send us actual logs

    Note a short problem description in the text field of the upload form.

    cd /home/lpar2rrd/lpar2rrd  # or where is your LPAR2RRD working dir

    tar cvhf logs.tar logs tmp/*txt

    gzip -9 logs.tar

    Send us logs.tar.gz via https://upload.lpar2rrd.com

  • mahmut
    mahmut
    I have uploaded the logs, thanks
  • Jindra
    Jindra
    this info pls:
    ps -ef|grep vmware_push_perf

  • mahmut
    mahmut
    command output: 
    ps -ef|grep vmware_push_perf
    lpar2rrd  23413      1  0 Mar28 ?        00:00:00 /bin/bash /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.sh
    lpar2rrd  23428  23413  0 Mar28 ?        00:00:01 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.pl
  • Jindra
    Jindra
    those processes are blocked, kill them
    kill 23413,23428
    wait about 2 hours and check gui (heatmap)

  • mahmut
    mahmut
    still not working, heatmap and other dashboards empty no data
  • Jindra
    Jindra
    pls again
    command output: 
    ps -ef|grep vmware_push_perf

  • mahmut
    mahmut
    output:

    lpar2rrd  48362      1  0 Apr06 ?        00:00:00 /bin/bash /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.sh
    lpar2rrd  48379  48362  0 Apr06 ?        00:06:16 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.pl

  • Jindra
    Jindra
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | head
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | tail
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | wc -l



  • mahmut
    mahmut
    ]# ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | head
    total 41434492
    -rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 00:01 1646427670_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  7936000 Mar  5 00:01 1646427692_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 00:31 1646429468_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  7895040 Mar  5 00:31 1646429489_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 01:01 1646431271_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  8028160 Mar  5 01:01 1646431290_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 01:31 1646433072_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  8314880 Mar  5 01:31 1646433090_..txt.tar
    -rw-r--r-- 1 lpar2rrd lpar2rrd  5171200 Mar  5 02:01 1646434870_..txt.tar
    # ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | tail               
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9808980 Apr  7 08:31 1649309484_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16914947 Apr  7 08:31 1649309509_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9818069 Apr  7 09:01 1649311289_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16922326 Apr  7 09:01 1649311311_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9824479 Apr  7 09:31 1649313083_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16972821 Apr  7 09:31 1649313108_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9829121 Apr  7 10:01 1649314883_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16975638 Apr  7 10:01 1649314909_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9851012 Apr  7 10:31 1649316685_..txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16980435 Apr  7 10:31 1649316709_..txt
    # ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE | wc -l
    3213

  • Jindra
    Jindra
    these ones yet
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | head
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | tail


  • mahmut
    mahmut
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16566605 Mar  6 06:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646535702_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9748424 Mar  6 06:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646537478_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16705286 Mar  6 06:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646537500_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9748369 Mar  6 07:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646539281_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16529399 Mar  6 07:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646539304_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9749625 Mar  6 07:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646541083_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16570496 Mar  6 07:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646541101_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9744184 Mar  6 08:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646542880_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16576499 Mar  6 08:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646542903_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9741649 Mar  6 08:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646544685_.txt


    -rw-r--r-- 1 lpar2rrd lpar2rrd  9824479 Apr  7 09:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649313083_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16972821 Apr  7 09:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649313108_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9829121 Apr  7 10:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649314883_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16975638 Apr  7 10:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649314909_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9851012 Apr  7 10:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649316685_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16980435 Apr  7 10:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649316709_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9866635 Apr  7 11:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649318485_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16963741 Apr  7 11:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649318514_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9862063 Apr  7 11:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649320287_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16984186 Apr  7 11:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649320316_.txt
  • Jindra
    Jindra
    As you can see,those files are continuously pushed to RRD files, and the up to data state will be probably reached tomorrow, check those files by
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | head
    and you can see they are disappearing

  • mahmut
    mahmut
    Yes but dashboards are still empty
  • Jindra
    Jindra
    seems it again stucks
    again this:

    ps -ef|grep vmware_push_perf
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | head
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | tail



  • mahmut
    mahmut
    lpar2rrd  48362      1  0 Apr06 ?        00:00:00 /bin/bash /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.sh
    lpar2rrd  48379  48362  0 Apr06 ?        00:06:18 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/vmware_push_perf_data_to_rrd.pl

  • mahmut
    mahmut
    forgot these
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16566605 Mar  6 06:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646535702_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9748424 Mar  6 06:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646537478_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16705286 Mar  6 06:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646537500_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9748369 Mar  6 07:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646539281_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16529399 Mar  6 07:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646539304_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9749625 Mar  6 07:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646541083_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16570496 Mar  6 07:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646541101_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9744184 Mar  6 08:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646542880_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16576499 Mar  6 08:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646542903_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9741649 Mar  6 08:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1646544685_.txt



    -rw-r--r-- 1 lpar2rrd lpar2rrd  9808688 Apr  8 09:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649399478_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16925619 Apr  8 09:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649399502_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9813485 Apr  8 10:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649401286_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16948545 Apr  8 10:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649401307_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9832450 Apr  8 10:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649403092_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16942053 Apr  8 10:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649403117_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9841041 Apr  8 11:01 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649404891_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16953722 Apr  8 11:02 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649404920_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd  9838115 Apr  8 11:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649406687_.txt
    -rw-r--r-- 1 lpar2rrd lpar2rrd 16978185 Apr  8 11:31 /home/lpar2rrd/lpar2rrd/tmp/VMWARE/1649406712_.txt
  • Jindra
    Jindra
    seems the vmware_push_perf_data_to_rrd.pl stucks
    kill the jobs
    kill 48362,48379 

    check after some hours either in graphs legend - "Updated. time" should change
    or
    ls -l /home/lpar2rrd/lpar2rrd/tmp/VMWARE/*.txt | head
    the files should start to disappear




  • mahmut
    mahmut
    updated time changed but dashboard still has no values, all of them empty
  • Pavel
    Pavel
    Hi,

    upgrade to the latest available version 7.40 and let us know if the probvlem persits.
  • mahmut
    mahmut
    Hi,

    upgraded 7.2 to 7.4  but graphs are still empty
  • mahmut
    mahmut
    Hi

    we renamed a vmware cluster, then graphs are created for that cluster on lpar2rrd,
    do we need some data cleaning?
    how can we solve this for other clusters? 
  • Jindra
    Jindra
    Hi,
    you mean, dashboard graphs for that renamed cluster are OK? and other graphs not?
    no cleaning is necessary
    send us againg logs as above (April 5 message)
  • mahmut
    mahmut
    Uploaded logs,
    yes dashboard graphs for that renamed cluster are OK, and other graphs not,
    thanks
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Categories

In this Discussion