graphs not updating after disk full

We're running stor2rrd 7.10-1 as a docker container, it's been running fine for a while for few weeks but few days ago the guest os running the container ran out of space.  After adding more space and restarting the container few times the graphs aren't updating anymore, they're just showing values from few days ago and then blank (it's been few hours since the disk space issue was fixed and container restarted).  I see the following errors in ~stor2rrd/stor2rrd/logs/error.log (we're using ibm v5030)

Thu Jul  8 17:03:59 2021: v5k2 : get_pool_csv_report error, no source: resp_t:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379

Thu Jul  8 17:04:06 2021: v5k2 : get_pool_csv_report error, no source: data_rate:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

Thu Jul  8 17:04:12 2021: v5k2 : get_pool_csv_report error, no source: io_rate:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

Thu Jul  8 17:04:21 2021: v5k2 : get_pool_csv_report error, no source: used:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

Thu Jul  8 17:05:34 2021: v5k1 : get_pool_csv_report error, no source: resp_t:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

Thu Jul  8 17:05:34 2021: v5k1 : get_pool_csv_report error, no source: data_rate:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

Thu Jul  8 17:05:34 2021: v5k1 : get_pool_csv_report error, no source: io_rate:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

Thu Jul  8 17:05:38 2021: v5k1 : get_pool_csv_report error, no source: used:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

any pointers on how to fix this would be appreciated

Comments

  • Hi,

    run this from the cmd line to see if there is not any corrupted files

    cd /home/stor2rrd/stor2rrd
    ./bin/check_rrdtool.sh


    It just prints files, remove them manually if there is any,
    let us know if that does not resolve it.
  • mpacyna
    edited July 2021
    Thanks Pavel,

    unfortunately it doesn't seem to have found anything (script name is slightly different though)

    4633c117e624:~/stor2rrd$ ./bin/check_rrdtool_stor2rrd.sh 
    ...Checked files: 360
    No corrupted files have been found
  • send us logs

    Note a short problem description in the text field of the upload form.

    cd /home/lpar2rrd/lpar2rrd  # or where is your LPAR2RRD working dir

    tar cvhf logs.tar logs tmp/*txt tmp/*json

    gzip -9 logs.tar

    Send us logs.tar.gz via https://upload.lpar2rrd.com

Sign In or Register to comment.